[RFC] Multiple source repositories per-chunk and git submodule handling
by Richard Maw
We currently handle definitions that need to pull multiple source
repositories together in an ad-hoc way.
For gcc we import the gmp, mpc and mpfr source trees in by checking them
into our delta branch directly.
Some upstreams handle importing the sources by adding submodules.
However, we have to make them fetch sources from our Trove for
reproducibilty and latency concerns, so we end up having to add a delta
on top to change where the submodules fetch their sources from.
This works against one of our goals for minimising the delta, so we
need a way to encode in our builds how to deal with components that need
sources from multiple repositories to function.
Our current approach from submodules introduces a _lot_ of extra work
when we need to update multiple submodules recursively, so we need a
better solution.
Proposal
========
To solve this, I propose we extend the source repository information from
just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
So rather than:
name: base
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
unpetrify-ref: baserock/morph
⋮
build-depends: []
We extend it to be able to take a "sources" field.
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
sources:
- repo: upstream:foo
ref: baserock/morph # used to validate commit is anchored
# properly and as a hint for where
# you ought to merge changes to
commit: cafef00d…
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The `path` field is used to specify that this tree should be placed in
the parent tree at this position.
The `path` field is optional, defaulting to `.`
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Sub-sources can be placed over an empty directory, when there's no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
If there's a file, symlink, non-empty directory or submodule that doesn't
match `submodule-commit` there, then it results in a build-failure
at build-time.
The `submodule-commit` check exists as a safeguard against the parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
If you get a build fail because the submodule isn't appropriate then
you have to check whether the version you specify works, then update
the commit.
Cache key changes
=================
This shouldn't require any cache key changes to existing definitions,
but builds that make use of multiple source repositories will also hash
the commit tree and path.
Alternative solutions for submodules
====================================
We could continue to use the current model, and deal with the pain of
having to make multiple branches in multiple repositories to satisfy
the change to the repository paths.
We could have a magic look-up table to replace repository urls when we
parse the submodules.
We could use git-replace to magically make git do the url correction, but
then we'd have to handle replacements properly, rather than dropping them.
7 years, 6 months
[PATCH 0/6] Firehose Implementation
by Lauren Perry
Hi all,
Back in November I sent to the list an RFC for Firehose. This preliminary patch series demonstrates the current work I have done, based on the pre-existing prototype Firehose, and the steps taken towards automating this process.
Currently, Firehose is not fully automated; there are still a number of steps that need to be performed manually, such as setting up a Firehose user, enabling the firehose.service and firehose.timer files and installing the firehose shell for general use. These steps are demonstrated in firehose.configure (prelim) and I will post a follow up patch with a README demonstrating steps that need to be taken currently.
Firehose will also need a Gerrit user configured with an appropriate set of ssh keys to access Gerrit.
A Firehose machine has been demonstrated to work properly on OpenStack, though it requires the firehose command to be done manually.
Regards,
Lauren
Lauren Perry (6):
Add tools for installing Firehose and its dependencies
Add systemd styled timers to automate Firehose checking process
Removed morph after placing its functionality in firehose.sh
Modify firehose email configuration and push location to ensure
commits are pushed to the testgerrit server with an appropriate
commit message and branch name
Install githook to add Change-Id value to commit message
Add firehose.configure, containing the necessary instructions to run
firehose
firehose.configure | 39 ++++++++++++++++++++++++++
firehose.service | 12 ++++++++
firehose.sh | 21 ++++++++++++++
firehose.timer | 11 ++++++++
{plugin => firehose/plugin}/firehose_plugin.py | 20 +++++++++----
morph | 17 -----------
setup.py | 16 +++++++++++
7 files changed, 113 insertions(+), 23 deletions(-)
create mode 100644 firehose.configure
create mode 100644 firehose.service
create mode 100755 firehose.sh
create mode 100644 firehose.timer
rename {plugin => firehose/plugin}/firehose_plugin.py (92%)
delete mode 100755 morph
create mode 100644 setup.py
--
1.8.4
--
1.7.10.4
7 years, 11 months
Tracking ABI of Baserock build-host systems
by Sam Thursfield
Hello
We have a weakness in the way Baserock systems are bootstrapped, and
today it finally bit me!
Right now Morph ignores the question of whether a chunk built in
'bootstrap' mode will run on other systems. Morph assumes that a chunk
built on an x86_64 Baserock system will function the same on any other
x86_64 Baserock system. So for example, the cache key of stage1-binutils
will be the same on any two Baserock systems, assuming it's the same
architecture, same build instructions and same source code commit.
Anyone who has been in software more than 5 minutes will know this is
fantasy. But it's a very convenient fantasy. I'm not sure how many
people have actually found themselves on the wrong side of this
assumption so far -- it can't be many, or the mailing list would be full
of complaints.
I'll quickly sum up the problem it caused me. On ARM hard-float systems,
the upgrade from EGLIBC 2.15 to GLIBC 2.20 introduced an ABI break:
/lib/ld-linux.so.3 became /lib/ld-linux-armhf.so.3. So dynamically
linked programs built on an ARM hard-float system using EGLIBC 2.15
won't actually run on a GLIBC 2.20 system: they'll give a 'not found'
error. I had an ARM Mason set up to use a shared artifact cache which
was already being used by a much older ARM distbuild network. The older
distbuild had built and uploaded the bootstrap chunks stage1-binutils,
stage1-gcc etc. The new Mason was working fine with these until it had
to rebuild a bootstrap chunk: I think it was stage2-make. At this point
it tried to *run* the tools from stage1-binutils and stage1-gcc to build
make, and they didn't work because they were linked against the wrong
ld.so. Result: a confusing build failure!
I think we can fix this ABI break by creating a compatibility symlink in
/lib, and I'll try doing that. But it highlights a hole in our story of
'everything is reproducible and it won't randomly break for you'. Here
is a list of things we could do to tie that hole closed.
Note when I say "host system" below I'm referring to the Baserock system
that is running `morph build`. Not VM hosts.
1) Include the cache key of the host system in the cache key of each
chunk built in 'bootstrap' mode.
- assumes Morph is running on a Baserock system
- makes cache.baserock.org less useful, because unless you're running
the exact same build-system as the Mason that built the artifacts, your
Morph will come up with different cache keys for the same commit of
definitions and will build everything locally.
2) Include the cache keys of certain chunks from the host system in the
cache key of each chunk built in 'bootstrap' mode.
- assumes Morph is running on a Baserock system, and makes assumptions
about the makeup of that system
- non-trivial to implement
At present the list of chunks would be something like: glibc, binutils,
gcc. The list would have to be maintained by a human in definitions.git,
and we'd have to make a judgement call about where to draw the line.
Once you start digging you realise every component *could* affect how
something is built, so I think going down this road is a bit pointless,
but it might be worthwhile stepping stone while we try to achieve (1).
3) Always build bootstrap chunks locally
- will make getting started slower for everyone (each new user will need
to build GCC twice locally before doing anything else)
4) Statically link bootstrap chunks.
- requires some big changes to the bootstrap, I don't know if it would
actually be possible to statically link everything in stage1 and stage2
- stage1 and stage2 artifacts would be bigger
- still vulnerable to broken versions of GCC/G++/Binutils generating bad
binaries
- will only fix our current example Linux/GNU C/C++/Fortran bootstrap
(the 'build-essential' stratum), will not solve the problem for everyone
(although the principle would remain the same)
5) Make sure we never break our reference bootstrap
- this is our current approach, but it clearly doesn't always work, and
it continually requires developers and reviewers to think whether a
change will might break the bootstrap on any of our platforms
Anyone have any more suggestions for how we could solve this? Any
preferred solutions?
In writing this email I've convinced myself that (1) is really the only
option and so we should make it happen sooner rather than later, on the
assumption that the longer we leave it the more painful it will become.
If the Mason systems are always running latest master of definitions,
then as long as you are also running latest master, you'll be able to
use artifacts from cache.baserock.org. Except if you run a devel-system
instead of a build-system you won't, because it'll have a different
cache key. I have a possible idea for how to fix this if we moved to
allowing arbitrary nesting of components, but I think this email has
gone on a while already, and it's not something that is going to get
done next week.
Please let me know your thoughts! This is a bit of a mindbender so don't
be afraid to ask questions. I know that Emmet has expressed a desire for
(1) multiple times already, but I think it's only been in IRC so I
haven't tried to hunt out previous discussions from the mailing list.
Thanks
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
8 years, 1 month
[PATCH] Add mips support to morph
by Simon Hoinkis
repo: ssh://git@github.com/mossmaurice/morph.git
branch: baserock/simonhoinkis/mips64
commit: 79bfcdbccdc5d45028b69df720f1f348c65843ed
This patch adds support to morph for MIPS32/64 in big and little endian.
Simon Hoinkis (1):
Change build environment to support MIPS32/64 BE and LE
morphlib/__init__.py | 4 ++--
morphlib/buildenvironment.py | 11 +++++++++++
morphlib/util.py | 4 ++++
3 files changed, 17 insertions(+), 2 deletions(-)
--
1.8.4
8 years, 1 month
[PATCH 0/8] Fix simple-network extension to handle systemd's .network files
by Javier Jardón
From: Javier Jardón <javier.jardon(a)codethink.co.uk>
REPO: ssh://git@git.baserock.org/baserock/baserock/morph.git
BRANCH: jjardon/simple-network-networkd
COMMIT: a4599ee823eb65c3a125c85149abab487df73bec
This will make possible to specify static networks at the deploy time again.
Also we can remove the default network configuration from the systemd morphology,
as the default here would generate the same .network file.
Note that the support is limited and the only configurable parameters at the
moment are 'address', 'netmask' and 'gateway'.
Probably is good to change the current format of CONFIG_NETWORK in the future
to support the .network format directly, so we can support all the networkd
options fully
Javier Jardón (8):
simple-network.configure: write to .network files
simple-network.configure: Generate .network format
simple-network.configure: No need to order interfaces in the
configuration file
simple-network.configure: No need to configure loopback interface
simple-network.configure: Add function to convert mask to cidr suffix
simple-network.configure: process pairs of parameters in a function
simple-network.configure: Update documentation
simple-network.configure: Use a better default for the interface name
morphlib/exts/simple-network.configure | 94 +++++++++++++++++++++++-----------
1 file changed, 63 insertions(+), 31 deletions(-)
--
2.3.0
8 years, 2 months
[PATCH 0/8] morph: Implement versioning of the definitions format
by Javier Jardón
From: Javier Jardón <javier.jardon(a)codethink.co.uk>
REPO: ssh://git@git.baserock.org/baserock/baserock/morph.git
BRANCH: jjardon/versioning
COMMIT: 3d2bb81628e0cf763826f0827684d77bbe7564ae
Following the discussions to improve the syntax of the current definitions[1],
here a series of patches to:
- Detect the presence of the VERSION file
- Support definitions format version '0'
as described in [1]. Basically, the VERSION file will look as this:
version: 0
In the current implementation morph will NOT fail if the VERSION file doesn't exist,
I can change that if you think its better to only support definitions with it (I can
remove the last patch of the series)
Bear in mind that its a long time since I touched the morph code, so any suggestions
to improve the quality of the patches are more than welcome
[1] http://wiki.baserock.org/meetup/declarative-definitions/
Javier Jardón (8):
morphlib/sourceresolver.py: Add _get_file_contents() helper function
morphlib/sourceresolver.py: Add VersionNotFoundError
morphlib/sourceresolver.py: Check VERSION file exists
morphlib/sourceresolver.py: Be sure VERSION file is a valid YAML file
morphlib/sourceresolver.py: Move VERSION file checks to a function
morphlib/sourceresolver.py: Add UnknownVersionError
morphlib/sourceresolver.py: Only support definitions format version
'0'
morphlib/sourceresolver.py: Do not fail if VERSION file doesn't exist
morphlib/sourceresolver.py | 75 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 75 insertions(+)
--
2.3.0
8 years, 2 months
The baserock UI - meetup notes
by Mike Smith
This is my write-up of the discussion from the Baserock meet up on 5th
February concerning the user interface of Baserock.
i will first go over the problems that where discussed followed by the
actions and potential solutions that where arrived at through further
discussion
It is my intention that this e-mail be commented on with counter
arguments to the problems, additions to the list of problems, addition
of any problems i have neglected to include on this list, and of course
discussion addition alteration and any other communication related to
the potential solutions that have been discussed.
*Problems with the UI*
The largest thing to be discussed was the entry level to using Baserock,
and the sharp learning curve. as a comparison people where asked how
long it took for them to feel they could call themselves "productive"
with Baserock. the reported times ranged from 3-4 days upwards getting
towards nearly 3 weeks in some cases.
This was due to the simple fact that the Baserock community are ,almost
all, very familiar and skilled with Git. And the assumption has been
made that users of Baserock would be equally as skilled. This in many
instances is not the case. A new starter to Baserock must first learn
Git, and then after they have learnt Git to a reasonable level, must
then proceed to learn Morph, which extends on Git, but often not in ways
that would be expected by a novice user. For example, branch handling in
Git and morph are two separate things. checking out and altering a Morph
branch does not necessarily mean that the related Git branch has also
been altered, this can lead to situations where the user is sure that
they have created and committed a new branch, but have in fact done
nothing other than local changes.
It was also pointed out that much of the terminology and short cuts used
for the community can be confusing for a new user. for example the
reference to a Sha1 of a branch rather than a fully qualified name of a
branch will lead an inexperienced user to simply leave this alone as it
is an un-known to them and best left alone.
*Solutions to Problems*
* Allow morph to be used /without/ workspaces. this reduces the
complexity of the overall system and removes the need for branches and
checkouts.
* Get rid of push and Pull in morph and add the ability to commit / add
Binarys
* Morph Build and Deploy should by default use "Git HEAD"
* Create some use-case storys. these should act as instructions and
how-to's of various common scenarios. I.E.
"I am in situation X, I need to be in situation Y, to do this I did the
steps A,B,C. there was a potential to use option D in C's place. But in
this case C was the simpler approach"
* These how-to's /*Must be tested by people outside of the current
Baserock community*/ as an acceptance criteria to them fulfilling the
need for a guide that any computer user can follow.
* Morph should potential have a one command "build a default generic
Baserock image and deploy" so that users can quickly get up and running
with the system.
* Potentially look into creating an online interactive tutorial for
Baserock to familiarise the new user with the set-up and use of Baserock
in a safe environment ( The inspiration for this was taken from the
Docker site at https://www.docker.com/tryit/ )
8 years, 2 months
[PATCH 0/6] morph: Various fixes to distbuild build logs
by Sam Thursfield
Repository: git://git.baserock.org/baserock/baserock/morph
Ref: sam/distbuild-build-logs
Sha1: eba2e42855e9413f035e5093d64543184dce6fae
Sam Thursfield (6):
Update copyright years
distbuild: Fix build logs being sent to the wrong log files
distbuild: Use source name, not artifact name, for build step logs
distbuild: Create a new directory to store build logs for each build.
distbuild: Remove the build-steps message
distbuild: Log in build-step-xx.log files when initiator cancels build
distbuild/__init__.py | 7 +-
distbuild/build_controller.py | 22 ++----
distbuild/initiator.py | 61 ++++++++++------
distbuild/initiator_connection.py | 24 +-----
distbuild/mainloop.py | 5 +-
distbuild/protocol.py | 6 +-
distbuild/worker_build_scheduler.py | 137 +++++++++++++++++++++--------------
morphlib/buildcommand.py | 10 ++-
morphlib/plugins/distbuild_plugin.py | 1 -
9 files changed, 143 insertions(+), 130 deletions(-)
--
1.9.3
8 years, 2 months
[PATCH 0/2] Fascilitating openstack swift integration.
by Patrick Darley
Repo: https://github.com/padrigali/definitions
Branch: integrate-swift-v3
Commit: b50fbcf9331f4d8c4106d15359fa92adb03ae913
Patrick Darley (2):
Update the version of rsync and install systemd unit provided with
this new verison
Install the systemd unit included with memcached source
strata/databases/memcached.morph | 13 +++++++++++++
strata/tools/rsync.morph | 3 +++
2 files changed, 16 insertions(+)
--
2.1.4
8 years, 3 months
