We currently handle definitions that need to pull multiple source
repositories together in an ad-hoc way.
For gcc we import the gmp, mpc and mpfr source trees in by checking them
into our delta branch directly.
Some upstreams handle importing the sources by adding submodules.
However, we have to make them fetch sources from our Trove for
reproducibilty and latency concerns, so we end up having to add a delta
on top to change where the submodules fetch their sources from.
This works against one of our goals for minimising the delta, so we
need a way to encode in our builds how to deal with components that need
sources from multiple repositories to function.
Our current approach from submodules introduces a _lot_ of extra work
when we need to update multiple submodules recursively, so we need a
To solve this, I propose we extend the source repository information from
just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
So rather than:
- name: foo
We extend it to be able to take a "sources" field.
- name: foo
- repo: upstream:foo
ref: baserock/morph # used to validate commit is anchored
# properly and as a hint for where
# you ought to merge changes to
- repo: delta:gnulib
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
The `path` field is used to specify that this tree should be placed in
the parent tree at this position.
The `path` field is optional, defaulting to `.`
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Sub-sources can be placed over an empty directory, when there's no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
If there's a file, symlink, non-empty directory or submodule that doesn't
match `submodule-commit` there, then it results in a build-failure
The `submodule-commit` check exists as a safeguard against the parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
If you get a build fail because the submodule isn't appropriate then
you have to check whether the version you specify works, then update
Cache key changes
This shouldn't require any cache key changes to existing definitions,
but builds that make use of multiple source repositories will also hash
the commit tree and path.
Alternative solutions for submodules
We could continue to use the current model, and deal with the pain of
having to make multiple branches in multiple repositories to satisfy
the change to the repository paths.
We could have a magic look-up table to replace repository urls when we
parse the submodules.
We could use git-replace to magically make git do the url correction, but
then we'd have to handle replacements properly, rather than dropping them.
Over the last month or so I've been exploring some possible improvements
for Baserock, in particular some simplifications to
- syntax for definitions
- code for parsing definitions
- code for overall build logic
- hashing algorithm for cache-keys
My approach has been to start from a clean base, and then bring in
ideas/code from morph where I needed them. I did consider trying to
implement these things in morph itself, but got scared off by the
codebases for morph and its tests.
I've ignored some real-world complications, and frankly was not
intending to publish yet - I wanted to get it actually building
But given the discussions on runtime-depends, I'm offering this now,
just in case the simplifications I'm proposing can help to steer us away
from further complications in morph itself.
So, that said:
To try it out
- git clone it into (say) /src/workspace
- git clone definitions into a sibling directory
- in the definitions directory
(or, for example) ../ybd/check.sh strata/core.def
- it doesn't build, but it does
- clone git repos from gbo if necessary (which is slow...)
- walk the definitions tree in build order, very fast
- calculate cache-keys for all required artifacts, very fast
- warn on duplicate definitions
(eg we have several u-boot definitions)
- error on recursive definitions
(eg ruby stratum includes ruby chunk)
- report last upstream tag for a given definition
- it can be run against any system/stratum/chunk
- less than 1000 lines of code at this point
I believe that
- this cache calculation is easier to understand than what's in morph
(and even if there are factors i'm missing, it would still be better
imo to do it this way)
- this logic allows us to assemble only components that are specified
for inclusion ('contents' in this syntax) in output artifacts, hence
no need (I think) for runtime-dependencies
Paul Sherwood Codethink Ltd.
Tel: +44 788 798 4900 302 Ducie House, Ducie Street,
http://www.codethink.co.uk/ Manchester, M1 2JW, United Kingdom.
Codethink provides advanced software design, development, integration &
test services: from embedded systems to high performance apps to cloud.
As the documentation changes I am working on are mostly in source files,
I am sending them for review in the same way as for patches which change
code. However, the normal patch review workflow (whereby changes don't
get merged until all comments have been satisfactorily addressed) seems
a little heavy handed, given that there is no chance of my changes
'breaking' existing code.
I would like to propose a change, whereby
1. 'Documentation only' changes can be merged *before* all comments have
been addressed / corrected - unless any reviewer feels the changes are
bad enough to warrant a -1;
2. Comments received will be addressed in the next patch set that is
sent for review
Or to express it another way, reviewers can give these changes +1 even
if they have identified changes that need to be made.
I believe that making this change would speed up the process of
improving the quality of the Baserock user documentation, without any
negative effects on the functionality of the software.
+44 7740 351755
Adam pointed out that the previous version introduced an inconsistency between
the build commands and the deploy command, this version makes both the build
commands and the deploy command treat the system arg as a path relative to
the current working directory. The patches at the end of this series contain
a couple of improvements for the tests.
This is being resubmitted as a new series because the relative_to_root_repo
function has been reworked so that it only needs to take a single argument:
the path we want to make relative to the root repo.
The SystemBranchDirectory class contains the root_repo_url,
so there's no need to pass this to our new function.
Also in the previous version, Richard Maw suggested that rather than merge the
previous version of this series we might prefer to perform
some more involved reworks of this area, from what I understand of this and
coupled with ideas taken from ybd, this would involve removing workspaces and
making definitions more path based so that, for example,
paths referenced in strata are relative to the stratum's location in the file
system. I'm eager that we get these suggestions implemented in morph, but
suggest that this series may form a useful baserock (sorry) for the larger
changes we have in mind.
Richard Ipsum (8):
Add relative_to_root_repo to sysbranchdir
Treat path given to build cmd as relative to cwd
Update tests to work with modified build commands
Add yarns to test building with relative paths
Treat path given to deploy cmd as relative to cwd
Add yarns to test deploying with relative paths
Make some existing deployment scenarios simpler
morphlib/plugins/build_plugin.py | 8 +++--
morphlib/plugins/deploy_plugin.py | 6 ++--
morphlib/sysbranchdir.py | 28 +++++++++++++++--
yarns/building.yarn | 24 +++++++++++++++
yarns/deployment.yarn | 46 +++++++++++++++++++---------
yarns/implementations.yarn | 61 ++++++++++++++++++++++++++++---------
6 files changed, 138 insertions(+), 35 deletions(-)
Over the past few weeks, I have been looking at the preliminary designs
of a prototype morph plugin called Firehose.
Currently, Baserock has very little automation between code review,
merging and pushing to upstream. Ideally, a Baserock system's source
code would never require modification, and any integration would only
affect the definitions repository of the system in question, meaning
that we would have little need for automation of processes. However,
systems often require integration of patches to fix bugs, improve
efficiency, add or remove features and so on.
What Firehose aims to do is automate the integration of software
components into component groups and then into full systems (i.e.
integrating chunks into strata into systems, using current Baserock
terminology). The upside of this is that it will save time for the
engineers who would typically have to do this themselves.
To do this, Firehose can examine the target branch of the definitions
repository, then create and push a system branch for each individual
change to Trove. If there are a number of refs requiring updating, there
should also be a consistent method of ordering them for updating.
While we need to check for changes to upstream ref values, this is
currently only performed when the Firehose tool is run. It may be
beneficial to instead perform these checks at regular intervals, given
how often a repo may be updated. One method could be a Cron job set to
check every five minutes or so, and update if required.
For optimal use, Firehose will require a patch tracker such as Gerrit
and a testing framework such as Zuul to continue automation into the
code review, testing and patch merging processes. These would be
triggered as normal on Firehose's git-commit and git-push commands.
For anyone interested in viewing a base structure of Firehose, there is
currently a prototype located on git.baserock.org and the CodethinkLabs
Any questions, suggestions and opinions on Firehose and its
implementation, or suggestions of things I've missed, are very much welcome.
Pedro Alvarez (6):
Add 'is_device' function to check if we are deploying to a device
Add force flag to 'mkfs.btrfs'
Disable loop mount, leave the system decide
Move DISK_SIZE detection inside create_raw_disk_image
Don't create the rawdisk image inside of create_local_system
Don't create rawdisk image if deploying to a device
morphlib/exts/kvm.write | 1 +
morphlib/exts/openstack.write | 1 +
morphlib/exts/rawdisk.write | 7 ++++++-
morphlib/exts/virtualbox-ssh.write | 1 +
morphlib/writeexts.py | 20 ++++++++++++--------
5 files changed, 21 insertions(+), 9 deletions(-)
This re-write runs all of the system-version-manager commands on
the target cluster via ssh, so in addition to self-upgrades,
cycle.sh can now be used to upgrade a target device (eg a devboard)
while working in Baserock VM or chroot
diff --git a/scripts/cycle.sh b/scripts/cycle.sh
index c0e2aa6..ca6983e 100755
@@ -14,6 +14,8 @@
# with this program; if not, write to the Free Software Foundation,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
echo "Usage: cycle.sh some-system some-cluster [newversion]"
@@ -38,24 +40,27 @@ if [ ! -z "$3" ] ; then
-if system-version-manager get-running | grep -q "^$newversion$"; then
- echo "You are currently running the $newversion system."
+location=`cat $2 | grep location | sed 's/ \+//g' | cut -d":" -f2`
+svm="ssh $location system-version-manager"
+if [ $runningversion = $newversion ]; then
+ echo "$newversion is currently running at $location"
echo "Maybe you want to boot into a different system version?"
-system-version-manager set-default $runningversion
-if system-version-manager list | grep -q "^$newversion$"; then
- system-version-manager remove $newversion
+$svm "set-default $runningversion"
+if $svm "list" | grep -q "^$newversion$"; then
+ $svm "remove $newversion"
+echo $(date +%Y-%m-%d\ %T) "Cyclng $location from $runningversion to
morph build "$1"
sed -i "s|^- morph: .*$|- morph: $1|" "$2"
-morph deploy --upgrade "$2" self.HOSTNAME=$(hostname)
+host=`ssh $location hostname`
+morph deploy --upgrade "$2" self.HOSTNAME=$host