[RFC] Multiple source repositories per-chunk and git submodule handling
by Richard Maw
We currently handle definitions that need to pull multiple source
repositories together in an ad-hoc way.
For gcc we import the gmp, mpc and mpfr source trees in by checking them
into our delta branch directly.
Some upstreams handle importing the sources by adding submodules.
However, we have to make them fetch sources from our Trove for
reproducibilty and latency concerns, so we end up having to add a delta
on top to change where the submodules fetch their sources from.
This works against one of our goals for minimising the delta, so we
need a way to encode in our builds how to deal with components that need
sources from multiple repositories to function.
Our current approach from submodules introduces a _lot_ of extra work
when we need to update multiple submodules recursively, so we need a
better solution.
Proposal
========
To solve this, I propose we extend the source repository information from
just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
So rather than:
name: base
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
unpetrify-ref: baserock/morph
⋮
build-depends: []
We extend it to be able to take a "sources" field.
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
sources:
- repo: upstream:foo
ref: baserock/morph # used to validate commit is anchored
# properly and as a hint for where
# you ought to merge changes to
commit: cafef00d…
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The `path` field is used to specify that this tree should be placed in
the parent tree at this position.
The `path` field is optional, defaulting to `.`
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Sub-sources can be placed over an empty directory, when there's no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
If there's a file, symlink, non-empty directory or submodule that doesn't
match `submodule-commit` there, then it results in a build-failure
at build-time.
The `submodule-commit` check exists as a safeguard against the parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
If you get a build fail because the submodule isn't appropriate then
you have to check whether the version you specify works, then update
the commit.
Cache key changes
=================
This shouldn't require any cache key changes to existing definitions,
but builds that make use of multiple source repositories will also hash
the commit tree and path.
Alternative solutions for submodules
====================================
We could continue to use the current model, and deal with the pain of
having to make multiple branches in multiple repositories to satisfy
the change to the repository paths.
We could have a magic look-up table to replace repository urls when we
parse the submodules.
We could use git-replace to magically make git do the url correction, but
then we'd have to handle replacements properly, rather than dropping them.
7 years, 6 months
[PATCH 0/3] Add the ability to deploy installer systems
by Pedro Alvarez
Repo: baserock:baserock/definitions
Branch: baserock/pedroalvarez/installer2
Sha1: fdd42736cec68a22388f0d008e5cb7078ed9b1c4
Land: master
Pedro Alvarez (3):
Update morph to a version that allows deployments to devices
Add the ability to deploy installer systems
Add example cluster of a baserock installer system
clusters/installer-build-system-x86_64.morph | 38 ++++++
installer/etc/install.conf | 3 +
installer/installer.py | 178 ++++++++++++++++++++++++++
installer/manifest | 3 +
strata/morph-utils.morph | 2 +-
5 files changed, 223 insertions(+), 1 deletion(-)
create mode 100644 clusters/installer-build-system-x86_64.morph
create mode 100644 installer/etc/install.conf
create mode 100755 installer/installer.py
create mode 100644 installer/manifest
--
1.7.10.4
8 years, 4 months
[RFC] YBD - proof-of-concept work for improving Baserock workflow
by Paul Sherwood
Over the last month or so I've been exploring some possible improvements
for Baserock, in particular some simplifications to
- syntax for definitions
- code for parsing definitions
- code for overall build logic
- hashing algorithm for cache-keys
My approach has been to start from a clean base, and then bring in
ideas/code from morph where I needed them. I did consider trying to
implement these things in morph itself, but got scared off by the
codebases for morph and its tests.
I've ignored some real-world complications, and frankly was not
intending to publish yet - I wanted to get it actually building
something first.
But given the discussions on runtime-depends, I'm offering this now,
just in case the simplifications I'm proposing can help to steer us away
from further complications in morph itself.
So, that said:
https://github.com/devcurmudgeon/ybd
To try it out
- git clone it into (say) /src/workspace
- git clone definitions into a sibling directory
- in the definitions directory
../ybd/morph-converter.sh
../ybd/check.sh systems/genivi-baseline-x86_64-generic.def
(or, for example) ../ybd/check.sh strata/core.def
Notes
- it doesn't build, but it does
- clone git repos from gbo if necessary (which is slow...)
- walk the definitions tree in build order, very fast
- calculate cache-keys for all required artifacts, very fast
- warn on duplicate definitions
(eg we have several u-boot definitions)
- error on recursive definitions
(eg ruby stratum includes ruby chunk)
- report last upstream tag for a given definition
- it can be run against any system/stratum/chunk
- less than 1000 lines of code at this point
I believe that
- this cache calculation is easier to understand than what's in morph
(and even if there are factors i'm missing, it would still be better
imo to do it this way)
- this logic allows us to assemble only components that are specified
for inclusion ('contents' in this syntax) in output artifacts, hence
no need (I think) for runtime-dependencies
br
Paul
--
Paul Sherwood Codethink Ltd.
Tel: +44 788 798 4900 302 Ducie House, Ducie Street,
http://www.codethink.co.uk/ Manchester, M1 2JW, United Kingdom.
Codethink provides advanced software design, development, integration &
test services: from embedded systems to high performance apps to cloud.
8 years, 4 months
Policy: Merging documentation changes
by Pete Fotheringham
As the documentation changes I am working on are mostly in source files,
I am sending them for review in the same way as for patches which change
code. However, the normal patch review workflow (whereby changes don't
get merged until all comments have been satisfactorily addressed) seems
a little heavy handed, given that there is no chance of my changes
'breaking' existing code.
I would like to propose a change, whereby
1. 'Documentation only' changes can be merged *before* all comments have
been addressed / corrected - unless any reviewer feels the changes are
bad enough to warrant a -1;
2. Comments received will be addressed in the next patch set that is
sent for review
Or to express it another way, reviewers can give these changes +1 even
if they have identified changes that need to be made.
I believe that making this change would speed up the process of
improving the quality of the Baserock user documentation, without any
negative effects on the functionality of the software.
Thoughts?
Pete
--
Pete Fotheringham
+44 7740 351755
Codethink Ltd
http://codethink.co.uk
8 years, 4 months
[PATCH] Update wpa-supplicant and iptables to latest upstream tags
by Paul Sherwood
repo: baserock:baserock/definitions
branch: baserock/ps/update-iptables-wpa-supplicant
http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.gi...
While looking into a query from GENIVI I've updated iptables and and
wpa-supplicant (hostap) to the latest upstream release tags.
These build successfully on x86_64 - I'm not sure what to test, though.
commit 165fbfc69a772ca7ff557ccc29eea47c01ef57f0
Author: Paul Sherwood <paul.sherwood(a)codethink.co.uk>
Date: Tue Nov 25 21:31:39 2014 +0000
Update hostap/wpa_supplicant and iptables to latest tag
diff --git a/strata/connectivity.morph b/strata/connectivity.morph
index a8697f3..2cc6bb6 100644
--- a/strata/connectivity.morph
+++ b/strata/connectivity.morph
@@ -11,12 +11,12 @@ chunks:
- name: wpa_supplicant
morph: strata/connectivity/wpa_supplicant.morph
repo: upstream:hostap
- ref: 1cdfb99c3a04235697dc6979b9eb5418861e42f8
- unpetrify-ref: baserock/genivi/morph
+ ref: c4a58c968044c2002706b1a69c089c4d63728e77
+ unpetrify-ref: baserock/hostap_2_3
build-depends:
- libnl
- name: iptables
repo: upstream:iptables
- ref: 3e6fa55d5e28c93f417afeae7a7d4f349ddffcf4
- unpetrify-ref: baserock/morph
+ ref: 482c6d3731e2681cb4baae835c294840300197e6
+ unpetrify-ref: v1.4.21
build-depends: []
8 years, 5 months
[PATCH v2 0/8] Treat path given to build cmd as relative to cwd
by Richard Ipsum
repo: git://git.baserock.org/baserock/baserock/morph
ref: baserock/richardipsum/build-command-tweak-v2
head: 4ae06dc0c9a944a755b2af278dd697e5f47b9884
Hello again,
Adam pointed out that the previous version introduced an inconsistency between
the build commands and the deploy command, this version makes both the build
commands and the deploy command treat the system arg as a path relative to
the current working directory. The patches at the end of this series contain
a couple of improvements for the tests.
This is being resubmitted as a new series because the relative_to_root_repo
function has been reworked so that it only needs to take a single argument:
the path we want to make relative to the root repo.
The SystemBranchDirectory class contains the root_repo_url,
so there's no need to pass this to our new function.
Also in the previous version, Richard Maw suggested that rather than merge the
previous version of this series we might prefer to perform
some more involved reworks of this area, from what I understand of this and
coupled with ideas taken from ybd, this would involve removing workspaces and
making definitions more path based so that, for example,
paths referenced in strata are relative to the stratum's location in the file
system. I'm eager that we get these suggestions implemented in morph, but
suggest that this series may form a useful baserock (sorry) for the larger
changes we have in mind.
Richard Ipsum (8):
Add relative_to_root_repo to sysbranchdir
Treat path given to build cmd as relative to cwd
Update tests to work with modified build commands
Add yarns to test building with relative paths
Treat path given to deploy cmd as relative to cwd
Add yarns to test deploying with relative paths
Quote variables
Make some existing deployment scenarios simpler
morphlib/plugins/build_plugin.py | 8 +++--
morphlib/plugins/deploy_plugin.py | 6 ++--
morphlib/sysbranchdir.py | 28 +++++++++++++++--
yarns/building.yarn | 24 +++++++++++++++
yarns/deployment.yarn | 46 +++++++++++++++++++---------
yarns/implementations.yarn | 61 ++++++++++++++++++++++++++++---------
6 files changed, 138 insertions(+), 35 deletions(-)
--
1.7.10.4
8 years, 5 months
[RFC] Automated Integration Tool (Firehose)
by Lauren Perry
Hi all,
Over the past few weeks, I have been looking at the preliminary designs
of a prototype morph plugin called Firehose.
Currently, Baserock has very little automation between code review,
merging and pushing to upstream. Ideally, a Baserock system's source
code would never require modification, and any integration would only
affect the definitions repository of the system in question, meaning
that we would have little need for automation of processes. However,
systems often require integration of patches to fix bugs, improve
efficiency, add or remove features and so on.
What Firehose aims to do is automate the integration of software
components into component groups and then into full systems (i.e.
integrating chunks into strata into systems, using current Baserock
terminology). The upside of this is that it will save time for the
engineers who would typically have to do this themselves.
To do this, Firehose can examine the target branch of the definitions
repository, then create and push a system branch for each individual
change to Trove. If there are a number of refs requiring updating, there
should also be a consistent method of ordering them for updating.
While we need to check for changes to upstream ref values, this is
currently only performed when the Firehose tool is run. It may be
beneficial to instead perform these checks at regular intervals, given
how often a repo may be updated. One method could be a Cron job set to
check every five minutes or so, and update if required.
For optimal use, Firehose will require a patch tracker such as Gerrit
and a testing framework such as Zuul to continue automation into the
code review, testing and patch merging processes. These would be
triggered as normal on Firehose's git-commit and git-push commands.
For anyone interested in viewing a base structure of Firehose, there is
currently a prototype located on git.baserock.org and the CodethinkLabs
github page.
Any questions, suggestions and opinions on Firehose and its
implementation, or suggestions of things I've missed, are very much welcome.
Regards,
Lauren
8 years, 6 months
[PATCH 0/6] Add support to deploy to devices in rawdisk.write
by Pedro Alvarez
Repo: baserock:baserock/morph
Branch: baserock/pedroalvarez/rawdisk-to-device4
Sha1: 8b3f1e02b6c5e1d69e34b1b924681f8c408bae54
Land: master
Pedro Alvarez (6):
Add 'is_device' function to check if we are deploying to a device
Add force flag to 'mkfs.btrfs'
Disable loop mount, leave the system decide
Move DISK_SIZE detection inside create_raw_disk_image
Don't create the rawdisk image inside of create_local_system
Don't create rawdisk image if deploying to a device
morphlib/exts/kvm.write | 1 +
morphlib/exts/openstack.write | 1 +
morphlib/exts/rawdisk.write | 7 ++++++-
morphlib/exts/virtualbox-ssh.write | 1 +
morphlib/writeexts.py | 20 ++++++++++++--------
5 files changed, 21 insertions(+), 9 deletions(-)
--
1.7.10.4
8 years, 6 months
[PATCH] Make cycle.sh work for target systems too
by Paul Sherwood
repo: baserock:baserock/definitions
branch: baserock/ps/cycle-from-chroot
This re-write runs all of the system-version-manager commands on
the target cluster via ssh, so in addition to self-upgrades,
cycle.sh can now be used to upgrade a target device (eg a devboard)
while working in Baserock VM or chroot
---
diff --git a/scripts/cycle.sh b/scripts/cycle.sh
index c0e2aa6..ca6983e 100755
--- a/scripts/cycle.sh
+++ b/scripts/cycle.sh
@@ -14,6 +14,8 @@
# with this program; if not, write to the Free Software Foundation,
Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+set -e
+
usage() {
echo "Usage: cycle.sh some-system some-cluster [newversion]"
echo
@@ -38,24 +40,27 @@ if [ ! -z "$3" ] ; then
fi
fi
-if system-version-manager get-running | grep -q "^$newversion$"; then
- echo "You are currently running the $newversion system."
+location=`cat $2 | grep location | sed 's/ \+//g' | cut -d":" -f2`
+svm="ssh $location system-version-manager"
+runningversion=`$svm get-running'`
+
+if [ $runningversion = $newversion ]; then
+ echo "$newversion is currently running at $location"
echo "Maybe you want to boot into a different system version?"
exit 1
fi
-set -e
-set -v
-
-runningversion=`system-version-manager get-running`
-system-version-manager set-default $runningversion
-if system-version-manager list | grep -q "^$newversion$"; then
- system-version-manager remove $newversion
+$svm "set-default $runningversion"
+if $svm "list" | grep -q "^$newversion$"; then
+ $svm "remove $newversion"
fi
+echo $(date +%Y-%m-%d\ %T) "Cyclng $location from $runningversion to
$newversion"
+
morph gc
morph build "$1"
sed -i "s|^- morph: .*$|- morph: $1|" "$2"
-morph deploy --upgrade "$2" self.HOSTNAME=$(hostname)
self.VERSION_LABEL=$newversion
-system-version-manager list
+host=`ssh $location hostname`
+morph deploy --upgrade "$2" self.HOSTNAME=$host
self.VERSION_LABEL=$newversion
+$svm "list"
--
cgit v0.9.0.3-67-gacbf
8 years, 6 months
[PATCH 0/4] Documentation for kvm and virtualbox-ssh write extensions
by Pete Fotheringham
Pushed to `pf-document-extensions` branch in githup
https://github.com/petefoth/morph/tree/pf-document-extensions
Pete Fotheringham (4):
Add more information on existing parameters. Add AUTOSTART
parameter
Improve readability by adding blank lines
Refere to .write.help file for documentation
More complete help documentation
morphlib/exts/kvm.write | 6 ++-
morphlib/exts/kvm.write.help | 30 +++++++++---
morphlib/exts/virtualbox-ssh.write | 31 +++---------
morphlib/exts/virtualbox-ssh.write.help | 87 +++++++++++++++++++++++++++++++--
4 files changed, 117 insertions(+), 37 deletions(-)
--
1.7.11.3
8 years, 6 months
