If a repo, ref is null it means the same repo, ref as the parent
This means you won't need to specify which repository and branch the
stratum is in any more, it will just use the same one as yours.
This can make edit and petrify simpler.
The branch itself is something thrown together quickly, and I'd prefer
to think of it as the proof of concept, rather than its final state,
but much of the code in question will be rewritten anyway, so it shouldn't
matter if this version is merged.
Richard Maw (4):
Allow building with null repo/ref
Make branch commands work with null refs
Move helper script to scripts for use in cmdtests
Add tests for building and deploying systems with null refs
morphlib/app.py | 8 ++-
morphlib/artifactresolver.py | 8 +--
morphlib/buildcommand.py | 4 +-
morphlib/morph2.py | 3 +-
morphlib/morphset.py | 8 +--
morphlib/plugins/branch_and_merge_new_plugin.py | 7 ++-
scripts/nullify-local-refs | 18 +++++++
tests.build/build-system-with-null-refs.script | 24 +++++++++
tests.build/build-system-with-null-refs.setup | 23 +++++++++
tests.deploy/deploy-with-null-refs.script | 35 +++++++++++++
yarns/branches-workspaces.yarn | 62 +++++++++++++++++++++++
yarns/implementations.yarn | 12 +++++
yarns/morph.shell-lib | 5 ++
13 files changed, 204 insertions(+), 13 deletions(-)
create mode 100755 scripts/nullify-local-refs
create mode 100755 tests.build/build-system-with-null-refs.script
create mode 100755 tests.build/build-system-with-null-refs.setup
create mode 100755 tests.deploy/deploy-with-null-refs.script
This patch is not with a usual patch series, since it involves merging
a large number of commits.
This patch does two things:
1. Merges git v1.8.4 into baserock/morph
2. Convers git.morph's JSON into YAML (because now seemed like a good
Our system artifacts are currently a tar archive of all the files that
go into the system. They can be quite big: a base system for x86-64 is
currently almost 900 megabytes, and a devel system is 1.5 gigabytes.
There's several reasons for this: a primary reason is that we do not
yet split out development tools and libraries from the base system;
work to fix that is going to happen soon.
However, another reason is that we store all the files in a system in
the tar archive. In principle, a system artifact is a union of stratum
artifacts, each of which is a union of chunk artifacts (so actually a
system is a union of chunks). We implement stratum artifacts already
as a list of (versioned!) chunk artifacts, so that a stratum does not
need to store the same files as the chunks already store. We should do
the same for systems.
Unfortunately, a system artifact is not purely a union of stratum
artifacts: it contains some changes to those, most especially we run
ldconfig at system construction time. There's other tools we need to
run then as well, such as constructing cache for gdk-pixbuf loading,
I've prototyped today a way to store system artifacts as a list of
stratum artifacts plus a binary delta between the pure union and after
running ldconfig. Roughly:
* Create a temporary directory, A.
* Hardlink all the chunks in the system into A.
- this is the same procedure as we do for creating staging areas
* Copy A into a new directory B.
- a hardlink copy should do, if we can make the next step safe
* Run ldconfig, and other such things, in B.
* Produce a Trebuchet delta (tbdiff-create) betweewn A and B.
* Create a system artifact consisting of the list of stratum
artifacts plus the Trebuchet delta.
I wrote some very hacky scripts to do this. Result:
916252160 current system artifact
46080 new system artifact
That's about 900 megabytes vs 45 kilobytes. Quite a difference.
When the system artifact is used, it can be reconstructed by unpacking
the stratum artifacts and applying the Trebuchet delta. Again, the
unpacked, cached chunk artifacts can, hopefully, be used to speed up
If we can make this happen, then:
* during system construction, we don't need to write a gigabyte or two
of data when creating the system directory tree, and then the same
amount again when creating the tar archive of the root filesystem
* during deployment, we don't need to write a gigabyte or two of data
for the configuration phase
There's a lot of details to get right, of course, and some
experimentation and research to make sure the unpacked chunk caches
are safe to hardlink into. (ldconfig, and any other programs that make
changes, will need to make them by writing to a temporary file and
renaming that over the real file, rather than modifying the real file
directly. That's usually the case anyway.)
This will not a full patch E-mail, since the only thing that affects
the rest of baserock is glib.
I have a branch on baserock:baserock/morphs named
baserock/xfce-build-rebase-2. It has two commits:
* the first one adds in morphs for building an Xfce system
* the second one updates glib to a newer version (because the Xfce
system requires a newer version of glib), and changes all the stratum
refs to point to the changed morphologies.
I have performed the following tests:
* Build a devel-system-x86_64-generic
* Deploy devel-system-x86_64-generic and test that a programme that
uses glib still works - in this case, I started the
node-startup-controller and checked that it did not raise errors.
* Build a devel-system-armv7b-highbank
* Deploy devel-system-armv7b-highbank and test the
node-startup-controller didn't break
The first patch is for move pyyaml from tools to core, we need pyyaml
in core to be able to build any system (base o devel) with cloudinit.
The second patch is for add cloudinit stratum and two base system to
use it. Cloudinit is needed to make the system suitable into an OpenStack
cloud, giving to the system the capacity of execute scripts at boot time.
Pedro Alvarez (2):
Move pyyaml from 'tools' to 'core'.
Add cloudinit-support Stratum morphology.
base-system-x86_32-openstack.morph | 29 ++++++++++++++++++++++
base-system-x86_64-openstack.morph | 29 ++++++++++++++++++++++
cloudinit-support.morph | 48 ++++++++++++++++++++++++++++++++++++
core.morph | 10 ++++++++
cross-bootstrap.morph | 13 +---------
tools.morph | 13 +---------
6 files changed, 118 insertions(+), 24 deletions(-)
create mode 100644 base-system-x86_32-openstack.morph
create mode 100644 base-system-x86_64-openstack.morph
create mode 100644 cloudinit-support.morph
There's a scary use of contextlib to make a context manager from a
generator function, which:
a) may not be the clearest approach, though I think it's a good one
b) possibly should be defined somewhere other than the branch and
There's a class that abstracts away the /baserock metadata directory
access to being just like a dict, which may need a better name.
Richard Maw (4):
b&m: Add system branch initializing context manager
b&m: checkout and branch use context manager
morphlib: Add SystemMetadataDir class
B&M: refactor branch-from-image
morphlib/__init__.py | 1 +
morphlib/plugins/branch_and_merge_new_plugin.py | 182 ++++++++++++++++++-----
morphlib/plugins/branch_and_merge_plugin.py | 12 +-
morphlib/systemmetadatadir.py | 63 ++++++++
morphlib/systemmetadatadir_tests.py | 66 ++++++++
tests.as-root/branch-from-image-works.script | 3 +-
6 files changed, 276 insertions(+), 51 deletions(-)
create mode 100644 morphlib/systemmetadatadir.py
create mode 100644 morphlib/systemmetadatadir_tests.py