[PATCH 0/4] Allow null refs when referring to strata
by Richard Maw
Repo: git://git.baserock.org/baserock/baserock/morph.git
Ref: baserock/richardmaw/S8938/null-repo-ref
SHA1: 1f4f3f5782f0c0d53beccf0d4b7be2eed1dd0c35
Land: origin/master
If a repo, ref is null it means the same repo, ref as the parent
morphology.
This means you won't need to specify which repository and branch the
stratum is in any more, it will just use the same one as yours.
This can make edit and petrify simpler.
The branch itself is something thrown together quickly, and I'd prefer
to think of it as the proof of concept, rather than its final state,
but much of the code in question will be rewritten anyway, so it shouldn't
matter if this version is merged.
Richard Maw (4):
Allow building with null repo/ref
Make branch commands work with null refs
Move helper script to scripts for use in cmdtests
Add tests for building and deploying systems with null refs
morphlib/app.py | 8 ++-
morphlib/artifactresolver.py | 8 +--
morphlib/buildcommand.py | 4 +-
morphlib/morph2.py | 3 +-
morphlib/morphset.py | 8 +--
morphlib/plugins/branch_and_merge_new_plugin.py | 7 ++-
scripts/nullify-local-refs | 18 +++++++
tests.build/build-system-with-null-refs.script | 24 +++++++++
tests.build/build-system-with-null-refs.setup | 23 +++++++++
tests.deploy/deploy-with-null-refs.script | 35 +++++++++++++
yarns/branches-workspaces.yarn | 62 +++++++++++++++++++++++
yarns/implementations.yarn | 12 +++++
yarns/morph.shell-lib | 5 ++
13 files changed, 204 insertions(+), 13 deletions(-)
create mode 100755 scripts/nullify-local-refs
create mode 100755 tests.build/build-system-with-null-refs.script
create mode 100755 tests.build/build-system-with-null-refs.setup
create mode 100755 tests.deploy/deploy-with-null-refs.script
--
1.7.10.4
9 years, 4 months
[PATCH] upgrade git
by Jonathan Maw
This patch is not with a usual patch series, since it involves merging
a large number of commits.
head: c7fd06b6411fb04eb4d9acd7f8822a288a50dc17
branch: baserock/jonathanmaw/S9007/upgrade-git
repo: git://git.baserock.org/delta/git
This patch does two things:
1. Merges git v1.8.4 into baserock/morph
2. Convers git.morph's JSON into YAML (because now seemed like a good
time)
9 years, 4 months
Reducing system artifacts
by Lars Wirzenius
Our system artifacts are currently a tar archive of all the files that
go into the system. They can be quite big: a base system for x86-64 is
currently almost 900 megabytes, and a devel system is 1.5 gigabytes.
There's several reasons for this: a primary reason is that we do not
yet split out development tools and libraries from the base system;
work to fix that is going to happen soon.
However, another reason is that we store all the files in a system in
the tar archive. In principle, a system artifact is a union of stratum
artifacts, each of which is a union of chunk artifacts (so actually a
system is a union of chunks). We implement stratum artifacts already
as a list of (versioned!) chunk artifacts, so that a stratum does not
need to store the same files as the chunks already store. We should do
the same for systems.
Unfortunately, a system artifact is not purely a union of stratum
artifacts: it contains some changes to those, most especially we run
ldconfig at system construction time. There's other tools we need to
run then as well, such as constructing cache for gdk-pixbuf loading,
etc.
I've prototyped today a way to store system artifacts as a list of
stratum artifacts plus a binary delta between the pure union and after
running ldconfig. Roughly:
* Create a temporary directory, A.
* Hardlink all the chunks in the system into A.
- this is the same procedure as we do for creating staging areas
already
* Copy A into a new directory B.
- a hardlink copy should do, if we can make the next step safe
* Run ldconfig, and other such things, in B.
* Produce a Trebuchet delta (tbdiff-create) betweewn A and B.
* Create a system artifact consisting of the list of stratum
artifacts plus the Trebuchet delta.
I wrote some very hacky scripts to do this. Result:
916252160 current system artifact
46080 new system artifact
That's about 900 megabytes vs 45 kilobytes. Quite a difference.
When the system artifact is used, it can be reconstructed by unpacking
the stratum artifacts and applying the Trebuchet delta. Again, the
unpacked, cached chunk artifacts can, hopefully, be used to speed up
the process.
If we can make this happen, then:
* during system construction, we don't need to write a gigabyte or two
of data when creating the system directory tree, and then the same
amount again when creating the tar archive of the root filesystem
* during deployment, we don't need to write a gigabyte or two of data
for the configuration phase
There's a lot of details to get right, of course, and some
experimentation and research to make sure the unpacked chunk caches
are safe to hardlink into. (ldconfig, and any other programs that make
changes, will need to make them by writing to a temporary file and
renaming that over the real file, rather than modifying the real file
directly. That's usually the case anyway.)
--
http://www.codethink.co.uk/ http://wiki.baserock.org/ http://www.baserock.com/
9 years, 4 months
[PATCH] Merge Xfce
by Jonathan Maw
This will not a full patch E-mail, since the only thing that affects
the rest of baserock is glib.
I have a branch on baserock:baserock/morphs named
baserock/xfce-build-rebase-2. It has two commits:
* the first one adds in morphs for building an Xfce system
* the second one updates glib to a newer version (because the Xfce
system requires a newer version of glib), and changes all the stratum
refs to point to the changed morphologies.
I have performed the following tests:
* Build a devel-system-x86_64-generic
* Deploy devel-system-x86_64-generic and test that a programme that
uses glib still works - in this case, I started the
node-startup-controller and checked that it did not raise errors.
* Build a devel-system-armv7b-highbank
* Deploy devel-system-armv7b-highbank and test the
node-startup-controller didn't break
9 years, 4 months
[PATCH 0/2] RT189: Baserock 11 should make disk-size and system-kind fields errors
by Daniel Firth
repo: git.baserock.org/baserock/baserock/morphs
branch: danielfirth/RT189
sha1: 5e0ad6ad29fa776088241828580d58b7d91e9fd1
Dan Firth (2):
Dropped support for 'disk-size' from system morphology files'
Dropped support for 'system-kind' from system morphology files
morphlib/morph2.py | 21 +------------
morphlib/morph2_tests.py | 36 ----------------------
morphlib/morphloader.py | 16 ----------
morphlib/morphloader_tests.py | 18 -----------
morphlib/morphologyfactory.py | 13 --------
morphlib/morphologyfactory_tests.py | 8 -----
morphlib/plugins/branch_and_merge_plugin.py | 3 --
scripts/setup-3rd-party-strata | 2 --
tests.as-root/archless-system-fails.script | 3 +-
.../metadata-includes-morph-version.setup | 1 -
tests.as-root/metadata-includes-repo-alias.setup | 1 -
.../rootfs-tarball-builds-rootfs-and-kernel.script | 3 --
tests.as-root/setup | 4 ---
tests.as-root/system-overlap.script | 2 --
tests.as-root/tarball-image-is-sensible.setup | 1 -
tests.branching.disabled/workflow-petrify.stdout | 6 ----
.../edit-updates-stratum-build-depends.stdout | 28 ++---------------
tests.branching/edit-updates-stratum.stdout | 5 ++-
tests.branching/setup | 1 -
tests.branching/tag-creates-commit-and-tag.stdout | 11 +++----
tests.branching/tag-tag-works-as-expected.stdout | 11 +++----
.../tag-works-with-multiple-morphs-repos.script | 2 --
.../tag-works-with-multiple-morphs-repos.stdout | 16 +++++-----
tests.build/setup | 2 --
tests.deploy/setup | 2 --
tests.merging/setup | 2 --
tests/setup | 2 --
tests/show-dependencies.setup | 1 -
28 files changed, 24 insertions(+), 197 deletions(-)
--
1.8.1.2
9 years, 4 months
[PATCH] RT219: "morph deploy" outside morphs dir and install-files
by Daniel Firth
repo: git.baserock.org/baserock/baserock/morph
branch: danielfirth/RT219
sha1 :88c72bf91f357e453e7ca23d3890d3412c5bd799
Dan Firth (1):
Changed deployment tests to look in the test morphs directory for
deployment images
Daniel Firth (1):
Morph now executes extensions in the repository containing the
morphologies
morphlib/plugins/deploy_plugin.py | 2 +-
tests.deploy/deploy-cluster.script | 12 ++++++------
tests.deploy/deploy-rawdisk.script | 4 +++-
3 files changed, 10 insertions(+), 8 deletions(-)
--
1.8.1.2
9 years, 4 months
[PATCH 0/2] Add cloudinit support.
by Pedro Alvarez
Repo: git://git.baserock.org/baserock/baserock/morphs.git
Branch: baserock/pedroalvarez/cloudinit
Sha1: 9c2f8a3046b684c7405a5bb19b54371534c96ffd
The first patch is for move pyyaml from tools to core, we need pyyaml
in core to be able to build any system (base o devel) with cloudinit.
The second patch is for add cloudinit stratum and two base system to
use it. Cloudinit is needed to make the system suitable into an OpenStack
cloud, giving to the system the capacity of execute scripts at boot time.
Pedro Alvarez (2):
Move pyyaml from 'tools' to 'core'.
Add cloudinit-support Stratum morphology.
base-system-x86_32-openstack.morph | 29 ++++++++++++++++++++++
base-system-x86_64-openstack.morph | 29 ++++++++++++++++++++++
cloudinit-support.morph | 48 ++++++++++++++++++++++++++++++++++++
core.morph | 10 ++++++++
cross-bootstrap.morph | 13 +---------
tools.morph | 13 +---------
6 files changed, 118 insertions(+), 24 deletions(-)
create mode 100644 base-system-x86_32-openstack.morph
create mode 100644 base-system-x86_64-openstack.morph
create mode 100644 cloudinit-support.morph
--
1.7.10.4
9 years, 4 months
[PATCH 0/4] Refactor branch-from-image
by Richard Maw
Repo: git://git.baserock.org/baserock/baserock/morph.git
Ref: baserock/richardmaw/S8847/branch-from-image-v2
SHA1: d540b508198c298a02474c97221713fba4d3cde0
Land: origin/master
There's a scary use of contextlib to make a context manager from a
generator function, which:
a) may not be the clearest approach, though I think it's a good one
b) possibly should be defined somewhere other than the branch and
merge plugin
There's a class that abstracts away the /baserock metadata directory
access to being just like a dict, which may need a better name.
Richard Maw (4):
b&m: Add system branch initializing context manager
b&m: checkout and branch use context manager
morphlib: Add SystemMetadataDir class
B&M: refactor branch-from-image
morphlib/__init__.py | 1 +
morphlib/plugins/branch_and_merge_new_plugin.py | 182 ++++++++++++++++++-----
morphlib/plugins/branch_and_merge_plugin.py | 12 +-
morphlib/systemmetadatadir.py | 63 ++++++++
morphlib/systemmetadatadir_tests.py | 66 ++++++++
tests.as-root/branch-from-image-works.script | 3 +-
6 files changed, 276 insertions(+), 51 deletions(-)
create mode 100644 morphlib/systemmetadatadir.py
create mode 100644 morphlib/systemmetadatadir_tests.py
--
1.7.10.4
9 years, 4 months
