Doesn't do silly things with the doc string
Raise a cliapp.AppException if manifest is invalid
Richard Ipsum (1):
Add optional overwrite option
morphlib/exts/install-files.configure | 39 ++++++++++++++++++++++++---------
1 file changed, 29 insertions(+), 10 deletions(-)
Now matching with regex rather than splitting on whitespace.
I've not used the ?P<tag> style, I think expressions look clearer without them.
I don't think it would be too difficult to write a test for this,
but there isn't time for that just now.
Richard Ipsum (1):
Add optional overwrite option
morphlib/exts/install-files.configure | 30 +++++++++++++++++++-----------
1 file changed, 19 insertions(+), 11 deletions(-)
Merge into: baserock/richardmaw/S9337/split-strata-rebase-v2
Morph's unit tests have far too much copy and paste code in them. This
is a first attempt at cleaning up morphloader_tests.
I was going to moan on Richard Maw's stratum splitting patch set about how
verbose the unit tests were in that patch, but it seemed more helpful to clean
up the whole file so that the code remains consistent.
Sam Thursfield (3):
Remove duplication in morphloader_tests.py, part 1
Move morphloader errors into separate module for easy access
Add validation-specific assert() helper functions to morphloader_tests
morphlib/morphloader.py | 144 +-----------
morphlib/morphloader_errors.py | 163 +++++++++++++
morphlib/morphloader_tests.py | 503 ++++++++++++++++-------------------------
without-test-modules | 2 +
4 files changed, 364 insertions(+), 448 deletions(-)
create mode 100644 morphlib/morphloader_errors.py
This adds the ability to split strata into multiple artifacts and have
systems select a subset of the produced strata for inclusion into the
There exists a TODO in a commit message as a reminder that it's an
open question what to do with the Morphology.builds_artifacts, as such,
this version of the patch series is not expected to be merged.
This series has been rebased to try and provide a working series for
as long as possible, but it is broken between "Generate dependencies
from split rules" and "Split Stratum artifacts according to new rules",
since artifacts are constructed with the old default rules, while the
dependencies are generated with the new ones.
It would be possible to keep it working by:
* itroducing the new split_rules methods with the old behaviour of
defaulting to 1 artifact
* after all the infrastructure changes, change the default rules and
update the tests
* merge commits that change things with their test-suite changes,
which were kept separate for readability.
However, we out test-suite passing policy only requires it to work during
merge commits, so this extra work probably isn't necessary.
Empty artifacts are created, since you can't know at dependency generation
time whether the artifact will contain anything. This causes a lot of
artifact metadata files to clutter the final system's /baserock directory.
As a consequence of dependencies being on the artifacts, the cache keys
for different artifacts of the same stratum source are different. This
may, or may not be a problem, since logically it makes a lot of sense
for stratum artifacts to only depend on the chunks they need, but it's
a change in expectation, that you can remove every artifact produced by
a source, by removing everything with the same cache key.
This could confuse the GC plugin, since if you have a system that only
uses a subset of the strata produced, then over time, you could gc the
artifacts that aren't used, but when you come to use a system that is
the same except for which strata it includes, it will find it doesn't
have all of them.
I have not yet tested how it will behave in this case. For strata it's
safe to construct the artifacts independently of each other, but may cause
confusion with the Builder only returning a subset of the artifacts it
Richard Maw (20):
yarns: Remove redundant repo in IMPLEMENTS THEN
yarns: Make test chunk create a bunch of files
yarns: Add a deploy test
unit tests: Fix invalid morphologies
Handle attempts to build a chunk/stratum more gracefully.
BuildCommand.find_root_artifact: Simplify root artifact location
morph2.Morphology: add trivial .get method
source: Memoize artifact creation in Source object
ArtifactResolver: Use Source.get_artifact
Rename chunk 'chunks' field to 'products'
MorphologyLoader: Validate new fields
MorphologyFactory: validate new fields using loader code
Add split rules to sources
CacheKeyComputer: Include split rules in computation
ArtifactResolver: Generate dependencies from split rules
Split chunk morphologies according to new rules
Unit tests: Fix up for changes to chunk construct api
Split Stratum artifacts according to new rules
cmdtests: Take into account new constructed artifacts
yarns: Add tests for building systems with splitting
morphlib/__init__.py | 3 +-
morphlib/artifact_tests.py | 4 +-
morphlib/artifactresolver.py | 204 +--
morphlib/artifactresolver_tests.py | 592 ++-----
morphlib/artifactsplitrule.py | 206 +++
morphlib/bins.py | 70 +-
morphlib/bins_tests.py | 39 +-
morphlib/buildcommand.py | 28 +-
morphlib/builder2.py | 146 +-
morphlib/cachekeycomputer.py | 4 +-
morphlib/cachekeycomputer_tests.py | 16 +-
morphlib/localartifactcache_tests.py | 4 +-
morphlib/morph2.py | 13 +-
morphlib/morph2_tests.py | 6 +-
morphlib/morphloader.py | 99 +-
morphlib/morphloader_tests.py | 94 +-
morphlib/morphologyfactory.py | 9 +-
morphlib/morphologyfactory_tests.py | 16 +-
morphlib/remoteartifactcache_tests.py | 4 +-
morphlib/source.py | 35 +-
tests.as-root/branch-from-image-works.script | 4 +-
tests.as-root/metadata-includes-repo-alias.script | 10 +-
...run-in-artifact-with-different-artifacts.script | 6 +-
...run-in-artifact-with-different-artifacts.stderr | 2 +-
...run-in-artifact-with-different-artifacts.stdout | 32 +-
tests.as-root/system-overlap.stdout | 2 +-
tests.as-root/tarball-image-is-sensible.stdout | 30 +-
tests.build/bootstrap-mode.script | 15 +-
tests.build/bootstrap-mode.stdout | 11 +-
tests.build/build-stratum-with-submodules.script | 4 +-
tests.build/build-stratum-with-submodules.stdout | 4 -
tests.build/build-system-autotools.script | 8 +-
tests.build/build-system-autotools.stdout | 5 -
tests.build/build-system-cmake.script | 8 +-
tests.build/build-system-cmake.stdout | 6 -
tests.build/build-system-cpan.script | 8 +-
tests.build/build-system-cpan.stdout | 1 -
tests.build/build-system-python-distutils.script | 8 +-
tests.build/build-system-python-distutils.stdout | 4 -
tests.build/build-system.script | 4 +-
tests.build/build-system.stdout | 4 -
tests.build/morphless-chunks.script | 8 +-
tests.build/morphless-chunks.stdout | 5 -
tests.build/prefix.script | 6 +-
tests.build/rebuild-cached-stratum.script | 16 +-
tests.build/rebuild-cached-stratum.stdout | 26 +-
tests.build/stratum-overlap-writes-overlap.script | 6 +-
tests.build/stratum-overlap-writes-overlap.stdout | 4 +-
tests/show-dependencies.stdout | 1776 ++++++++++++++++++--
without-test-modules | 1 +
yarns/building.yarn | 2 +-
yarns/deployment.yarn | 9 +
yarns/implementations.yarn | 138 +-
yarns/regression.yarn | 2 +-
yarns/splitting.yarn | 128 ++
55 files changed, 2939 insertions(+), 956 deletions(-)
create mode 100644 morphlib/artifactsplitrule.py
create mode 100644 yarns/splitting.yarn
Baserock 12 is released
The [Baserock team] and [Codethink Limited] are proud to announce
version 12 of the Baserock system.
Baserock is a toolset and development methodology for developing
embedded and appliance Linux systems.
What's new in this version?
We updated the version of the cmdtest component used in Baserock. This
brings in a newer version of the yarn testing tool which has allowed us
to improve the Morph test suites to be cleaner and thus better.
SSH configuration extension
The ssh configuration extension's functionality has been replaced by
(non-identical, but equivalent) functionality in the install-files
configuration extension. Following the principle of simpler-is-better
we have therefore removed the ssh configuration extension and we
encourage users to migrate to install-files.
As an example of how to use this extension, add the environment variable
INSTALL_FILES to the deploy stanza for a system in the cluster
- morph: base-system-x86_64-generic
Then create the 'foo-dir' directory:
Where id_rsa and id_rsa.pub are the ssh key pair the root user on the
deployed system should have. Then create foo-dir/manifest:
0040755 0 0 /root
0040700 0 0 /root/.ssh
0100600 0 0 /root/.ssh/id_rsa
0100600 0 0 /root/.ssh/id_rsa.pub
After this, install-files will put the files in the right places on
the deployed system, with the right permissions.
Refactoring morph subcommands
Work began on refactoring the 'build' command, but the new code will
not be available in Baserock until the next cycle.
Minor changes to morph
* A number of bugs have been fixed during the Baserock 12 cycle,
* prevent 'edit' subcommand being used without a system as argument.
* prevent building uncommmitted systems.
* deploy a non-cluster morphology now displays an error message.
* prevent cross-building.
* morph can now build when checking out a tag.
* We have improved the morph scenario test suite by:
* adding new scenarios, including regression scenarios for fixed
bugs, and replacing some of the old cmdtest based suite with
* reworking some of the statement language to be more consistent,
* making the voice of the scenarios slightly more active.
Changes to morphologies
Reducing redundancy in build dependencies
Since at this time, build-depending on a stratum implicitly
build-depends on the build-dependencies of that stratum, we have
simplified the build-depends sections of the standard stratum
morphologies. This should make it easier to understand the relationship
between strata involved in building systems.
* pyfilesystem, kexec-tools have been added to the 'tools' stratum.
* 'six' has been moved from 'glanceclient' to the 'tools' stratum.
* boto has been added to 'tools' to finish the integration of
cloud-init in Baserock. There is a short guide about how to deploy to
OpenStack on the [developing with Baserock] wiki page.
New systems and strata
We have added a ceph services system which provides the daemons
necessary for the Ceph object, block and file storage services.
These daemons are contained within a ceph-service stratum which can be
included into your systems if you want them to be able to serve Ceph
This new system is an example and is not directly supported at this
time. However please do try it out if Ceph is of interest to you, and
let us know how you get on.
How do I get started?
You can find a [quick start] guide on the [Baserock wiki] and also
a short guide on [developing with Baserock] which follows on from
the quick start guide and shows you how to get to the point of proving
you can build Baserock within Baserock.
>From that point on, your imagination is the limit. You can follow our
development in the [Git repositories] we publish.
How do I get in contact?
The Baserock project has an [IRC channel] and [mailing list] for
developers to gather and discuss anything associated with
Baserock. It is strongly recommended that you use the IRC and
lists to contact the team for anything associated with the public
development of Baserock. We also have a mailing list for
[announcements] which will be notified of any new releases or big
developments in Baserock.
If you manage to find a bug in Baserock, we'd like to hear
from you. You can find our [bug reporting guidelines] on the
Baserock wiki and we will do our best to help.
We hope you enjoy experimenting with Baserock and look forward to
hearing about any cool things you do with our work.
[Codethink limited]: http://www.codethink.co.uk/
[Baserock wiki]: http://wiki.baserock.org/
[Baserock team]: http://wiki.baserock.org/team/
[Git repositories]: http://git.baserock.org/cgi-bin/cgit.cgi
[IRC Channel]: http://wiki.baserock.org/irc-channel/
[mailing list]: http://wiki.baserock.org/mailinglist/
[download page]: http://wiki.baserock.org/download/
[quick start]: http://wiki.baserock.org/quick-start/
[developing with Baserock]: http://wiki.baserock.org/devel-with/
[bug reporting guidelines]: http://wiki.baserock.org/bug-reporting/
[Baserock GENIVI Baseline]:
Ben Brown (1):
Add function to raise an error when a given chunk ref is not a string
morphlib/morphloader.py | 20 +++++++++++++++++++-
morphlib/morphloader_tests.py | 20 +++++++++++++++++++-
2 files changed, 38 insertions(+), 2 deletions(-)
From: Richard Ipsum <richard.ipsum(a)codethink.co.uk>
This might be useful if we want to replace existing config files
for certain deployments.
Richard Ipsum (1):
Add optional overwrite option
morphlib/exts/install-files.configure | 19 ++++++++++---------
1 file changed, 10 insertions(+), 9 deletions(-)