** This branch depends on the samthursfield/morph-update-improvements
I sent a patch for this script a while back and it received no verdict.
Here it is again, with a couple of changes:
- commands to convert a list of files between JSON and YAML.
- removed one command which was more or less a fragile hack
The to-yaml command takes a list of .morph files and outputs
to .morph.yaml files. The to-json command does the reverse.
Sam Thursfield (3):
Allow forcing output format (useful for edit-morph script)
scripts/edit-morph: Add to-yaml and to-json commands
morphlib/morph2.py | 12 ++-
scripts/edit-morph | 205 ++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 213 insertions(+), 4 deletions(-)
create mode 100755 scripts/edit-morph
Previously adding fields to morphologies involved adding a hack to
decide whether to write it back out based on whether it was set to
the default. This was always imperfect. This branch makes the code
handle defaults automatically by tracking what was set to its default
value on load rather than being specified explicitly on the morphology
and then only writing out those values if they have changed.
This doesn't do much for in-place editing of YAML morphologies.
Currently we will mostly destroy formatting (possibly even comments)
of YAML morphologies with 'morph edit', 'morph merge' etc. This will
be quite difficult to get right, but the current API allows consulting
the original text when writing out the update which makes it at least
possible to work on preserving formatting and comments in the YAML
Sam Thursfield (4):
Catch only JSON parse exceptions, not Exception
Detect and abort on empty morphologies
Handle morphs that are not dicts (By failing)
Make writing morphologies back out properly non-destructive
morphlib/morph2.py | 182 +++++++++++++++++++--------
morphlib/morph2_tests.py | 122 +++++++++++-------
morphlib/plugins/branch_and_merge_plugin.py | 102 +++++++--------
3 files changed, 258 insertions(+), 148 deletions(-)
Background: Compressing build artifacts on ARM is currently taking
a lot of time. We were discussing some options to improve that,
including switching from gzip to lz4. Below is results from a
* "busybox gzip" is busybox's own gzip
* gzip-1.2.4/gzip is the "real" gzip, built locally
* pygzip is a simple wrapper around the Python gzip.GzipFile class
that reads from stdin and writes the compressed result to stdout;
it does the compression in the Python process, but calls to the
* lz4-read-only/lz4demo is the lz4 demo program
* pylz4 is like pygzip, except using a wrapper like GzipFile to
wrap the lz4 Python bindings in a nicer API
Output from the script:
+ time busybox gzip -1
real 1m 28.82s
user 1m 28.58s
sys 0m 0.24s
+ time gzip-1.2.4/gzip -1
real 0m 6.67s
user 0m 6.43s
sys 0m 0.24s
+ time ./pygzip
real 0m 10.51s
user 0m 10.23s
sys 0m 0.28s
+ time lz4-read-only/lz4demo stdin stdout
*** Compression CLI using LZ4 algorithm , by Yann Collet (Feb 15 2013) ***
Using stdin for input
Using stdout for output
Compressed 50862080 bytes into 22655618 bytes ==> 44.54%
Done in 1.26 s ==> 38.50 MB/s
real 0m 1.26s
user 0m 1.06s
sys 0m 0.20s
+ PYTHONPATH=python-lz4/src time ./pylz4
real 0m 6.87s
user 0m 4.31s
sys 0m 2.47s
* busybox gzip is really slow; I assume it is ignoring the -1 option
* Python adds around 4 seconds of overhead over real gzip -1
* lz4demo is blindingly fast, about five seconds faster than real gzip -1
* Python adds around 5 seconds of overhead over lz4demo
This changes how build ordering is done: we build a directed acyclic
graph of build dependencies in any case, and can build things in
the graph order, rather than computing, in a complicated way, a
topological ordering. This allows us to have some less code to
Lars Wirzenius (3):
Replace builder order graph with just a single artifact
Stop computing the old build ordering
Rewrite show-dependencies to work without BuildOrder
morphlib/__init__.py | 1 -
morphlib/artifact.py | 23 +++-
morphlib/buildcommand.py | 49 +++++----
morphlib/buildorder.py | 117 --------------------
morphlib/buildorder_tests.py | 141 ------------------------
morphlib/plugins/deploy_plugin.py | 3 +-
morphlib/plugins/graphing_plugin.py | 30 ++---
morphlib/plugins/show_dependencies_plugin.py | 22 ++--
tests/show-dependencies.stdout | 153 ++++++++++----------------
9 files changed, 132 insertions(+), 407 deletions(-)
delete mode 100644 morphlib/buildorder.py
delete mode 100644 morphlib/buildorder_tests.py
One issue we have in baserock is that the hack-test cycle is often
longer than we would like. As an extreme use-case, consider the following:
Alice wants to add some new linux api features and the corresponding
support in systemd.
In the current design, Alice will be rebuilding her whole system every
change, and will be forced to work outside of the baserock system.
To solve this problem, I propose 'morph hack/unhack'
morph hack [chunk1] [chunk2] ...
this would create a new top-level stratum that depends on all other
strata in the system. This strata would be flagged as a hack strata, and
the chunks passed on the command line would be copied from their
original strata into the new stratum.
There may only ever be one 'hack' stratum, and when a hack stratum is in
existance, the artefact construction process will skip chunks that are
in hack stratum for other strata. Any further calls to morph hack simply
add more chunks to the current hack strata.
Any branch with a hack strata would be ineligible for merging.
The end result should be that only the components in the hack strata
will get rebuilt and any build dependencies of these will not get
tracked while the developer is 'hacking'.
When the developer is happy with their work, they call
Which pulls the changes in the appropriate chunks back to where they
should be in their original strata, ready for a full build to confirm
the system is still fully buildable.
This functionality would also be good for smoke testing checkins in a
continuous integration system.
Opinions? Does the explanation make sense?
Rob Taylor, CTO, Codethink Ltd.
Office: +44 161 236 5575
Cell: +44 7891 533856
This series of patches add a morphology to build a highbank ARM
Javier Jardón (5):
Add morphologies to build a highbank armv7 system
highbank: Adapt to new build-essential changes
base-system-armv7-highbank.morph: Adapt to the changes in
Add devel-system-armv7-highbank morphology
highbank: Change refs in preparation to merge
base-system-armv7-highbank.morph | 30 ++++++++++++++++++++++++++++++
bsp-armv7-highbank.morph | 20 ++++++++++++++++++++
devel-system-armv7-highbank.morph | 35 +++++++++++++++++++++++++++++++++++
3 files changed, 85 insertions(+)
create mode 100644 base-system-armv7-highbank.morph
create mode 100644 bsp-armv7-highbank.morph
create mode 100644 devel-system-armv7-highbank.morph
On 13-02-13 15:30:50, Rob Taylor wrote:
> On 13/02/13 15:28, Jannis Pohlmann wrote:
> > Hey,
> > On 13-02-13 15:20:57, Richard Maw wrote:
> >> I'm not sure calling it `morph hack` is a good idea, since software
> >> developers who are less aware of Open Source development may not
> >> recognise this definition of hack.
> > small anecdote: one of my professors used to (and I guess still does)
> > rant about "hackers", who in his view are bad programmers who merely
> > hack away and don't think about what they are doing. Of course we all
> > know that the terms "hacker" and "hacking" are a bit more universal. ;)
> > I agree with Richard though, the phrase "a hack" might float around in
> > people's minds if they see "morph hack", so perhaps we should choose
> > something more serious.
> The point, is if you have a live 'morph hack', is *IS* a hack. Its
> something you will need to tidy up later, and its something you wouldn't
> hand to someone else.
it may or may be a hack, I guess that really depends. Perhaps the
question is: should it be a hack, i.e., should we really provide people
with a hacky alternative to avoid branch + edit + merge?
I agree full system rebuilds are annoying and slow things down too much
and it's a good idea to provide people with a fast way to do it. But
perhaps its better to optimise the clean (unfinished) workflow we have
rather than inventing a second one. Or, if the one we have is flawed in
general, replace it with a new concept, if necessary.
Ok, so that's my basic conceptual input, beyond pure naming
Senior Software Developer
The implementation is untidy, as I'm unfamiliar with this code.
The refactor to move this logic into a library can't come soon enough.
I may have missed the point with make_available(), but I found I could
not use it since it would stomp over the changes made to the strata.
Patch 2's all_strata may not be logically correct, since last time
I checked it was possible to build a system where the strata could
build-depend on strata that aren't included in the final system.
Richard Maw (2):
Amend tests to trigger failure to recursively update strata
branch-and-merge: Recursively edit strata
morphlib/plugins/branch_and_merge_plugin.py | 109 +++++++++++++-------
.../build-handles-stratum-build-depends.script | 6 +-
.../building-a-system-branch-works-anywhere.script | 2 +-
tests.as-root/disk-builds-rootfs-and-kernel.stdout | 6 +
tests.as-root/setup | 34 ++++++
.../syslinux-disk-builds-rootfs-and-kernel.stdout | 6 +
6 files changed, 122 insertions(+), 41 deletions(-)