[PATCH 0/2] Add the ability of including system integration commands.
by Pedro Alvarez
Repo: baserock:morph
Ref: baserock/pedroalvarez/system-integration3
Sha1: 20d6028c8bc62ca00a9af52ae5fada736ce673dc
Card: S10680
The purpose of this patch is to add the ability of including system
integration commands in the chunk morphologies.
An example of how to add system integration commands would be:
name: linux
kind: chunk
build-commands:
...
install-commands:
...
system-integration-commands:
linux-libs: |
depmod -a
touch /baserock/FILE
for FOLDER in $(ls /)
do
touch "$FOLDER"/.somefile
done
Pedro Alvarez (2):
Generate scripts with the system-integration-commands of chunk
morphologies.
Run the system integration commands when building.
morphlib/builder2.py | 84 ++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 81 insertions(+), 3 deletions(-)
--
1.7.10.4
9 years, 6 months
[RFC] Moving common build system definitions
by Richard Maw
This is a write-up of a discussion Rob Taylor, Daniel Silverstone,
Lars Wirzenius and I had last week.
One goal we've got for the future is to have less magic hard-coded
into morph.
One of these features is to avoid needing to write a chunk morphology,
when the project we're trying to build has a common build system.
For example, if there's a `autogen` or `configure` script, we can guess
that this is an autotools project, so we can build it by running ./autogen
if it's there; `autoreconf -fiv` if we still don't have a `./configure`
script, then `./configure --prefix "$PREFIX"` to configure it; `make`
to build it, and `make DESTDIR="$DESTDIR" install` to install it.
We initially devised a complicated mechanism, where we define a python
class, that provides lists of commands, and a `used_by_project(file_list)`
method, which determines whether this project uses this build system by
inspecting the list of files.
This is over-complicated, since we, with only one exception, checked
against a static list of files, and if any were in there, it was that
build system.
The exception was QMake checking that any of the files ended with '.pro'.
The order of these detections is hard-coded, and the first match wins.
The contents of the build system definitions do not make it into the
cache key, so if we change them, we have to change a variable for chunk
morphology version, which causes a full rebuild of everything, even if
it does not use the changed build system.
The the new design
==================
Since our `used_by_project` functions were fairly trivial in practise,
we can get away with a declarative format.
I suggest a yaml file, something like this:
kind: build-system
# not really a new morphology type, just to distinguish its type
# when it may be in with morphologies
name: autotools
indicators: # list of regular expressions
- autogen(.sh)?
- configure
- configure.ac
- configure(.in)+ # yes really
configure-commands:
- |
export NOCONFIGURE=1 # some autogen scripts run ./configure themselves
if [ -e autogen ]; then
./autogen
elif [ -e autogen.sh ]; then
./autogen.sh
elif [ ! -e ./configure ]; then
autoreconf -fiv #fallback if no autogen
fi
./configure --prefix "$PREFIX"
build-commands:
- make
install-commands:
- make DESTDIR="$DESTDIR" install
The odd-ball QMake would look like this:
kind: build-system
name: qmake
indicators:
- ".*\.pro"
configure-commands:
- qmake -makefile
build-commands:
- make
install-commands:
- make INSTALL_ROOT="$DESTDIR" install
These yaml files will be located in either the morphologies repository,
or morphlib.
Every build system used by Baserock will be put in morphlib, so if you
have another morphologies repository, you don't need to copy the build
system definitions.
Putting a new build-system definition in to the morphologies repository
allows handling of overly specific build-system, or a new one that hasn't
yet been merged into morph to be used for new chunks.
definitions in the morphologies repository can override the version
in morphlib. This allows a transition path, if we decide to change or
remove the versions in morphlib.
There is no specified order for matches to be run; all of them are,
and if multiple match, then an Exception is raised.
The contents of the build system files are not directly cached, to prevent
spurious rebuilds when one changes its indicators, or a command that is
overridden by the morphology itself.
Instead, the morphologies get their command list included in the cache
key after the build system is determined, and before its cache key
is determined.
Dissenting opinion
==================
No more build-systems
---------------------
One of the main reasons for autodetecting the build-system was that we
could avoid creating morphology files for everything then.
Given these will eventually be migrated away from individual chunk's
repositories, this is less of a problem, since they are easier to manage
in the same place.
Chunks containing repo/ref
--------------------------
Another idea we've had was to move the repo and ref fields for chunks
from the stratum morphology into the chunk morphology. At this point,
auto-detection no-longer allows us to not have a chunk morphology.
We would still be able to avoid specifying any commands if autodetection
works it out, but it's fairly trivial to add `build-system: autotools`,
and it's potentially clearer what is going on in this case.
Of course, full clarity has all the commands in every morphology, but
this is duplication of code, so suffers from needing to keep it in sync.
Put build-systems with what provides them
-----------------------------------------
We can only use the python build-system after we have python, so we
could make the python morphology define the build-system, so if we're
building a system without python, we don't need to care if it exists.
9 years, 6 months
[PATCH 0/6] Fix the test suite on x86_32
by Richard Maw
Repo: git://git.baserock.org/baserock/baserock/morph.git
Ref: baserock/richardmaw/S10630-32-bit-tests
SHA1: a0cf62b0152b68cd85ae69b47edc2745044e2353
Land: origin/master
This is a bunch of mostly low-effort changes to get the test suite going
on 32-bit x86.
It also contains my changes to the test runner to allow me to focus
on individual parts of the test suite. The old behaviour is preserved,
but you can also specify only running one test.
Richard Maw (6):
check script: allow fine-grained control of which tests are run
Use morph print-architecture to generate morphologies
Raise the disk image size for deployment test
Remove morph tag cmdtests
Remove bootstrap-mode cmdtest
Allow run-in-artifact cmdtest to pass on x86_32
check | 99 ++++++++++++++++++----
.../metadata-includes-morph-version.setup | 2 +-
tests.as-root/metadata-includes-repo-alias.setup | 2 +-
.../rootfs-tarball-builds-rootfs-and-kernel.script | 2 +-
...run-in-artifact-with-different-artifacts.script | 2 +-
...run-in-artifact-with-different-artifacts.stderr | 1 -
tests.as-root/setup | 4 +-
tests.as-root/system-overlap.script | 2 +-
tests.as-root/tarball-image-is-sensible.setup | 2 +-
tests.branching/setup | 2 +-
tests.branching/tag-creates-commit-and-tag.script | 39 ---------
tests.branching/tag-creates-commit-and-tag.stdout | 44 ----------
tests.branching/tag-tag-works-as-expected.script | 46 ----------
tests.branching/tag-tag-works-as-expected.stdout | 48 -----------
tests.build/bootstrap-mode.script | 43 ----------
tests.build/bootstrap-mode.stdout | 13 ---
tests.build/setup | 2 +-
tests.deploy/setup | 4 +-
tests.merging/setup | 2 +-
tests/show-dependencies.setup | 2 +-
yarns/deployment.yarn | 2 +-
21 files changed, 97 insertions(+), 266 deletions(-)
delete mode 100644 tests.as-root/run-in-artifact-with-different-artifacts.stderr
delete mode 100755 tests.branching/tag-creates-commit-and-tag.script
delete mode 100644 tests.branching/tag-creates-commit-and-tag.stdout
delete mode 100755 tests.branching/tag-tag-works-as-expected.script
delete mode 100644 tests.branching/tag-tag-works-as-expected.stdout
delete mode 100755 tests.build/bootstrap-mode.script
delete mode 100644 tests.build/bootstrap-mode.stdout
--
1.8.5.rc2
9 years, 6 months
[PATCHv2 00/12] [morph] Allow nested deployments
by Richard Maw
Repo: git://git.baserock.org/baserock/baserock/morph.git
Ref: baserock/richardmaw/S10630-multi-deploy-v2
SHA1: 50e6195509600058327e73e29a2ea9cd0879733e
Land: origin/master
Changes since last version:
1. It's called nested deployment, rather than recursive
2. All yarn tests now use the new vocabulary for creating clusters
3. There's two new patches at the end to make it output which system
and deployment the status message is for.
It makes for very long status lines, since it includes the full
system name.
It's at the end so it's easier to drop those changes if they're
deemed unworthy.
Things forgotten:
1. Add to the deployment command's docstring how to specify nested
deployments?
Richard Maw (12):
tests: Allow running ./check on 32-bit x86
yarns: set deployment name based on system name
Remove ugly loading of old b&m plugin
Patch buildcommand once, rather that once per system
Move tempdir creation out of the loop
Move deploy logic into multiple methods
Add the ability to do nested deployments
scripts: Add helper commands for altering cluster morphologies
Add sysroot write extension
Test nested deployments
deploy: Make extension output display which it is
deploy: Set status prefix to show which deployment the status is for
morphlib/exts/sysroot.write | 29 ++++
morphlib/plugins/deploy_plugin.py | 181 +++++++++++++--------
scripts/edit-morph | 108 ++++++++++--
.../metadata-includes-morph-version.setup | 2 +-
tests.as-root/metadata-includes-repo-alias.setup | 2 +-
.../rootfs-tarball-builds-rootfs-and-kernel.script | 4 +-
...run-in-artifact-with-different-artifacts.stderr | 2 +-
tests.as-root/setup | 4 +-
tests.as-root/system-overlap.script | 2 +-
tests.as-root/tarball-image-is-sensible.setup | 2 +-
tests.branching/setup | 2 +-
tests.branching/tag-creates-commit-and-tag.stdout | 6 +-
tests.branching/tag-tag-works-as-expected.stdout | 6 +-
tests.build/bootstrap-mode.stdout | 6 +-
tests.build/setup | 2 +-
tests.deploy/setup | 4 +-
tests.merging/setup | 4 +-
tests/show-dependencies.setup | 2 +-
yarns/deployment.yarn | 71 +++++++-
yarns/implementations.yarn | 75 ++++++---
yarns/regression.yarn | 7 +-
yarns/splitting.yarn | 7 +-
22 files changed, 390 insertions(+), 138 deletions(-)
create mode 100755 morphlib/exts/sysroot.write
--
1.8.5.rc2
9 years, 6 months
[PATCH] Upgrades of manually-deployed systems
by Sam Thursfield
Repository: git://git.baserock.org/baserock/baserock/definitions
Ref: baserock/sam/devel-upgrades
Sha1: 60c0d3e418e7dd8b2029f9c2ea0e47b2f2ac335d
Presumably users will be interested in self-deploying upgrades. Seems to me the
best way format is to provide a cluster morph to let them do so.
We should be very clear that existing Baserock 13 systems cannot be upgraded
in this way (currently you get an unhelpful error about 'menu.c32'; they also
lack the /state subvolumes that allow sharing user data between system
versions).
Sam Thursfield (1):
Add example cluster morph for upgrading an existing devel system
upgrade-devel.morph | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
create mode 100644 upgrade-devel.morph
--
1.8.5.3
9 years, 6 months
[PATCH 00/10] [Morph] Support recursive deployments
by Richard Maw
Repo: git://git.baserock.org/baserock/baserock/morph.git
Ref: baserock/richardmaw/S10630-multi-deploy
SHA1: 8dbe248aafb33143367e3e05640c7e12ad59c05c
Land: origin/master
This lets you do recursive deployments by specifying subsystems in the
system list.
For now, this is required to build the SDK of a target system, since we
have the sysroot, but we also need the libraries, and barring a rework
of the core build code to allow specifying:
1. strata in strata
2. strata of a different architecture
3. strata extracted at another path
including the system at deploy-time is our best bet.
This is not just a hack-job to get us there, rather than doing it
properly, since it's not a fundamentally bad approach, and we have other
uses of recursive deployments, such as:
1. Baserock installer CD/USB
2. Deploying a system with containerised applications
3. Deploying an NFS server with nfsroots already prepared
4. Deploying a VM host with its VMs pre-created at deploy time
5. Any combination of the above
Possibly worth doing before this is merged:
1. Make other yarns that create cluster morphologies use the new verbs
2. Fix the deploy output to list the deployment name of each command
it runs
3. Simplify the cluster morphology layout as I described in my RFC email,
I didn't do it for this as it wasn't necessary to get recursive
deployments working, but it ends up with a weird layout, and the
scripts for modifying cluster morphologies are a bit more fragile.
Richard Maw (10):
tests: Allow running ./check on 32-bit x86
yarns: set deployment name based on system name
Remove ugly loading of old b&m plugin
Patch buildcommand once, rather that once per system
Move tempdir creation out of the loop
Move deploy logic into multiple methods
Add the ability to do recursive deployments
scripts: Add helper commands for altering cluster morphologies
Add sysroot write extension
Test recursive deployment
morphlib/exts/sysroot.write | 29 +++++++
morphlib/plugins/deploy_plugin.py | 98 ++++++++++++++--------
scripts/edit-morph | 98 +++++++++++++++++++---
.../metadata-includes-morph-version.setup | 2 +-
tests.as-root/metadata-includes-repo-alias.setup | 2 +-
.../rootfs-tarball-builds-rootfs-and-kernel.script | 4 +-
...run-in-artifact-with-different-artifacts.stderr | 2 +-
tests.as-root/setup | 4 +-
tests.as-root/system-overlap.script | 2 +-
tests.as-root/tarball-image-is-sensible.setup | 2 +-
tests.branching/setup | 2 +-
tests.branching/tag-creates-commit-and-tag.stdout | 6 +-
tests.branching/tag-tag-works-as-expected.stdout | 6 +-
tests.build/bootstrap-mode.stdout | 6 +-
tests.build/setup | 2 +-
tests.deploy/setup | 4 +-
tests.merging/setup | 4 +-
tests/show-dependencies.setup | 2 +-
yarns/deployment.yarn | 37 +++++++-
yarns/implementations.yarn | 50 ++++++++++-
yarns/regression.yarn | 2 +-
yarns/splitting.yarn | 2 +-
22 files changed, 285 insertions(+), 81 deletions(-)
create mode 100755 morphlib/exts/sysroot.write
--
1.8.5.rc2
9 years, 6 months
Caching, cloning and --no-no-git-update
by Paul Sherwood
Hi
I'm confused about why morph defaults to updating all the repos all the
time.
1) If i set --no-git-update, and try to add a new chunk (e.g.
postgresql), morph stops with
'not cached yet'
2) so I have to allow git-update, which means morph pulls everything,
then during the build it reports
2014-03-11 21:05:08 [Build 1274/1448] [postgresql-devel] Cloning
upstream:postgresql
Do caching and cloning mean the same thing?
Should the error in 1) be 'not cloned yet'?
Can't we be smarter than this and default to no-git-update, only
checking upstream (to cache or clone) if the commit we need is not
already available locally?
br
Paul
9 years, 6 months
