[RFC] Multiple source repositories per-chunk and git submodule handling
by Richard Maw
We currently handle definitions that need to pull multiple source
repositories together in an ad-hoc way.
For gcc we import the gmp, mpc and mpfr source trees in by checking them
into our delta branch directly.
Some upstreams handle importing the sources by adding submodules.
However, we have to make them fetch sources from our Trove for
reproducibilty and latency concerns, so we end up having to add a delta
on top to change where the submodules fetch their sources from.
This works against one of our goals for minimising the delta, so we
need a way to encode in our builds how to deal with components that need
sources from multiple repositories to function.
Our current approach from submodules introduces a _lot_ of extra work
when we need to update multiple submodules recursively, so we need a
better solution.
Proposal
========
To solve this, I propose we extend the source repository information from
just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
So rather than:
name: base
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
unpetrify-ref: baserock/morph
⋮
build-depends: []
We extend it to be able to take a "sources" field.
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
sources:
- repo: upstream:foo
ref: baserock/morph # used to validate commit is anchored
# properly and as a hint for where
# you ought to merge changes to
commit: cafef00d…
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The `path` field is used to specify that this tree should be placed in
the parent tree at this position.
The `path` field is optional, defaulting to `.`
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Sub-sources can be placed over an empty directory, when there's no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
If there's a file, symlink, non-empty directory or submodule that doesn't
match `submodule-commit` there, then it results in a build-failure
at build-time.
The `submodule-commit` check exists as a safeguard against the parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
If you get a build fail because the submodule isn't appropriate then
you have to check whether the version you specify works, then update
the commit.
Cache key changes
=================
This shouldn't require any cache key changes to existing definitions,
but builds that make use of multiple source repositories will also hash
the commit tree and path.
Alternative solutions for submodules
====================================
We could continue to use the current model, and deal with the pain of
having to make multiple branches in multiple repositories to satisfy
the change to the repository paths.
We could have a magic look-up table to replace repository urls when we
parse the submodules.
We could use git-replace to magically make git do the url correction, but
then we'd have to handle replacements properly, rather than dropping them.
8 years
[PATCH 0/6] Firehose Implementation
by Lauren Perry
Hi all,
Back in November I sent to the list an RFC for Firehose. This preliminary patch series demonstrates the current work I have done, based on the pre-existing prototype Firehose, and the steps taken towards automating this process.
Currently, Firehose is not fully automated; there are still a number of steps that need to be performed manually, such as setting up a Firehose user, enabling the firehose.service and firehose.timer files and installing the firehose shell for general use. These steps are demonstrated in firehose.configure (prelim) and I will post a follow up patch with a README demonstrating steps that need to be taken currently.
Firehose will also need a Gerrit user configured with an appropriate set of ssh keys to access Gerrit.
A Firehose machine has been demonstrated to work properly on OpenStack, though it requires the firehose command to be done manually.
Regards,
Lauren
Lauren Perry (6):
Add tools for installing Firehose and its dependencies
Add systemd styled timers to automate Firehose checking process
Removed morph after placing its functionality in firehose.sh
Modify firehose email configuration and push location to ensure
commits are pushed to the testgerrit server with an appropriate
commit message and branch name
Install githook to add Change-Id value to commit message
Add firehose.configure, containing the necessary instructions to run
firehose
firehose.configure | 39 ++++++++++++++++++++++++++
firehose.service | 12 ++++++++
firehose.sh | 21 ++++++++++++++
firehose.timer | 11 ++++++++
{plugin => firehose/plugin}/firehose_plugin.py | 20 +++++++++----
morph | 17 -----------
setup.py | 16 +++++++++++
7 files changed, 113 insertions(+), 23 deletions(-)
create mode 100644 firehose.configure
create mode 100644 firehose.service
create mode 100755 firehose.sh
create mode 100644 firehose.timer
rename {plugin => firehose/plugin}/firehose_plugin.py (92%)
delete mode 100755 morph
create mode 100644 setup.py
--
1.8.4
--
1.7.10.4
8 years, 6 months
[PATCH 0/2] More checks for kvm exts
by Richard Ipsum
repo: git://git.baserock.org/baserock/baserock/morph
ref: baserock/richardipsum/kvm-ext-more-checks
head: 6348a43612459694501db5a5ce043c8d1dcbec87
This adds 2 new checks to the kvm deploy check ext so morph can fail sooner.
The first check just tests that we can write to the location we intend to
store the vm at.
The second check makes sure any virtual networks a vm needs have been started.
./check doesn't cover this at all, so to test try out some kvm deploys,
you can use the NIC_CONFIG environment variable to specify extra networks
besides the usual 'default' one, you can specify multiple nics too
like,
NIC_CONFIG: user bridge=foom default,mac=FF:29:5C:01:B9:25,model=e1000 pineapple
in this case the kvm.check ext should ignore 'user' and 'bridge' parameters
and just check that 'default' and 'pineapple' are networks that exist and
are started.
Richard Ipsum (2):
Check file can be created at location
Add check for virtual networks
morphlib/exts/kvm.check | 71 ++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 70 insertions(+), 1 deletion(-)
--
1.7.10.4
8 years, 9 months
[PATCH 0/6 v2] Toolchain update
by Tiago Gomes
repo: git://git.baserock.org/baserock/baserock/definitions
branch: baserock/tiagogomes/update-toolchain-v2
commit: 941c6a9653c784962aab1579ff066d7a274e8d84
This second version of the patch series
* Addresses Sam Thursfield's review comments.
* Moves M4 to (stage 3) build-essential.
* Adds a workaround for a build failure on ARMv7.
* Includes Paul Sherwood's patch to embed morph-arch and
morph-arch-config in definitions.
These changes were tested on x86_64, x86_32, ppc64 and armv7lhf archs.
Paul Sherwood (1):
Embed morph-arch and morph-arch-config
Tiago Gomes (5):
build-essential: separate each chunk definition with a line feed
Improve the readability and docs of gcc morphologies
Update binutils to 2.25
Move M4 from core to build-essential
Update GCC to 4.9.2
strata/bsp-x86_32-generic.morph | 4 +-
strata/bsp-x86_64-generic.morph | 4 +-
strata/bsp-x86_64-generic/syslinux.morph | 4 +-
strata/bsp-x86_both-tools.morph | 4 +-
strata/build-essential.morph | 79 ++++++++++++++----
strata/build-essential/gcc.morph | 46 ++++++-----
strata/build-essential/linux-api-headers.morph | 16 +++-
strata/build-essential/m4.morph | 5 ++
strata/build-essential/stage1-binutils.morph | 18 +++--
strata/build-essential/stage1-gcc.morph | 94 ++++++++++++++--------
strata/build-essential/stage2-binutils.morph | 11 +--
strata/build-essential/stage2-busybox.morph | 4 +-
strata/build-essential/stage2-gawk.morph | 2 +-
strata/build-essential/stage2-gcc.morph | 74 ++++++++++-------
strata/build-essential/stage2-libstdc++.morph | 36 +++++++++
.../build-essential/stage2-linux-api-headers.morph | 16 +++-
strata/build-essential/stage2-make.morph | 2 +-
strata/core.morph | 10 +--
strata/core/m4.morph | 5 --
strata/coreutils-common/coreutils.morph | 2 +
strata/tools.morph | 1 +
strata/tools/gdb.morph | 5 ++
22 files changed, 302 insertions(+), 140 deletions(-)
create mode 100644 strata/build-essential/m4.morph
create mode 100644 strata/build-essential/stage2-libstdc++.morph
delete mode 100644 strata/core/m4.morph
create mode 100644 strata/tools/gdb.morph
--
2.1.3
8 years, 9 months
[PATCH 0/2 v5] Distbuild crashing on invalid input
by Lauren Perry
Included changes to copyright years and better error catching on initiator_connection
There are tests to check this patch series on git.baserock.org:baserock/baserock/morph, branch sam/distbuild-tests
SHA1: 3c0a65dabc6d6d9841dc8e4fc522873ac169eb16
Repo: ssh://git@github.com/perryl/morph
Ref: baserock/lauren/distbuild-invalid-input-crash
Lauren Perry (2):
Fix distbuild controller crashing on some invalid inputs
Update copyright years
distbuild/initiator_connection.py | 24 ++++++++++++++----------
distbuild/jm.py | 11 ++++++++---
distbuild/socketsrc.py | 2 +-
distbuild/sockserv.py | 2 +-
4 files changed, 24 insertions(+), 15 deletions(-)
--
1.7.10.4
8 years, 9 months
[PATCH 00/11] Speed up construction of the build graph
by Adam Coldrick
Hi all,
This patch series speeds up the construction of the build graph. It
can be frustrating starting a build and having to wait at "Deciding
task order..." for nearly a minute sometimes before the build actually
starts. This series addresses that by reducing the build graph
construction time significantly on multiple builds.
There will not be much of an effect on the first build, since nothing
this series caches will yet be cached, but on subsequent builds the
build graph is constructed in around 1 second. This is achieved by
caching tree SHAs by (repo, commit_sha) and caching build-systems
with the same key. The caching of build systems means that lots of
getting file lists from remote repositories is avoided.
This introduces a dependency for morph on pylru[0], so I'll update the
"Using latest morph" documentation on merge. I'll send a patch to
add a pylru chunk once I've merged this into morph master too.
[0]: https://github.com/jlhutch/pylru
Repo: git://git.baserock.org/baserock/baserock/morph.git
Ref: baserock/adamcoldrick/build-graph-speedups
SHA1: a4b9f28cb54ff4d445cc3a035d9fccbcb90ff16a
Target: master
Adam Coldrick (4):
Fix some yarns to shut down their git servers when finished
Convert the empty-stratum test into a yarn
Add a cache manager utilising pylru and pickle
Add tests for sourceresolver
Sam Thursfield (7):
sourceresolver: Add comments and factor out common function
Move MorphologyFactory into SourceResolver
Add an LRU cache for resolved tree refs
Add an LRU cache for detected build-systems
Split up traverse_morphs to improve readability
Add a mechanism for extracting all files from a given commit to a dir
Read files from a local clone of definitions where possible
morphlib/__init__.py | 1 -
morphlib/buildcommand.py | 1 +
morphlib/cachedrepo.py | 27 +-
morphlib/cachedrepo_tests.py | 30 +-
morphlib/gitindex.py | 22 +-
morphlib/gitindex_tests.py | 16 +-
morphlib/morphologyfactory.py | 90 -----
morphlib/plugins/list_artifacts_plugin.py | 3 +-
morphlib/sourceresolver.py | 407 +++++++++++++++++---
...ogyfactory_tests.py => sourceresolver_tests.py} | 130 +++++--
tests.build/empty-stratum.exit | 1 -
tests.build/empty-stratum.script | 36 --
tests.build/empty-stratum.stderr | 1 -
without-test-modules | 4 -
yarns/branches-workspaces.yarn | 1 +
yarns/building.yarn | 12 +
yarns/deployment.yarn | 1 +
yarns/implementations.yarn | 24 ++
18 files changed, 565 insertions(+), 242 deletions(-)
delete mode 100644 morphlib/morphologyfactory.py
rename morphlib/{morphologyfactory_tests.py => sourceresolver_tests.py} (63%)
delete mode 100644 tests.build/empty-stratum.exit
delete mode 100755 tests.build/empty-stratum.script
delete mode 100644 tests.build/empty-stratum.stderr
--
1.7.10.4
8 years, 9 months
[PATCH 0/5] cross-bootstrap improvements
by Pedro Alvarez
From: Pedro Alvarez <palvarez89(a)gmail.com>
Repo: baserock:baserock/morph
Branch: baserock/pedroalvarez/cross-bootstrap-improvements
Sha1: 82518147020f5e5930d0273933c926413e7b8918
Landing: master
The 'cross-bootstrap' subcommand is becoming popular so I've
decided to do some improvements.
This patch attempts to modify 'cross-bootstrap' to be used like:
morph cross-bootstrap armv7lhf systems/cross-bootstrap-system-armv7lhf-generic.morph
But keeping the original in a new subcommand called
'cross-bootstrap-morphology' given that its behaviour is similar to
the 'build-morphology' one.
I'm not 100% confident with this patch, so please, review carefully,
and be nice with me :)
Pedro Alvarez (5):
Rename cross-bootstrap with cross-bootstrap-morphology
Add cross-bootstrap plugin
yarns: Update tests to follow the cross-bootstrap renaming
yarns: Modify existing test to use the new cross-bootstrap command
tests.build: Fix cross-bootstrap tests to use
cross-bootstrap-morphology
morphlib/plugins/cross-bootstrap_plugin.py | 71 ++++++++++++++++++++++++++--
tests.build/cross-bootstrap.script | 8 ++--
yarns/architecture.yarn | 5 +-
yarns/implementations.yarn | 9 +++-
4 files changed, 80 insertions(+), 13 deletions(-)
--
1.7.10.4
8 years, 10 months
[PATCH] Distbuild fails if input isn't valid JSON
by Lauren Perry
Currently, when running morph controller-daemon, if it receives any input either not in JSON format or in invalid JSON (e.g. {"sha":"bla"}) it crashes. This patch adds exception and error handling to raise an error for invalid input and pass over the input entirely.
SHA1: 583cbc457fd36f039425f0dceef479297001d6d2
Repo: ssh://git@github.com/perryl/morph
Ref: baserock/lauren/distbuild-invalid-input-crash
Lauren Perry (1):
Running controller-daemon should display an error message and
pass on invalid input rather than crashing
distbuild/jm.py | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
--
1.7.10.4
8 years, 10 months
[PATCH 0/4] Autodefrag /var by default
by Richard Ipsum
repo: git://git.baserock.org/baserock/baserock/morph
ref: baserock/richardipsum/btrfs-fstab-layout
head: ba19bb0de8e13732ae0be1d34cbf75e37e88719b
Richard Ipsum (4):
Rename vars in complete_fstab_for_btrfs_layout
Add optional options for mount points
Mount /var with autodefrag
Update Copyright
morphlib/writeexts.py | 25 +++++++++++++------------
1 file changed, 13 insertions(+), 12 deletions(-)
--
1.7.10.4
8 years, 10 months
[PATCH] lorry-controller: Actually honour proxy configuration
by Sam Thursfield
Repository: git://git.baserock.org/baserock/baserock/lorry-controller
Ref: sam/proxy-fixes
Sha1: 18dec2bbe81b1d0d2bb5aca77034aecd42192779
After several hours of debugging a Trove behind a proxy, we discovered that
urllib2 actually ignores any proxy config we give it. Switching to Requests
fixes the issue.
Sam Thursfield (1):
Use 'requests' instead of the awful 'urllib2'
lorrycontroller/gitano.py | 18 +++++++-----------
lorrycontroller/proxy.py | 10 ----------
2 files changed, 7 insertions(+), 21 deletions(-)
--
1.9.3
8 years, 10 months
