[RFC] Multiple source repositories per-chunk and git submodule handling
by Richard Maw
We currently handle definitions that need to pull multiple source
repositories together in an ad-hoc way.
For gcc we import the gmp, mpc and mpfr source trees in by checking them
into our delta branch directly.
Some upstreams handle importing the sources by adding submodules.
However, we have to make them fetch sources from our Trove for
reproducibilty and latency concerns, so we end up having to add a delta
on top to change where the submodules fetch their sources from.
This works against one of our goals for minimising the delta, so we
need a way to encode in our builds how to deal with components that need
sources from multiple repositories to function.
Our current approach from submodules introduces a _lot_ of extra work
when we need to update multiple submodules recursively, so we need a
better solution.
Proposal
========
To solve this, I propose we extend the source repository information from
just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
So rather than:
name: base
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
unpetrify-ref: baserock/morph
⋮
build-depends: []
We extend it to be able to take a "sources" field.
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
sources:
- repo: upstream:foo
ref: baserock/morph # used to validate commit is anchored
# properly and as a hint for where
# you ought to merge changes to
commit: cafef00d…
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The `path` field is used to specify that this tree should be placed in
the parent tree at this position.
The `path` field is optional, defaulting to `.`
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Sub-sources can be placed over an empty directory, when there's no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
If there's a file, symlink, non-empty directory or submodule that doesn't
match `submodule-commit` there, then it results in a build-failure
at build-time.
The `submodule-commit` check exists as a safeguard against the parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
If you get a build fail because the submodule isn't appropriate then
you have to check whether the version you specify works, then update
the commit.
Cache key changes
=================
This shouldn't require any cache key changes to existing definitions,
but builds that make use of multiple source repositories will also hash
the commit tree and path.
Alternative solutions for submodules
====================================
We could continue to use the current model, and deal with the pain of
having to make multiple branches in multiple repositories to satisfy
the change to the repository paths.
We could have a magic look-up table to replace repository urls when we
parse the submodules.
We could use git-replace to magically make git do the url correction, but
then we'd have to handle replacements properly, rather than dropping them.
7 years, 6 months
[PATCH 0/6] Firehose Implementation
by Lauren Perry
Hi all,
Back in November I sent to the list an RFC for Firehose. This preliminary patch series demonstrates the current work I have done, based on the pre-existing prototype Firehose, and the steps taken towards automating this process.
Currently, Firehose is not fully automated; there are still a number of steps that need to be performed manually, such as setting up a Firehose user, enabling the firehose.service and firehose.timer files and installing the firehose shell for general use. These steps are demonstrated in firehose.configure (prelim) and I will post a follow up patch with a README demonstrating steps that need to be taken currently.
Firehose will also need a Gerrit user configured with an appropriate set of ssh keys to access Gerrit.
A Firehose machine has been demonstrated to work properly on OpenStack, though it requires the firehose command to be done manually.
Regards,
Lauren
Lauren Perry (6):
Add tools for installing Firehose and its dependencies
Add systemd styled timers to automate Firehose checking process
Removed morph after placing its functionality in firehose.sh
Modify firehose email configuration and push location to ensure
commits are pushed to the testgerrit server with an appropriate
commit message and branch name
Install githook to add Change-Id value to commit message
Add firehose.configure, containing the necessary instructions to run
firehose
firehose.configure | 39 ++++++++++++++++++++++++++
firehose.service | 12 ++++++++
firehose.sh | 21 ++++++++++++++
firehose.timer | 11 ++++++++
{plugin => firehose/plugin}/firehose_plugin.py | 20 +++++++++----
morph | 17 -----------
setup.py | 16 +++++++++++
7 files changed, 113 insertions(+), 23 deletions(-)
create mode 100644 firehose.configure
create mode 100644 firehose.service
create mode 100755 firehose.sh
create mode 100644 firehose.timer
rename {plugin => firehose/plugin}/firehose_plugin.py (92%)
delete mode 100755 morph
create mode 100644 setup.py
--
1.8.4
--
1.7.10.4
7 years, 11 months
[PATCH] lorry-controller: Actually honour proxy configuration
by Sam Thursfield
Repository: git://git.baserock.org/baserock/baserock/lorry-controller
Ref: sam/proxy-fixes
Sha1: 18dec2bbe81b1d0d2bb5aca77034aecd42192779
After several hours of debugging a Trove behind a proxy, we discovered that
urllib2 actually ignores any proxy config we give it. Switching to Requests
fixes the issue.
Sam Thursfield (1):
Use 'requests' instead of the awful 'urllib2'
lorrycontroller/gitano.py | 18 +++++++-----------
lorrycontroller/proxy.py | 10 ----------
2 files changed, 7 insertions(+), 21 deletions(-)
--
1.9.3
8 years, 3 months
[PATCH 0/8] Implement Mason with Zuul and turbo-hipster
by Adam Coldrick
Hi all,
A while ago I sent an RFC about using Zuul (and Gearman) and
turbo-hipster to improve Mason. I went ahead with this after the
response I had, and this patch series is the result.
This patch series adds Zuul, turbo-hipster and their dependencies
to definitions and creates a Mason system containing them. It uses
mason.configure to insert some configuration files and set up some
SSH keys and the like. An Ansible playbook runs on boot which
fills in the template configuration files and starts the Zuul and
turbo-hipster services that are required.
It is important to note that there is some configuration that needs
to be done to the Gerrit instance you wish to deploy this against.
Namely, you will need to create a Mason user, add a Verified label
and allow the Mason user to vote on that label, and add the required
SSH key to the Mason user.
I have only tested using this in OpenStack, as I understand that
is how the Baserock project intends to use it.
Best regards,
Adam
Adam Coldrick (8):
Update python-prettytable and pbr
Add python-swiftclient and dependencies
Add pycrypto and ecdsa to Ansible stratum
Add a stratum containing Zuul, turbo-hipster, and dependencies
Make mason.configure install the new Mason config
Add a system for Mason
Add an example cluster morphology for mason
Add a README for mason
.../mason-system-x86_64-openstack-deploy.morph | 148 ++++++++++++
mason.configure | 144 ++++++-----
mason/README | 120 ++++++++++
mason/ansible/mason-setup.yml | 130 ++++++----
mason/{httpd.service => lighttpd.service} | 2 +-
mason/mason-generator.sh | 101 --------
mason/mason-report.sh | 252 --------------------
mason/mason.service | 10 -
mason/mason.sh | 93 --------
mason/mason.timer | 10 -
mason/share/lighttpd.conf | 21 ++
mason/share/mason.conf | 14 --
mason/share/turbo-hipster-config.yaml | 47 ++++
mason/share/zuul-layout.yaml | 23 ++
mason/share/zuul-logging.conf | 44 ++++
mason/share/zuul.conf | 26 ++
mason/turbo-hipster.service | 10 +
mason/zuul-merger.service | 10 +
mason/zuul-server.service | 10 +
strata/ansible.morph | 15 +-
strata/ansible/pycrypto.morph | 3 +
strata/openstack-clients.morph | 26 +-
strata/zuul-ci.morph | 132 ++++++++++
strata/zuul-ci/pycrypto.morph | 3 +
strata/zuul-ci/system-tests.morph | 5 +
systems/mason-system-x86_64-generic.morph | 48 ++++
26 files changed, 858 insertions(+), 589 deletions(-)
create mode 100644 clusters/mason-system-x86_64-openstack-deploy.morph
create mode 100644 mason/README
rename mason/{httpd.service => lighttpd.service} (69%)
delete mode 100755 mason/mason-generator.sh
delete mode 100755 mason/mason-report.sh
delete mode 100644 mason/mason.service
delete mode 100755 mason/mason.sh
delete mode 100644 mason/mason.timer
create mode 100644 mason/share/lighttpd.conf
delete mode 100644 mason/share/mason.conf
create mode 100644 mason/share/turbo-hipster-config.yaml
create mode 100644 mason/share/zuul-layout.yaml
create mode 100644 mason/share/zuul-logging.conf
create mode 100644 mason/share/zuul.conf
create mode 100644 mason/turbo-hipster.service
create mode 100644 mason/zuul-merger.service
create mode 100644 mason/zuul-server.service
create mode 100644 strata/ansible/pycrypto.morph
create mode 100644 strata/zuul-ci.morph
create mode 100644 strata/zuul-ci/pycrypto.morph
create mode 100644 strata/zuul-ci/system-tests.morph
create mode 100644 systems/mason-system-x86_64-generic.morph
--
1.7.10.4
8 years, 3 months
[PATCH 0/6] Add xwayland support to the xserver and the weston session
by Javier Jardón
REPO: ssh://git@git.baserock.org/baserock/baserock/definitions.git
BRANCH: baserock/jjardon/xwayland
COMMIT: 6ae134c46c8ecf2f98712a2eb95726f2d42a6a30
To run weston with xwayland support you need to launch it like this (in a qemu VM):
'weston-launch -- --backend=fbdev-backend.so --modules=xwayland.so'
(You need to use weston-launch as seems weston has a bug [1] that we are investigating upstream)
Note the last patch is not to meant to be merged, only in case you want to try a pure
X application in your weston session to test xwayland its working
Javier Jardón (6):
xserver.morph: build minimal xserver with only the XWayland binary
strata/x-common.morph: clean not needed dependencies
strata/x-generic.morph: clean not needed dependencies
weston.morph: Enable xwayland in weston
Add xserver to weston-system
strata/x-generic.morph: build xterm
strata/graphics-common.morph | 1 +
strata/weston-common/weston.morph | 2 -
strata/x-common.morph | 77 -----------------
strata/x-generic.morph | 128 +++++------------------------
strata/x-generic/xserver.morph | 10 ++-
systems/weston-system-x86_64-generic.morph | 4 +
6 files changed, 36 insertions(+), 186 deletions(-)
--
2.1.3
[1] https://bugs.freedesktop.org/show_bug.cgi?id=87322
8 years, 3 months
[PATCH v3 0/9] Treat path given to build cmd as relative to cwd
by Richard Ipsum
repo: git://git.baserock.org/baserock/baserock/morph
ref: baserock/richardipsum/build-command-tweak-v3
head: 9a3b1b2182689a1d0f49f362e61ee1a82da09e45
Richard Ipsum (9):
Add relative_to_root_repo to sysbranchdir
Treat path given to build cmd as relative to cwd
Update tests to work with modified build commands
Add yarns to test building with relative paths
Treat path given to deploy cmd as relative to cwd
Add yarns to test deploying with relative paths
Quote variables and use = instead of ==
Make some existing deployment scenarios simpler
Fix an existing build yarn
morphlib/plugins/build_plugin.py | 8 +++--
morphlib/plugins/deploy_plugin.py | 6 ++--
morphlib/sysbranchdir.py | 6 ++++
yarns/building.yarn | 30 +++++++++++++++++-
yarns/deployment.yarn | 45 ++++++++++++++++++---------
yarns/implementations.yarn | 61 ++++++++++++++++++++++++++++---------
6 files changed, 123 insertions(+), 33 deletions(-)
--
1.7.10.4
8 years, 4 months
[PATCH V2 0/3] Add an ARMv7 hard-floot rootfs
by James Thomas
Branch: baserock/james/add_rootfs
It's very useful to be able to simply download a hard float rootfs to use for
testing (e.g for board bring-up), these patches create such a rootfs, and adds
it to the release.morph so it's built
The patches also add a simple configuration extension to create the required
file to allow this to be used with crouton on a ARM chromebook
The intention here is that the image will be availible to download on the BR
website
James Thomas (3):
Add a generic ARMv7 rootfs
Add a configuration extension for crouton
Add rootfs to release cluster
clusters/release.morph | 5 +++++
crouton.configure | 27 ++++++++++++++++++++++++
systems/build-system-armv7lhf-rootfs.morph | 33 ++++++++++++++++++++++++++++++
3 files changed, 65 insertions(+)
create mode 100755 crouton.configure
create mode 100644 systems/build-system-armv7lhf-rootfs.morph
--
1.9.3
8 years, 4 months
Baserock Import tool "version 1.0"
by Sam Thursfield
Hi all
The Baserock Import tool is in a usable state. It can generate
.lorry and .morph files for many of the projects on
http://rubygems.org/. Initial support for importing stuff from
http://pypi.python.org/ is on its way shortly too.
There are a number of important improvements that I think it needs (see
the TODO file), but I feel like that the overall approach it takes is
sound, and I'm hopeful that after reading the tutorial you will feel
that using the tool sounds more attractive than the equivalent manual
process. I hope it also seems nicer than putting .gem files directly
into your production systems. (There's no reason not to continue using
'gem' and 'bundler' during development, by the way, I think it's only
once audit and reproducibility become a concern that the Baserock
tooling becomes interesting).
The tutorial is here:
<http://wiki.baserock.org/guides/import-tool-rubygems/>. It's quite long
(and the plot is quite weak). Future improvements to the tool
will allow us to remove sections of the tutorial (while also making it a
bit less "hands-on", I think).
The source code is here:
<http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/import.git/>
An import of the core Chef client software that I did a while back
exists in this branch of definitions.git:
<http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.gi...>
As this mail marks the end of the initial development phase I think we
should start using the normal review policy on this repo, although since
not many of us are familiar with the codebase yet I think myself and
Richard Ipsum should be free to +2 import tool patches for a bit,
to avoid changes getting stuck for lack of reviewers!
Happy importing!
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
8 years, 4 months
[PATCH 0/3] morph: Clean up CachedRepo module
by Sam Thursfield
Repository: git://git.baserock.org/baserock/baserock/morph
Ref: sam/cached-repo-cleanup
Sha1: 7e975343c7f00e98962d2edd07ac87630c4936c4
This patch series should have no functional changes, beyond changing
the wording of a couple of error messages. Its goal is to remove some
duplicate code paths: I think all Git access should be done using the
'gitdir' module, so I've made the 'cachedrepo' module use the 'gitdir'
code internally where possible.
This caused quite a lot of faff in the unit tests. I think we'd be able
to make the test code a lot less whacky if we used Mock[1], instead of
manually monkeypatching things everywhere.
Sam Thursfield (3):
Separate resolution of ref->commit and commit->tree
Rework CachedRepo to use the GitDirectory class where possible
Remove unused code from morphlib.git
morphlib/buildcommand.py | 4 +-
morphlib/builder2_tests.py | 6 +-
morphlib/cachedrepo.py | 150 ++++++-----------
morphlib/cachedrepo_tests.py | 180 ++++++---------------
morphlib/git.py | 33 ----
morphlib/gitdir.py | 44 +++--
morphlib/gitdir_tests.py | 27 ++++
morphlib/localrepocache.py | 8 +-
morphlib/localrepocache_tests.py | 7 +
morphlib/morphologyfactory.py | 5 +-
morphlib/morphologyfactory_tests.py | 26 +--
morphlib/plugins/artifact_inspection_plugin.py | 4 +-
morphlib/plugins/branch_and_merge_plugin.py | 4 +-
morphlib/sourceresolver.py | 28 ++--
tests.branching/branch-cleans-up-on-failure.stderr | 2 +-
.../checkout-cleans-up-on-failure.stderr | 2 +-
tests.build/missing-ref.stderr | 2 +-
17 files changed, 214 insertions(+), 318 deletions(-)
--
1.9.3
8 years, 4 months
[PATCH 0/3] Add the ability to deploy installer systems
by Pedro Alvarez
Repo: baserock:baserock/definitions
Branch: baserock/pedroalvarez/installer2
Sha1: fdd42736cec68a22388f0d008e5cb7078ed9b1c4
Land: master
Pedro Alvarez (3):
Update morph to a version that allows deployments to devices
Add the ability to deploy installer systems
Add example cluster of a baserock installer system
clusters/installer-build-system-x86_64.morph | 38 ++++++
installer/etc/install.conf | 3 +
installer/installer.py | 178 ++++++++++++++++++++++++++
installer/manifest | 3 +
strata/morph-utils.morph | 2 +-
5 files changed, 223 insertions(+), 1 deletion(-)
create mode 100644 clusters/installer-build-system-x86_64.morph
create mode 100644 installer/etc/install.conf
create mode 100755 installer/installer.py
create mode 100644 installer/manifest
--
1.7.10.4
8 years, 4 months
