Public facing Mason instance.
by Pedro Alvarez
Hello all!
As you probably know, we have been thinking about Continuous Delivery and
Continuous Integration. As a result we have started the implemented of Mason.
Today we have deployed a public facing Mason instance so everyone can see it.
What Mason does is:
- Check that the current definition of a devel system (x86_64) can be built.
- Populate an artifact cache server with the artifacts of the previous build.
We are currently working on it, and it will have more improvements soon.
The current IP for Mason is: http://85.199.252.95/
If you want to use the artifacts you only have to add this line to your
morph.conf file:
artifact-cache-server = http://85.199.252.93:8080/
--
Pedro
8 years, 6 months
[PATCH 0/2] Lorry Controller: Add lists of all, failed jobs
by Lars Wirzenius
repo: git://git.baserock.org/baserock/baserock/lorry-controller
branch: baserock/liw/lc-list-failed-jobs
commit: c491acc03665213e01e40fc9ef81f5e326cf9ff9
land: master
card: S11751
This adds a list of failed jobs, and another list of all jobs, to the
status page for each lorry specification that Lorry Controller knows
about. This will make it a bit easier to debug lorrying problems on a
Trove.
Lars Wirzenius (2):
Add tests for jobs, failed_jobs attributes in lorry specs
Add lists of failed, all jobs to lorry spec info
lorrycontroller/showlorry.py | 35 +++++++++++++++--------
lorrycontroller/statedb.py | 13 +++++++++
templates/lorry.tpl | 25 +++++++++++++++++
yarns.webapp/040-running-jobs.yarn | 57 ++++++++++++++++++++++++++++++++++++--
4 files changed, 115 insertions(+), 15 deletions(-)
--
1.8.4
8 years, 6 months
[RFC} Freezing upstream tags for reproducibility
by Paul Sherwood
Hi all
in a previous thread [1] we discussed issues around consuming upstream tags.
I think my proposal
(baserock-<upstream-tag>-<short-sha-of-upstream-commit>) was defensible,
and better from a usability point of view than what we're currently
doing, but so far we haven't adopted the practice of locking our own tags.
Over the weekend another possibility sprang to mind: could we manage our
git repos so as to guarantee integrity of upstream tags?
In other words, we somehow ensure that no (published) tag is ever
deleted or changed. This would mean we could directly use (locked) tags
in our ref: fields, which would be much nicer from a user standpoint.
- maybe lorry could bounce and any attempt to change/delete by an upstream?
- maybe gitano could bounce and report any attempt to change/delete by a
user
- maybe tags would only become set-in-stone after a cooling off period
of weeks/months (to allow for fixing mistakes)?
- and/or maybe we just keep a separate list of tags and their refs for
all the upstreams we care about, just in case, and report any
differences the moment they happen?
- maybe we could/should campaign to establish 'tags are forever' with
the git community and wider upstream?
br
Paul
[1]
http://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2014...
--
Paul Sherwood Codethink Ltd.
Tel: +44 788 798 4900 302 Ducie House, Ducie Street,
http://www.codethink.co.uk/ Manchester, M1 2JW, United Kingdom.
Codethink provides advanced software design, development, integration &
test services: from embedded systems to high performance apps to cloud.
8 years, 6 months
Simplifying workflows: morph branch issues
by Pete Fotheringham
Some issues to address with `morph branch`
1. No `morph merge`
===================
The 'normal' git workflow is: create branch, hack on branch, commit to
branch, push branch, review branch, merge branch to master, push master
morph provides functionality /commands to perform those operations
across all the git repositories needed for a Baserock system, *except*
for 'merge branch to master'.
Possible solutions
1. add `morph merge`
2. something else??
Things to bear in mind:
- operation would need to be atomic: rollback changes in case of failure
of merge in one or more repos;
- do we need to support merging from one system branch to another (i.e.
merge to a branch other than 'master'? (See old mustard file
mustard/req/morph/branch-and-merge.yaml)
2: `morph checkout -b` ?
========================
git users can create a branch either by using `git branch
new-branch-name`, *or* by using `git checkout -b new-branch-name` which
will create the new branch, and switch the working tree to it
(equivalent to `git branch new-branch-name` followed by `git checkout
new-branch-name`). `morph checkout` does not support this - you need to
do both steps separately, `morph branch new-branch-name` followed by
`morph checkout new-branch-name`.
Possible solutions
1. add `-b` functionality to `morph checkout`
2. clearly document the difference in command line help, and on the wiki
3. create a wiki page which documents as many as possible of the
differences between morph commands and their git equivalents
I think I would prefer to avoid 1 - it's writing code which doesn't add
very much to the usability. I think we should go for 2 and / or 3.
3: Different branches of components from the same repo
======================================================
Baserock does not currently support creating different branches of
components that come from the same repo e.g Linux Kernel and Linux API
headers. (This issue was identified in the assessment work I did back in
July)
* Is this a real use case? If so, do we want / need to fix it, and how?
Regards
Pete
--
Pete Fotheringham
+44 7740 351755
Codethink Ltd
http://codethink.co.uk
8 years, 7 months
[PATCH] Log elapsed time for each chunk build
by Paul Sherwood
This patch gives feedback from `morph build` on how long each chunk
actually takes to build (including staging etc). Using "Elapsed" makes
it easy to grep these entries from the log. I'm hoping others are
interested in logging this info by default, and in how we could speed
the times up.
repo: git://git.baserock.org/baserock/baserock/morph.git
branch: baserock/ps/log-chunk-build-times
---
diff --git a/morphlib/buildcommand.py b/morphlib/buildcommand.py
index edd2f0c..c5ba05e 100644
--- a/morphlib/buildcommand.py
+++ b/morphlib/buildcommand.py
@@ -19,6 +19,7 @@ import os
import shutil
import logging
import tempfile
+import datetime
import morphlib
import distbuild
@@ -320,6 +321,7 @@ class BuildCommand(object):
in either the local or remote cache already.
'''
+ starttime = datetime.datetime.now()
self.app.status(msg='Building %(kind)s %(name)s',
name=source.name,
kind=source.morphology['kind'])
@@ -367,6 +369,11 @@ class BuildCommand(object):
self.build_and_cache(staging_area, source, setup_mounts)
self.remove_staging_area(staging_area)
+ td = datetime.datetime.now() - starttime
+ td_string = "%02d:%02d:%02d" % (td.seconds/3600,
+ td.seconds%3600/60,
td.seconds%60)
+ self.app.status(msg="Elapsed time %(duration)s",
duration=td_string)
+
def get_recursive_deps(self, artifacts):
deps = set()
ordered_deps = []
--
cgit v0.9.0.3-67-gacbf
8 years, 7 months
[PATCH 00/11] Start on removing build-mode: test
by Richard Maw
Repo: http://richard.maw.name/cgit/cgit.cgi/personal/richardmaw/morph.git
Ref: baserock/richardmaw-os/bootstrap-build-tests
Land: git://git.baserock.org/baserock/baserock/morph.git
This is work in progress to get rid of build-mode: test. It's not complete
since I didn't finish cleaning up cmdtests that use it, but I'm most of
the way through.
Richard Maw (11):
Add test shell for use in staging area tests
Modify yarns to use test-shell instead of build-mode: test
Remove unused cmdtests
Stop tests.branching needing build-mode: test
Make show-dependencies not use build-mode: test
Move the branch-from-image test to a yarn
Move architecture existing test to yarn
Remove tarball image format tests from tests.as-root
Remove run-in-artifact subcommand
Remove overlap detection logic
Remove remaining test.as-root cmdtests
check | 30 --
morphlib/builder2.py | 64 ----
morphlib/plugins/artifact_inspection_plugin.py | 34 +--
scripts/list-overlaps | 45 ---
scripts/test-shell.c | 144 +++++++++
tests.as-root/archless-system-fails.exit | 1 -
tests.as-root/archless-system-fails.script | 35 ---
tests.as-root/archless-system-fails.stderr | 1 -
tests.as-root/branch-from-image-works.script | 57 ----
tests.as-root/branch-from-image-works.setup | 1 -
tests.as-root/branch-from-image-works.stdout | 1 -
.../build-handles-stratum-build-depends.script | 50 ----
tests.as-root/build-with-external-strata.script | 62 ----
tests.as-root/build-with-push.script | 38 ---
...iple-times-doesnt-generate-new-artifacts.script | 43 ---
...system-branch-picks-up-committed-removes.script | 62 ----
.../building-a-system-branch-works-anywhere.script | 61 ----
tests.as-root/lib | 36 ---
.../metadata-includes-morph-version.script | 52 ----
.../metadata-includes-morph-version.setup | 36 ---
tests.as-root/metadata-includes-repo-alias.script | 49 ----
tests.as-root/metadata-includes-repo-alias.setup | 36 ---
.../run-in-artifact-propagates-exit-code.exit | 1 -
.../run-in-artifact-propagates-exit-code.script | 33 ---
.../run-in-artifact-propagates-exit-code.stderr | 3 -
...run-in-artifact-with-different-artifacts.script | 47 ---
...run-in-artifact-with-different-artifacts.stdout | 32 --
tests.as-root/setup | 194 ------------
tests.as-root/setup-build | 35 ---
tests.as-root/system-overlap.script | 113 -------
tests.as-root/system-overlap.stdout | 3 -
tests.as-root/tarball-image-is-sensible.script | 44 ---
tests.as-root/tarball-image-is-sensible.setup | 70 -----
tests.as-root/tarball-image-is-sensible.stderr | 1 -
tests.as-root/tarball-image-is-sensible.stdout | 42 ---
tests.branching/edit-updates-stratum.stdout | 4 +-
tests.branching/setup | 2 +-
tests.branching/setup-second-chunk | 4 +-
tests.build/stratum-overlap-warns.script | 39 ---
tests.build/stratum-overlap-warns.setup | 102 -------
tests.build/stratum-overlap-writes-overlap.script | 35 ---
tests.build/stratum-overlap-writes-overlap.setup | 1 -
tests.build/stratum-overlap-writes-overlap.stdout | 4 -
tests.deploy/deploy-cluster.script | 60 ----
tests.deploy/deploy-cluster.stdout | 2 -
.../deploy-rawdisk-without-disk-size-fails.script | 30 --
tests.deploy/deploy-rawdisk.script | 33 ---
tests.deploy/setup | 216 --------------
tests.deploy/setup-build | 35 ---
tests/setup | 74 -----
tests/show-dependencies.setup | 32 +-
yarns/architecture.yarn | 12 +
yarns/branches-workspaces.yarn | 11 +-
yarns/deployment.yarn | 45 +++
yarns/implementations.yarn | 324 +++++++++++----------
yarns/regression.yarn | 6 +-
yarns/splitting.yarn | 27 +-
57 files changed, 415 insertions(+), 2239 deletions(-)
delete mode 100755 scripts/list-overlaps
create mode 100644 scripts/test-shell.c
delete mode 100644 tests.as-root/archless-system-fails.exit
delete mode 100755 tests.as-root/archless-system-fails.script
delete mode 100644 tests.as-root/archless-system-fails.stderr
delete mode 100755 tests.as-root/branch-from-image-works.script
delete mode 120000 tests.as-root/branch-from-image-works.setup
delete mode 100644 tests.as-root/branch-from-image-works.stdout
delete mode 100755 tests.as-root/build-handles-stratum-build-depends.script
delete mode 100755 tests.as-root/build-with-external-strata.script
delete mode 100755 tests.as-root/build-with-push.script
delete mode 100755 tests.as-root/building-a-system-branch-multiple-times-doesnt-generate-new-artifacts.script
delete mode 100755 tests.as-root/building-a-system-branch-picks-up-committed-removes.script
delete mode 100755 tests.as-root/building-a-system-branch-works-anywhere.script
delete mode 100644 tests.as-root/lib
delete mode 100755 tests.as-root/metadata-includes-morph-version.script
delete mode 100755 tests.as-root/metadata-includes-morph-version.setup
delete mode 100755 tests.as-root/metadata-includes-repo-alias.script
delete mode 100755 tests.as-root/metadata-includes-repo-alias.setup
delete mode 100644 tests.as-root/run-in-artifact-propagates-exit-code.exit
delete mode 100755 tests.as-root/run-in-artifact-propagates-exit-code.script
delete mode 100644 tests.as-root/run-in-artifact-propagates-exit-code.stderr
delete mode 100755 tests.as-root/run-in-artifact-with-different-artifacts.script
delete mode 100644 tests.as-root/run-in-artifact-with-different-artifacts.stdout
delete mode 100755 tests.as-root/setup
delete mode 100644 tests.as-root/setup-build
delete mode 100755 tests.as-root/system-overlap.script
delete mode 100644 tests.as-root/system-overlap.stdout
delete mode 100755 tests.as-root/tarball-image-is-sensible.script
delete mode 100755 tests.as-root/tarball-image-is-sensible.setup
delete mode 100644 tests.as-root/tarball-image-is-sensible.stderr
delete mode 100644 tests.as-root/tarball-image-is-sensible.stdout
delete mode 100755 tests.build/stratum-overlap-warns.script
delete mode 100755 tests.build/stratum-overlap-warns.setup
delete mode 100755 tests.build/stratum-overlap-writes-overlap.script
delete mode 120000 tests.build/stratum-overlap-writes-overlap.setup
delete mode 100644 tests.build/stratum-overlap-writes-overlap.stdout
delete mode 100755 tests.deploy/deploy-cluster.script
delete mode 100644 tests.deploy/deploy-cluster.stdout
delete mode 100755 tests.deploy/deploy-rawdisk-without-disk-size-fails.script
delete mode 100755 tests.deploy/deploy-rawdisk.script
delete mode 100755 tests.deploy/setup
delete mode 100644 tests.deploy/setup-build
--
1.9.1
8 years, 7 months
[PATCH] morph: Fix issue with caused problems on public Mason
by Sam Thursfield
Repository: ssh://git@git.baserock.org/baserock/baserock/morph
Ref: sam/gbo-read-access
Sha1: baa65ed0a238b0c51d21af214600d84978476544
We have a public Mason instance doing continuous build at
http://85.199.252.101/. Builds were failing due to errors like this:
2014-09-26 15:34:52 Building stratum build-essential-devel
2014-09-26 15:34:52 Cloning baserock:baserock/definitions
2014-09-26 15:34:52 Failed to fetch tarball, falling back to git clone.
ERROR: Cannot find remote git repository: baserock:baserock/definitions
Unable to extract tarball http://git.baserock.org/tarballs/ssh___git_git_baserock_org_baserock_base...: Command failed: wget -q -O- http://git.baserock.org/tarballs/ssh___git_git_baserock_org_baserock_base...
tar: short read
Unable to clone from ssh://git@git.baserock.org/baserock/baserock/definitions to /srv/distbuild/gits/tmpWQzN3y: Command failed: git clone --mirror -n ssh://git@git.baserock.org/baserock/baserock/definitions /srv/distbuild/gits/tmpWQzN3y
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password,keyboard-interactive).
fatal: Could not read from remote repository.
The issue here is that the user running the continuous build script is
trying to pull over SSH from git.baserock.org, but does not have the
necessary access. Normally Morph should pull using the git:// protocol,
which doesn't require credentials to access.
It turned out that the 'repo-alias' setting was responsible, and was
configuring Morph to pull over ssh:// because the /etc/morph.conf on
that machine specified 'trove-id = baserock'.
The attached patch changes the behaviour to something I find less
confusing: 'baserock' is ignored when specified in the 'trove-id'
list. So 'baserock:' will always expand to 'git://TROVE_HOST/...' for
read access, rather than 'ssh://TROVE_HOST/...'. This means that it
now works in the same way as the 'upstream:' prefix.
Sam Thursfield (1):
Never require SSH access to the Trove where baserock: content is
hosted
morphlib/util.py | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
--
1.9.3
8 years, 7 months
[PATCH] Improve cloud-init.configure to detect cloud-init
by Pedro Alvarez
Repo: baserock:baserock/definitions
Branch: baserock/pedroalvarez/improve-cloud-init-configuration
Sha1: 8470fbec1a26cd39e9621746e7456095327008f5
Land: master
Pedro Alvarez (1):
Improve cloud-init.configure extension to detect cloud-init
cloud-init.configure | 28 ++++++++++++++++++++--------
1 file changed, 20 insertions(+), 8 deletions(-)
--
1.7.10.4
8 years, 7 months
[PATCH 0/3] Move llvm to mesa-common
by Javier Jardón
REPO: git://git.baserock.org/baserock/baserock/definitions.git
BRANCH: jjardon/llvm
COMMIT: fad73985ed710c33f7f0e214d4c58d092b55865c
Mesa is the only chunk that optionally depends on llvm
This series are meant to be on top of the previous
"[V3] Move dbus-glib from genivi-foundation to foundation"
Javier Jardón (3):
Move llvm to mesa-common
strata/mesa-common.morph: Update llvm to latest version (3.5)
llvm.morph: Compile the default options with some exceptions
strata/mesa-common.morph | 9 ++++++++-
strata/mesa-common/llvm.morph | 8 ++++++++
strata/x-common.morph | 8 +-------
strata/x-common/llvm.morph | 8 --------
4 files changed, 17 insertions(+), 16 deletions(-)
create mode 100644 strata/mesa-common/llvm.morph
delete mode 100644 strata/x-common/llvm.morph
--
2.1.1
8 years, 7 months
[PATCH] updates: Allow replacing a file in /etc with a symlink
by Sam Thursfield
Repository: git://git.baserock.org/baserock/baserock/tbdiff
Ref: sam/allow-file-migrations
Sha1: 6cb1a8772364e96e62cfda74490db9848c885635
This change means that it's possible to replace a file in /etc with
a symlink to a file with identical content somewhere else in the
filesystem.
There's still a bunch of stuff in /etc that really belongs in /lib, and
it might be nice to be able to migrate that content sometime.
If a file is replaced with a symlink that points to different content,
baserock-system-config-sync will raise an error, as before, which will
cause `morph upgrade` to abort with an error.
There are a few cleanups overdue on the baserock-system-config-sync
code, including making the test suite a bit easier to comprehend. The
weirdness in this diff is due to the test suite being implemented
directly in the file system.
Sam Thursfield (1):
Allow replacing a file with a compatibility symlink
baserock-system-config-sync/baserock-system-config-sync | 13 +++++++++++--
.../regular.symb.in/systems/default | 0
.../regular.symb.in/systems/factory/orig/etc/1 | 0
.../regular.symb.in/systems/factory/run/etc/1 | 0
.../regular.symb.in/systems/factory/run/etc/afile | 0
.../regular.symb.in/systems/version2/orig/etc/1 | 0
.../regular.symb.in/systems/version2/run/etc/1 | 0
tests/bscs-merge.pass/regular.symb.out/systems/default | 1 +
.../regular.symb.out/systems/factory/orig/etc/1 | 0
.../regular.symb.out/systems/factory/run/etc/1 | 1 +
.../regular.symb.out/systems/factory/run/etc/afile | 0
.../regular.symb.out/systems/version2/orig/etc/1 | 0
.../regular.symb.out/systems/version2/run/etc/1 | 0
.../regular.symb.out/systems/version2/run/etc/afile | 0
.../symblinks.in/systems/factory/orig/etc/os-release | 2 ++
.../symblinks.in/systems/factory/run/etc/os-release | 2 ++
.../symblinks.in/systems/version2/orig/etc/os-release | 1 +
.../symblinks.in/systems/version2/orig/usr/lib/os-release | 2 ++
.../symblinks.in/systems/version2/run/etc/os-release | 1 +
.../symblinks.in/systems/version2/run/usr/lib/os-release | 2 ++
.../symblinks.out/systems/factory/orig/etc/os-release | 2 ++
.../symblinks.out/systems/factory/run/etc/os-release | 2 ++
.../symblinks.out/systems/version2/orig/etc/os-release | 1 +
.../symblinks.out/systems/version2/orig/usr/lib/os-release | 2 ++
.../symblinks.out/systems/version2/run/etc/os-release | 1 +
.../symblinks.out/systems/version2/run/usr/lib/os-release | 2 ++
26 files changed, 33 insertions(+), 2 deletions(-)
rename tests/{bscs-merge.fail => bscs-merge.pass}/regular.symb.in/systems/default (100%)
rename tests/{bscs-merge.fail => bscs-merge.pass}/regular.symb.in/systems/factory/orig/etc/1 (100%)
rename tests/{bscs-merge.fail => bscs-merge.pass}/regular.symb.in/systems/factory/run/etc/1 (100%)
rename tests/{bscs-merge.fail => bscs-merge.pass}/regular.symb.in/systems/factory/run/etc/afile (100%)
rename tests/{bscs-merge.fail => bscs-merge.pass}/regular.symb.in/systems/version2/orig/etc/1 (100%)
rename tests/{bscs-merge.fail => bscs-merge.pass}/regular.symb.in/systems/version2/run/etc/1 (100%)
create mode 120000 tests/bscs-merge.pass/regular.symb.out/systems/default
create mode 100644 tests/bscs-merge.pass/regular.symb.out/systems/factory/orig/etc/1
create mode 120000 tests/bscs-merge.pass/regular.symb.out/systems/factory/run/etc/1
create mode 100644 tests/bscs-merge.pass/regular.symb.out/systems/factory/run/etc/afile
create mode 100644 tests/bscs-merge.pass/regular.symb.out/systems/version2/orig/etc/1
create mode 100644 tests/bscs-merge.pass/regular.symb.out/systems/version2/run/etc/1
create mode 100644 tests/bscs-merge.pass/regular.symb.out/systems/version2/run/etc/afile
create mode 100644 tests/bscs-merge.pass/symblinks.in/systems/factory/orig/etc/os-release
create mode 100644 tests/bscs-merge.pass/symblinks.in/systems/factory/run/etc/os-release
create mode 120000 tests/bscs-merge.pass/symblinks.in/systems/version2/orig/etc/os-release
create mode 100644 tests/bscs-merge.pass/symblinks.in/systems/version2/orig/usr/lib/os-release
create mode 120000 tests/bscs-merge.pass/symblinks.in/systems/version2/run/etc/os-release
create mode 100644 tests/bscs-merge.pass/symblinks.in/systems/version2/run/usr/lib/os-release
create mode 100644 tests/bscs-merge.pass/symblinks.out/systems/factory/orig/etc/os-release
create mode 100644 tests/bscs-merge.pass/symblinks.out/systems/factory/run/etc/os-release
create mode 120000 tests/bscs-merge.pass/symblinks.out/systems/version2/orig/etc/os-release
create mode 100644 tests/bscs-merge.pass/symblinks.out/systems/version2/orig/usr/lib/os-release
create mode 120000 tests/bscs-merge.pass/symblinks.out/systems/version2/run/etc/os-release
create mode 100644 tests/bscs-merge.pass/symblinks.out/systems/version2/run/usr/lib/os-release
--
1.9.3
8 years, 7 months
