Proposal: a new repo to hold the Baserock definitions format specification
by Sam Thursfield
Hi
I'd like to propose a new repo that would be the canonical location of
the Baserock definitions format specification.
Here is what I think it should contain: https://www.github.com/ssssam/spec
And it would live at git://git.baserock.org/baserock/baserock/spec
The repo contains a Makefile that converts the text part to HTML. I
don't have time to immediately do this last part, but it should be
less than a day's faff to set up an hourly cron job on the webserver
that hosts http://download.baserock.org to build 'master' of the spec
and serve the results at http://docs.baserock.org.
I want to do this mainly because I think it makes it more obvious how
to contribute improvements to the format. Everything that needs to be
updated for a new version of the format is in this repo, and so each
proposed change is basically a patch to this repo and can go through
our usual patch review process (possibly on the mailing list instead
of Gerrit -- I think changes to the format need to have as much
visibility as possible).
The reference system definitions in definitions.git would no longer
contain anything to do with the definitions format. Thus it's clear
that the two things are separate, and we can introduce a new version
without necessarily migrating everything immediately.
We've discussed this in the past, and already agreed that the
migrations need to move out of the definitions.git repo (if you're
updating a fork of our definitions, you need to run the migrations
*before* merging 'master' of definitions.git, so keeping them *in*
'master' of definitions.git is really
stupid). Hopefully keeping everything in one repo makes sense to
others. Let me know what you think.
Oh and if anyone can think of a better name than "the Baserock
definitions format specification" that might be handy. (And not BDFS
:-).
Currently we have no overall license info for definitions.git, I
suggest using the same license as the wiki (Creative Commons) for
spec.git. The migrations all contain license info already, so this
just covers the text and the JSON-Schema schemas.
Sam
7 years, 2 months
YBD 16.09 is released
by Paul Sherwood
Mainly this release adds
- support for Definitions Version 8
- warnings for overlapping files (it's a start - more work still
needed)
- some documentation about the current state of releasing for ybd
itself
- foo.no-build as cache_key for 'no-build' mode, to avoid accidental
empty artifacts
32616da Various readme fixes
3393fe7 Revert "Add a gash .travis.yml file"
20108d6 First pass notes on releasing
cfc8289 Add a gash .travis.yml file
5c8f125 fs is now a dependency
4072533 Add support for Definitions VERSION 8
2d7f4c9 Tidyup riemann code to lose some lines
1f66db5 Fix copyright year
da771a5 Split out write_metafile function
ffaf1af No need for chdir
2bbe3d8 Rename regexps => rules, and move logic to compile_rules
e30cf75 Split out compile_regexps function
9f020cd Clusters add to task count too
345498c Pep8
cbbff09 get_metadata requires definition, not path
141d3c5 metafile was not declared
0a009b0 Set no-build cache_keys to '.no-build'
e55ba30 No need for 'safename', definition['name'] is now safe
76bb5ae Fix misplaced bracket from 6f0533f
4659eb3 WIP - report where the overlaps are
cafab9e Only report overlaps once
6f0533f Use setdefault
7055554 Update readme with pyfilesystem dependency
4797a47 Merge pull request #179 from locallycompact/lc/osfs-broke
38b8df4 Revert "Use setdefault"
81dc395 Revert "Use OSFS in set_mtime_recursively"
1b85701 Missed bracket
f67fcb1 Merge pull request #178 from locallycompact/lc/osfs
4a4bb06 Dictionary notation
7f5e038 Replace mark_used_path with lambda expression
3da20cc Use OSFS to walkdirs
ba8cdd0 Use OrderedDict comprehension
8f1fd90 Use OrderedDict for regexps
1506a05 Chunk products is an array, not a dict
4503ce4 Use MultiFS in find_extensions
8066267 Use OSFS in detect_format
974a6b9 Use OSFS in set_mtime_recursively
3fd7e8e Use setdefault
3d93738 Code reduction in load_schemas
3af68b7 Report which meta files contain overlaps
c2f7c0e Report overlapping files when creating systems
7 years, 3 months
YBD & Aboriginal - progress report
by Tristan Van Berkom
The milestones outlined in the previous mail are not exactly complete
but as I've been off on my own for a little while it's important for me
to share my status somewhere, so here it goes.
Aboriginal Build Sandbox
~~~~~~~~~~~~~~~~~~~~~~~~
This was my first milestone, and basically consists of setting up a
chroot environment inside the aboriginal/qemu runtime where:
o Binaries required for running scripts are all statically linked
(this avoids the trouble of setting up any runtime libs in the early
build stages)
o The distcc compiler/toolchain is bind mounted into the
chroot so that we can compile.
This was fairly simple to accomplish and is done, have a fancy little
sandboxing script which assumes that some artifacts may have been
staged at a given directory and then sets up any missing pieces and
runs a chroot on it.
The script in question can be found here:
https://github.com/gtristan/aboriginal-controller/blob/master/control/b
in/sandbox
It works well when run in my aboriginal fork's 'extra-changes' branch
here:
https://github.com/gtristan/aboriginal/tree/extra-changes
Automating the aboriginal sandbox
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is currently very ugly but works.
Basically it assumes that there is an image with a directory containing
some staged artifacts and something to build, it just wraps the
aboriginal emulator startup scripts, runs a specified command (which
can be the invocation of a build script) inside the emulator
environment and propagates back the return value of the commands run in
the sandbox in the emulator.
I have a starting point for this in the same git repo above here:
https://github.com/gtristan/aboriginal-controller/blob/master/aborigina
l-sandbox
I've come to realize this approach really has to change sooner than
later, more on that later in this email.
NOTE: Eventually this 'controller' repository should assume that the
image is always running and be able to accept commands over some simple
IPC, run those in a sandbox and report the result. This approach will
also assume there is some file system sharing going on.
Building the stack
~~~~~~~~~~~~~~~~~~
In my previous mail I put this milestone later, but as I was putting
together these sandboxing scripts I got ahead of myself because I
wanted to test that this was really going to work.
This is where I made most progress, and the outlook is good, I was able
to build my own interpretation of build-essential in aboriginal and
have some build instructions saved which perform the following steps:
o Build static gawk against musl libc
o Build static sed against musl libc
o Install linux kernel headers
o Build glibc (requires some patches)
o Build bash against new glibc
o Build GCC runtime libraries including libstdc++
against the new glibc
After these steps are complete, I have tarballs (artifacts) of each
component... when they are staged and the sandbox script is run with
the new /lib/ld.so runtime linker specified, we can now compile C and
C++ programs against the fresh new runtime using the musl-based
compiler over distcc.
Hacking YBD to use the aboriginal sandbox
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Now this is where I am having trouble.
My original plan as a first iteration was to 'simply':
o stage the artifacts into a mounted disk image
o serialize the build commands into a script on that image
o run the aboriginal wrapper script asking it to execute
that script in the staging area
o Collect the output and create an artifact
It turns out that ybd requires more changes than I expected to make
this first iteration work.
So, since the eventual and saner approach is to use some filesystem
sharing (either nfs or ideally the virtfs qemu option), and since
having a shared directory would dramatically reduce the amount of
changes required to ybd, I started to try to get that to work.
But this is where I fall flat on my face, firstly even though the
aboriginal kernel is compiled with the relevant virtio and 9p options
enabled, I was unable so far to mount the virtfs share from the
aboriginal image.
I also tried to compile the aboriginal kernel with nfs support and
export a directory using nfs-kernel-server on my host, but was unable
to find the magic incantations to nfs mount a share from the image
either.
Networking is really my weakest point, when someone says the words
routing table or gateway, my brain just starts to fry and I am unable
to retain any useful information, it's like an allergy.
So in summary
~~~~~~~~~~~~~
o The concept works well, we can build packages against the libc of
our choosing using the cross compiler and aboriginal bootstrapping
process.
o Plugging this into YBD is tricky especially without any kind of
nfs.
o I'm trying to get the fs sharing working (nfs/virtfs or otherwise)
but I could use a hand, I have a feeling that this could be much
easier for someone who knows networking better than I do.
Cheers,
-Tristan
7 years, 3 months
[PATCH 0/2] Definitions V8
by Daniel Firth
This is here:
https://github.com/locallycompact/definitions/tree/lc/defsv8nu
And works with the ybd at this branch:
https://github.com/locallycompact/ybd/tree/lc/submodules
Dan
Dan Firth (2):
Add migration tool for explicitly declared submodules
Added submodules field for chunks with submodules; Updated tratum
schema to match
VERSION | 2 +-
migrations/008-submodules-in-strata.py | 219 ++++++++++++++++++++++++++++
schemas/stratum.json-schema | 9 ++
strata/ansible.morph | 9 ++
strata/ceph-service.morph | 15 ++
strata/chef.morph | 6 +
strata/core.morph | 11 ++
strata/coreutils-common.morph | 17 +++
strata/devtools.morph | 3 +
strata/foundation.morph | 3 +
strata/genivi-demo-platform.morph | 5 +-
strata/gnome.morph | 46 +++++-
strata/multimedia-gstreamer-0.10.morph | 9 ++
strata/multimedia-gstreamer.morph | 17 +++
strata/multimedia-hardware-codecs-x86.morph | 3 +
strata/openstack-services.morph | 20 +++
strata/ostree-core.morph | 5 +
strata/qt4-sdk.morph | 3 +
strata/qt5-sdk.morph | 3 +
strata/qt5-tools.morph | 3 +
strata/tools.morph | 3 +
strata/trove.morph | 19 +++
strata/vala-common.morph | 3 +
strata/virtualization.morph | 26 ++++
strata/x-common.morph | 12 ++
strata/xdg-app-common.morph | 3 +
26 files changed, 471 insertions(+), 3 deletions(-)
create mode 100755 migrations/008-submodules-in-strata.py
--
2.6.4
7 years, 3 months
YBD 16.08 is released
by Paul Sherwood
Various fixes, and addition of schema validation modes:
- False: no validation
- strict: error and if any validation fails
- anything-else: warn and continue
This was in part prompted by ssam's proposal to migrate the spec and
schemas to a new place.
e9f3795 Log schema-validation mode
cda17b1 Udpate readme on schema values
a40635a Add schema validation - off by default
4c97541 c03a482 was double-counting :/
2dfd790 Support upgrade-location as well as location
c03a482 Hack fix for #175
7cd7485 Files => Artifacts
538211b Partial fix for #170
7784bf2 Brute force fix for #164
ff544e3 Remove unused import
e3069f9 Add defaults into the cache-key calculation
35b9d8e Lose a line
7cd474b Merge pull request #166 from benjamb/benbrown/fix-fetch
96dda36 Revert "Revert "Call update_mirror() from get_tree() and
mirror()""
0b3d878 Prune stale branches while fetching to prevent conflicts
5454415 Revert "Call update_mirror() from get_tree() and mirror()"
a212ae9 Allow for log-verbose missing entirely
7 years, 3 months
YBD 16.07 is released
by Paul Sherwood
Mainly this has been about some tidyups, adding SOURCE_DATE_EPOCH
support, and getting 'reproduce: True' mode to output useful stuff.
FWIW the components in ci.morph x86_64 seem to be about 40% bit-for-bit
reproducible [1] - i would expect this to increase significantly if/when
we create an expressly b4b reproducible toolchain.
https://raw.githubusercontent.com/devcurmudgeon/build-logs/df90c503b0c62c...
90beed4 Tider list
2f8154f Print list of reproduced items
1765c92 Remove spurious print, fixes #165
a6ecec1 Eek - medata => metadata
f4c513e Trap and report exceptions in single-instance claim
b83d175 Strip / from any definitions name
088f9ad Shuffle metadata logic a bit
50a4d2c Move splitting.load_metafile() => cache.get_metadata()
2b961f3 Remove a few lines
f6fe7b3 No need for chdir in write_stratum_metafiles
90046f7 Merge pull request #160 from benjamb/benbrown/non-bare-repos
b902dae Call update_mirror() from get_tree() and mirror()
11cf753 update_mirror() no longer requires sensible config to update
e970fd1 Pep8
b89174a Make default build_system NOT FOUND work correctly
cd9bca4 Claim logic now works for instances: 1, too
a3960ba Export TMPDIR for deployment extensions
a3d80e0 Merge pull request #159 from locallycompact/lc/sandboxfix
d865730 Sandbox setup is a contextmanager
5fbacd9 Add support for SOURCE_DATE_EPOCH
b312a95 Hopefully a fix for 'template not found'
d7b565e Gah... missed from b110c85
07f07f4 Merge branch 'master' of git://github.com/devcurmudgeon/ybd
b110c85 Fix a couple of Server Error 500
feaf848 If we're run somewhere (eg ybd) alongside a definitions dir, go
there
c142ff1 Call get_cache to ensure unpacked directory exists
886cadc Oops - remove errant debug print
a3c591e Merge pull request #156 from
mwilliams-ct/mwilliams/riemann-events
bbd1128 Update readme with details on Riemann
aa5398e Adds Riemann support
7 years, 3 months
Removing the requirement that Morph from Baserock definitions release N can build Baserock definitions release N+1
by Sam Thursfield
Hi all
Rather embarrassingly, it turns out the patch I did to add support for
definitions version 7 to Morph wasn't actually correct:
https://storyboard.baserock.org/#!/story/72
Javier now wants to add a new build system to DEFAULTS in definitions
and use it: https://gerrit.baserock.org/#/c/1786/
Previously we had a rule about the reference system definitions: that
the version of Morph in release N had to be usable to build release N+1.
This often created lots of extra work, but it was necessary because of a
design choice in Baserock made years ago, that we would only support
running the Morph build tool within the Baserock reference systems.
That made sense on some fronts (we could have wasted endless amounts of
time trying to make Morph work on every possible distro) but also
created a horrible circular dependency for anyone trying to upgrade
Baserock build infrastructure. Imagine: you have an ARM distbuild
network running the 'build' system from baserock-14.x. You want to
update to baserock-14.y but your build network can't *build* that version.
Right now Tristan is working on cross compile support for YBD, which
would basically remove this problem. You could bootstrap the new version
of Baserock from an x86_64 machine and update the distbuild network that
way. This would also allow updating from (theoretically) any release to
any other release, too, which is much more practical than the current
"update one step at a time" approach that we theoretically require.
So, I propose relaxing that rule for the time being, to avoid us having
to block Javier's patches until someone has time to create and release
new VM images of the reference systems.
Please speak up you think this affects you!
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
7 years, 3 months
[PATCH] Go lorry
by Daniel Firth
Daniel Firth (1):
Added new go lorry
open-source-lorries/go-git.lorry | 6 ++++++
1 file changed, 6 insertions(+)
create mode 100644 open-source-lorries/go-git.lorry
--
2.6.4
7 years, 3 months
[PATCH] Added ansible playbook and systemd service for spinning up riemann
by Daniel Firth
sha1: 6c463f7681c1a3b8d233dae2ff0590aa422f7ea6
repo: baserock/baserock/infrastructure
Dan Firth (1):
Added ansible playbook and systemd service for spinning up riemann
baserock_riemann/instance-config.yml | 64 ++++++++++++++++++++++++++++++++++++
baserock_riemann/riemann.config | 29 ++++++++++++++++
baserock_riemann/riemann.service | 16 +++++++++
3 files changed, 109 insertions(+)
create mode 100644 baserock_riemann/instance-config.yml
create mode 100644 baserock_riemann/riemann.config
create mode 100644 baserock_riemann/riemann.service
--
2.6.4
7 years, 3 months