April 2016 - baserock-dev - listmaster.pepperfish.net

by Baumann, Christoph (C.)

Hello, currently I'm struggling with setting up a trove. The building in Baserock (16.13) and also deploying it to VirtualBox worked. After some struggle with the correct network settings I was able to log in and noticed that systemd failed to bring up trove-setup: " Apr 28 11:55:24 durlach-trove ansible-playbook[331]: fatal: [localhost] => Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the remote directory. Consider changing the remote temp path in ansible.cfg to a path rooted in "/tmp". Failed command was: mkdir -p $HOME/.ansible/tmp/ansible-tmp-1461844524.24-189921729673453 && echo $HOME/.ansible/tmp/ansible-tmp-1461844524.24-189921729673453, exited with result 127 Apr 28 11:55:24 durlach-trove ansible-playbook[331]: TASK: [trove-setup | fail msg='TROVE_ID is mandatory'] ************************ Apr 28 11:55:24 durlach-trove ansible-playbook[331]: skipping: [localhost] Apr 28 11:55:24 durlach-trove ansible-playbook[331]: FATAL: all hosts have already failed -- aborting Apr 28 11:55:24 durlach-trove ansible-playbook[331]: PLAY RECAP ******************************************************************** Apr 28 11:55:24 durlach-trove ansible-playbook[331]: localhost : ok=0 changed=0 unreachable=1 failed=0 " After calling trove-setup in the login shell (root) the setup started and after a while reported success. Then the mirroring of the upstream trove that was configured in the build process started. Also the web interface came up. But there seems to be an issue with git. The web interface reports "Repository seems to be empty" for any repository I click on. And also 'git clone' fails with a similar error. But: if I use the ssh ID of the lorry user the checkout succeeds. This also works to checkout the trove-config. But it fails to push changes. Further the git user has /bin/bash as login shell and no commands for git-shell are available. Could it be that running trove-setup from an interactive root-shell broke the setup? Regards, Christoph

7 years, 5 months

2
1
0 / 0

Live Updates of Downstream Troves

by Ben Brown

Hello all, I'm currently trying to work out a sensible way of allowing updates to an upstream trove's repositories to propagate down to any downstream troves as they happen. The idea behind this being that there is a team of developers working from multiple (possibly cross time zone) sites on the same project, that may not have the most reliable or speedy network. For the purposes of speeding things up, we can provide a trove at each site to synchronise repositories across the sites, allowing local development to continue, and also (hopefully) making collaboration between sites easier. To further this "story", a CI pipeline would be running at the "master" site. that we wouldn't want to cause a bottleneck in with work done at the others sites, they would instead have their own CI setup. Currently, the solution would be to provide troves at each site, with one site designated as the "master" trove, which will lorry back anything from the downstream troves in a separate namespace. Enjoy this ASCII art: master / \ / \ slave1 slave2 The current issue with such an idea being that you would configure lorry-controller to scan for updates, say, every 2 hours, or higher depending on the number of repositories to ensure it isn't constantly playing catch-up; lorry-controller can be quite resource intensive. This would not be speedy in any sense of the word, since it could be 2 hours or longer before the repository is updated from upstream. The only way I believe something like this would currently be possible is to hack gitano/bin/gitano-post-receive-hook to run `ssh root@trove 'curl -X POST -d "path=repo/path" http://localhost:12765/1.0/move-to-top` for each site, which would work, providing you've added the public key of the master trove to authorized_keys, but this results in the upstream trove requiring knowledge of a(ny) downstream troves, and regardless is a truly yucky hack. My rather vague idea is to do some work on Gitano that allows you to register some form of an event listener, with an address, port and namespace (or list of repositories) you wish to receive updates from. Then on specific git events (post-receive would be the plan for now), would send, for example, "delta/pip master oldsha newsha" to any registered listeners. Something would then be written for lorry-controller which would listen for these events and act accordingly. This definitely needs some ironing out, however it's possible this has been overthought (or not thought out enough), and it could be simpler and easier to just configure lorry-controller to constantly cycle through its repositories. Any ideas and/or feedback is welcome. Ben

7 years, 5 months

2
1
0 / 0

Problems building reference system definitions

by Baumann, Christoph (C.)

Hello, I'm using the Baserock 16.13 chroot and tried to build trove-system-x86_64 of the reference system definitions. Morph produced this error: " 2016-04-27 10:53:53 Looking for uncommitted changes (pass --local-changes=ignore to skip) 2016-04-27 10:53:53 Creating temporary branch in /src/Workspace/definitions-master named refs/heads/baserock/builds/master/4fcbeb23acf241308a417dd26e9fa2ab 2016-04-27 10:53:54 Deciding on task order Traceback (most recent call last): File "/src/morph-master/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph-master/morphlib/app.py", line 299, in process_args cliapp.Application.process_args(self, args) File "/src/morph-master/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph-master/morphlib/plugins/build_plugin.py", line 207, in build with source_pool_context as source_pool: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/src/morph-master/morphlib/definitions_repo.py", line 199, in source_pool original_ref=original_ref, status_cb=status_cb) File "/src/morph-master/morphlib/sourceresolver.py", line 506, in create_source_pool definitions_original_ref=original_ref) File "/src/morph-master/morphlib/sourceresolver.py", line 440, in add_morphs_to_source_pool add_to_pool, predefined_split_rules) File "/src/morph-master/morphlib/sourceresolver.py", line 287, in _process_definitions_with_children morphology = get_morphology(filename) File "/src/morph-master/morphlib/sourceresolver.py", line 282, in get_morphology filename) File "/src/morph-master/morphlib/sourceresolver.py", line 255, in _get_morphology morph = morph_loader.load_from_string(text, filename) File "/src/morph-master/morphlib/morphloader.py", line 395, in load_from_string obj = yaml.safe_load(string) File "/usr/lib/python2.7/site-packages/yaml/__init__.py", line 93, in safe_load return load(stream, SafeLoader) File "/usr/lib/python2.7/site-packages/yaml/__init__.py", line 69, in load loader = Loader(stream) File "/usr/lib/python2.7/site-packages/yaml/loader.py", line 24, in __init__ Reader.__init__(self, stream) File "/usr/lib/python2.7/site-packages/yaml/reader.py", line 85, in __init__ self.determine_encoding() File "/usr/lib/python2.7/site-packages/yaml/reader.py", line 124, in determine_encoding self.update_raw() File "/usr/lib/python2.7/site-packages/yaml/reader.py", line 178, in update_raw data = self.stream.read(size) AttributeError: 'NoneType' object has no attribute 'read' " I tried updating morph and pylru to current the latest version in the git, but the problem persisted. Am I missing something here? Regards, Christoph

7 years, 5 months

2
3
0 / 0

YBD + Aboriginal status update and HOWTO

by Tristan Van Berkom

Hi all, Warning, this email will be a bit long... I've included a new and revised list of milestones towards getting the ybd-aboriginal-baserock thing into a mergable state. Also, I have a prototype so I've included instructions at the end of this email for anyone who might be interested in trying to reproduce this setup. As an additional note which doesn't fit anywhere else, I've discovered another, surmountable obstacle just yesterday, which is concerning building baserock with aboriginal as a regular user and dropping the requirement to run the build as root. What I have found is that some of our build instructions try to install things explicitly as root, instructions such as: install -m644 -o root -g root ... These instructions work when run in an emulator where the emulator has a filesystem image mounted and is installing in a chroot on that filesystem. These instructions however do not work when running in the emulator and installing onto a 9p virtfs mount (a mount of a shared host directory), this is because in the virtfs share scenario, the emulator can only really write to the target filesystem using the privileges it has. I think that essentially this can be worked around by: o Disallowing any install-commands which try to explicitly install things as the root user/group. o Performing some additional chown commands in the system-integration commands. This would be a practical approach because using a virtfs mount for the builds is very practical, as one need not create fixed size filesystem images to slave to each qemu worker, and then extract the content of those fs images in between builds. However, once all of the artifacts are prepared for a given system image, it can be trivial enough to create a fixed size, large filesystem image for the emulator to perform the system integration commands on. Revised status and milestones ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Integration with YBD ~~~~~~~~~~~~~~~~~~~~ Some progress here already, still things to do: o Currently one must specify the emulator & controller directories in the ybd.conf file, this needs to be smarter so that depending on the target arch ybd uses the correct emulator for that arch... somehow. o YBD is not shutting down the emulators when exiting yet. o Eventually we probably want to absorb parts of the aboriginal-controller scripts into YBD and upstream some of this to aboriginal startup scripts proper, but it's not a hard blocker. Build base system ~~~~~~~~~~~~~~~~~ Currently we have most definitions for build-essential ported, build essential at this moment builds glibc, I would have ported my gcc runtime library builds by now (which has been tested and works) except that it takes a very long time to build and validate this. Once this is done, need to get through a build of a complete base system. Run Deployment in the emulator ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This item will essentially let us have a more robust deployment technique which is not dependent on host tooling in any way. The basic idea at this stage is to create a special chroot environment inside the emulator where we stage all the artifacts necessary to run the build extensions in a given deployment, and run the deployment in that chroot. o need to figure out how to identify which chunks are required to be staged, this may be hard coded in a first pass before becoming a part of the specification. o need to figure how to run the extensions inside a staged build area in the emulator in order to deploy the image, probably this means staging the build extension scripts into the same chroot area. Avoiding the emulator for native builds ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Need to sort out how we can optionally use a simple chroot and avoid the emulator workers. Here we'll want to use the same aboriginal compiler and the same build-essential of course. Aboriginal already has a 'chroot-splice.sh' script which sets up a directory with it's compiler tooling suitable for chrooting into which we should be able use almost as is. This should be fairly simple, possibly we will want the option to keep using the emulator even for a native build only in the final system integration and deployment stages. Doing this in combination with fixing any issues we have with linux-user-chroot would most probably allow us to shed the root requirement even for native builds. Need to assert the version of the aboriginal image from the definitions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In the interest of perfect reproducibility, we dont want arbitrary versions of the compiler used in conjunction with definitions and ybd. So, we will need a way to have the definitions themselves assert the exact version of sources used in the aboriginal kit and use a very specific version. There are a few ways to skin this cat, from creating a versioning scheme for the aboriginal compiler builds to just committing compiled copies of the aboriginal kit (they are relocatable) and specifying an sha1sum of the kit we want to use. NOTE: A more complex approach would be to internalize the aboriginal build completely and integrate it as a part of build essential. Personally I would rather find a way to cooperate with upstream Aboriginal as much as possible and avoid duplicating it inside of Baserock. Implement important arches in upstream aboriginal ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is not strictly a part of the exercise, but it should be noted that aboriginal has support for the following arches: armv4l, armv4tl, armv5l, armv6l, i486, i586, i686, x86_64, m68k, mips, mips64, mipsel, powerpc, powerpc-440fp, sh2eb, sh2elf, sh4, sparc A hand full of these have not been ported to build with musl libc in aboriginal and as such will not build with the new GCC 5.3 toolchain. Some of these build for musl but still need some minor tweaks to aboriginal sources. I have only tested arm and mips target arches. Most importantly, we're lacking armv7, armv7lhf and aarch64. HOWTO Build (for now) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are now a few moving parts, in order to try this out you are going to need the following components: Step 1, Getting the source ~~~~~~~~~~~~~~~~~~~~~~~~~~ Aboriginal fork ~~~~~~~~~~~~~~~ URL: https://github.com/gtristan/aboriginal Branch: extra-changes This fork contains all the parts of the work intended for upstreaming to aboriginal proper (port to gcc 5.3, some patches to the kernel config), some things from the controller (below) may be added to this downstream over time as well. Aboriginal Controller ~~~~~~~~~~~~~~~~~~~~~ URL: https://github.com/gtristan/aboriginal-controller Branch: master This is a wrapper for the aboriginal launch scripts, it allows running the qemu environment in the background as a build slave and sending some commands to it over an IPC. YBD Fork ~~~~~~~~ URL: https://github.com/gtristan/ybd/ Branch: aboriginal My downstream ybd work branch which contains the experimental support for now. Definitions wip branch ~~~~~~~~~~~~~~~~~~~~~~ URL: https://git.baserock.org/git/baserock/baserock/definitions.git Branch: baserock/tristan/wip/aboriginal Step 2, Building stuff ~~~~~~~~~~~~~~~~~~~~~~ First, remember to take note of the branches I pointed out in the previous section. First build aboriginal ~~~~~~~~~~~~~~~~~~~~~~ You will have to choose 2 things at this point: - HOST: The arch you are building on - TARGET: The arch you are building for These are file names in ${ABORIGINAL}/sources/targets, but note that not all targets are supported with the newer gcc 5.3 toolchain we require. You can safely use intel arches, armv5l (and probably all the supported arm variants) and mips. To build, choose your host and target and run: cd /path/to/aboriginal ENABLE_GPLV3=1 \ CROSS_COMPILER_HOST=${HOST} \ SYSIMAGE_TYPE=ext2 \ ./build.sh ${TARGET} A reasonable invocation is: ENABLE_GPLV3=1 \ CROSS_COMPILER_HOST=x86_64 \ SYSIMAGE_TYPE=ext2 \ ./build.sh armv5l This will take anywhere between 30min (modern x86_64 quad core cpu with plenty of SSD) to 4hours to complete (reported for older dual-core x86 laptop and regular HDD). Now build the controller ~~~~~~~~~~~~~~~~~~~~~~~~ To build the controller, you need to have mksquashfs installed. To run the controller, you will need socat installed. To build, just: cd /path/to/aboriginal-controller make And your done. Configure YBD ~~~~~~~~~~~~~ For now ybd is hooked up to the emulator in a very nasty way, it needs to know both the path to the emulator (aboriginal system image) and it needs to know the path to the controller. Edit the file /path/to/ybd/ybd/config/ybd.conf and add the following: aboriginal-controller: '/path/to/aboriginal-controller' aboriginal-system: '/path/to/aboriginal/build/system-image-armv5l' To run multiple emulators in parallel, you should remember to also add: instances: 2 NOTE: The aboriginal system image and cross compilers dont really have to stay in the aboriginal build directory, but the cross compiler must be *beside* the system image wherever it is located. Build build-essential using YBD / Aboriginal ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now you can run YBD in the regular way, assuming you have the correct definitions branch checked out: cd /path/to/definitions /path/to/ybd/ybd.py strata/build-essential.morph armv5l The first time you run this, startup will take at least a minute, this is because the first time you startup an emulator it will create a 1GB swap file. Subsequent YBD invocations once the swap is built will just pause for the time it takes to boot up the emulators. Logging is copied to the regular place after each build command is run, however if you are running a long lived build command and want to observe the output, you can look in the staging directory which should be somewhere like: ~/ybd/tmp/<tmpdir>/build.log I did have a 'tail -f build.log > realoutput.log' running on the host system at one point, but this was just expensive for nothing so I removed that.

7 years, 5 months

4
14
0 / 0

Baserock Concourse ci pipelines are now live at http://concourse.baserock.org

by Paul Sherwood

Hi all, as described at [1], Codethink has contributed implementation of Concourse pipelines for the Baserock project. Basically this involves - parsing Baserock definitions and generating Concourse manifests - deploying the results to live Concourse infrastructure The initial code is being developed on Github at [2] and you can see the pipelines themselves at [3]. Note that some browsers/machines may struggle to render them. Chrome on an x86 machine is recommended. The actual pipelines are using ybd [3], and are populating the default artifact cache for ybd at [4]. The current implementations are not optimised for performance or efficiency, so I expect we'll be tweaking things over the coming months. br Paul [1] http://www.codethink.co.uk/blog/articles/2016/visualising-the-software-in... [2] https://github.com/perryl/concourse-scripts [3] https://github.com/devcurmudgeon/ybd [4] http://artifacts1.baserock.org:8000

7 years, 5 months

1
0
0 / 0

Adding GitLab support to lorry-controller

by Ben Brown

Hello all, I've recently (on Monday) sent a patch series to gerrit[1] to allow the mirroring of source code to and from a GitLab server. The ability to specify a GitLab instance as an upstream trove, rather than creating a lorry per repository certainly makes things easier. However it might make some sense to split this into a new gitlab "type" within lorry-controller.conf (feedback appreciated). Allowing you to use GitLab in the same way Baserock currently uses gerrit, i.e. GitLab hosts the repo for code review (or CI), (while not something Baserock will specifically make use of) provides another option for any who prefer GitLab's UI/features. Further Work ------------ * Allow the setting of default refspecs for a trovehost. Any repos found on a trovehost are converted into lorry files that are stored in lorry-controller's database with the default refspecs set to '+refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*', so "force push(pull) all refs" essentially. This is fine when the trovehost is actually set to a trove, and one that you trust, however you may want to disallow the upstream trove (/gitlab instance) from changing history. * Allow lorry-controller to mark branches as protected? Running this on a GitLab instance means you miss out on the protection that gitano provides, having a setting that allows lorry-controller to set refs it knows about as protected would replicate this behaviour to a certain degree. This is probably a little outside lorry-controller's scope however. There may be other wackier ideas later down the line, but for now this is all. Any feedback is welcome. Ben [1] https://gerrit.baserock.org/#/q/topic:benbrown/gitlab

7 years, 5 months

2
2
0 / 0

YBD repeatable builds

by Edward Cragg

I have been working on trying to ensure that YBD builds the same thing each time it makes a Baserock system. There are some notable occasions when this has been shown to go wrong, e.g. when systemd init finds itself overwritten with busybox init, and the system doesn't boot as expected. I wrote a test [1] which repeatedly builds a system, and analyses the contents of it using checksums. This has shown that, as long as instances > 1 (when deliberate randomisation is enabled), YBD will build a system containing different files on each successive build, for 100% of builds. It shows that packages are being overwritten by others, randomly. It seems that this is happening in the 'install contents' part of the build, and the following patch fixes the test: diff --git a/ybd/assembly.py b/ybd/assembly.py index 02efda1..d8afd8e 100644 --- a/ybd/assembly.py +++ b/ybd/assembly.py @@ -165,7 +165,6 @@ def install_contents(defs, component): '''Install recursed contents of component into component's sandbox.''' def install(defs, component, contents): - shuffle(contents) for it in contents: content = defs.get(it) if os.path.exists(os.path.join(component['sandbox'], 'baserock', This disables the shuffling for all builds, not just systems, since i haven't been able to find a sensible way to avoid it for only a system build. However, i feel it is likely to be a sensible thing to disable this shuffling behaviour even for stratum and chunk builds, since the same random overwriting may also affect the sandboxes for these builds in a similar way as in systems. The payoff, however, is that in disabling shuffling, there are indications that the build takes four times as long as with it. I'd be grateful for any thoughts on this, whether it is worth looking further into disabling the disabling randomisation, and optimising the build time, or if there might be any alternative approaches. [1] https://github.com/nuxeh/ybd-check-repeat-build

7 years, 5 months

5
11
0 / 0

concourse.baserock.org - Downtime Tuesday 19th April

by Mike Williams

Dear Baserockers, As some people know, we have set up a test deployment of Concourse for Baserock systems. This notice is for all users with access to this test deployment. Tomorrow, Tuesday 19th April 2016, at 1830 UK time (GMT+1), I will be updating this system to Concourse version 1.1.0. I hope that downtime will be extremely short, but please consider the system at risk of downtime until 2030 UK time. I will reply to this list to confirm when this is done. If this time is deeply inconvenient for you, please let me know and we can try to reschedule. Regards, Mike

7 years, 5 months

2
2
0 / 0

Templating and conditionals in definitions

by Sam Thursfield

Hi all I've prototyped an simple approach to having conditionals and templating in Baserock definitions. You can see what I'm proposing in this branch: <http://git.baserock.org/cgit/baserock/baserock/definitions.git/log/?h=sam...> Files that need templating are now called .morph.in, rather than .morph. Before using definitions.git, you need to run `./configure`, which generates the real .morph files. You can run ./configure multiple times to create different configurations. For example, run `./configure x86_64-generic` to create build-system.x86_64-generic.morph, and then run `./configure jetson` to create build-system.jetson.morph. The 'configure' script has various named configurations built into it based on what already existed in the repo. It's also possible to specify custom configurations, e.g: ./configure armv8l64-chroot --arch=armv8l64 --bsp=None There are not many config flags right now, and it should stay that way, but we can see a few more useful ones, like x11/wayland flags for the GNOME system. Underneath it uses Jinja2 templates to substitute variables for their values. You can probably do quite clever stuff with them, i've kept it simple in this example, but more can be done. I have two questions in my mind right now... 1. is the Jinja2 template language a good one? Should we use a different preprocessor system, like GNU m4? Or should we use Nix expression language? Should we try and come up with our own simple DSL for definitions? I don't feel too strongly about this, except that we should avoid having generic looping constructs or any kind of side effects. 2. Should this be built into definitions format, rather than being an add-on? The Baserock definitions format is currently a good serialization of the actual build+integration instructions, in a way that can be used more or less directly by tools. The addition of conditionals, branching and templating means that is no longer true. For tools to be able to work on the data, there needs to be a step that goes from the human-readable input to the machine-readable data. Right now that's the 'configure' script. If we encode these templating rules into the spec, then effectively each tool that wants to access the definitions data needs to have the 'configure' script built into itself. On the other hand, code generation is a bit ugly, and indeed it can lead to the most horrific toppling towers of hacks: see the GNU Autotools. --- There have been a couple of previous thoughts on this. Tristan proposed 'flavours': those allow definining multiple variants of a component in a single chunk .morph file, and merging the 'forks' in the dependency graph that result. I still like that idea, and I think what I'm proposing doesn't really conflict. It's discussed here: <https://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/201...> A discussion of templating from 2014 is here: <https://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/201...> Let me know what you think. I haven't tested the resulting branch very hard, I will definitely have broken stuff. I do have the minimal-system.x86_64-generic.morph system building with YBD though and it seems to be going fine so far. Sam -- Sam Thursfield, Codethink Ltd. Office telephone: +44 161 236 5575

7 years, 5 months

3
4
0 / 0

YBD 16.16 is released

by Paul Sherwood

x86_32 builds work again! plus various improvements to documentation. 1cae701 result => expected.result 3ec6636 [ci-skip] Improve documentation of config vars, move it all into conf file 2dc8be1 [ci-skip] explain culling behaviour in ybd.conf 3e30a0b [ci skip] explain artifact-version use-cases 3f9a7f5 Adding "workers" directory to the config a16ad76 Remove comparison with morph section a291975 Add notes about 'instances' variable 3f6d588 Move cpus map into defaults.conf 18ac1e9 Fix for x86_32 dict was the wrong way round 6c9cb44 Add comments to ybd.conf 696d949 More testing needed on bit-for-bit reproducibility of stratum artifact

7 years, 5 months

2
4
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

baserock-dev April 2016