Trove services vs GitLab
by Paul Sherwood
Hi all,
on a current project I've been asked to document what I mean by 'Trove
services' that are not available from GitLab by default.
The ones I can think of (and maybe GitLab has or could easily have
equivalents) are:
- tarfile of a base version of each repo, so that clones can start with
that
(which can significantly reduce the initial cloning time for big
repos...)
- an api to return the tree for a given committish, which means
YBD/morph can avoid cloning repos to calculate cache-keys
- lorry/lorry-controller
- do conversion from (svn, hg, tar, bzr etc) where required as well
as periodic mirroring
- mirror frequency can be scheduled
- all their config is provided as json into git
- all Trove config (and changes to config, including adding users,
permissions etc) are stored in git, so there's full audit trail on this
too
- Troves do not allow re-writing of history by lorry/lorry-controller,
so we can spot when upstreams misbehave
- namespacing (ie we can have delta/*, delta/openstack/*,
delta/gnome/*, baserock/*, baserock/*, customer/*, customer/project/*,
customer/supplier/* etc)
- software catalog: fast cgit, configured to support navigating the
whole set QUICKLY in a browser eg:
- find all commits by <name> in the linux repo
- go to a specific checkout of weston, and browse the source
- cgit linkbacks for navigating definitions
Are we aware of any others?
br
Paul
6 years, 3 months
YBD 16.11 is released
by Paul Sherwood
This release includes
- better cull behaviour (now ybd won't delete artifacts needed for
current run)
- some documentation about planned regression tests
- new mode: [keys-only, no-build, normal]
- the beginnings of .gitlab-ci.yml
dc23f7c Fix pep8 issues
6c56563 Add pep8 test
d2da5b8 Start testing.md, and move to docs/
32dc673 Put cache_keys in .trees file, print target key if mode:
keys-only
6cf956a New mode: [keys-only, no-build, normal]
ef2cd4d Add installation test
0a2240a Run ybd without args is success, show version too
2f26e7c More notes on test cases for ybd
d247047 Add a .gitlab-ci.yml file
42b814d Some improvements to releasing.md
d793179 Revert "Change default min-gigabytes config to 0"
f3cf4bf Log last upload and password fail times
f52a8e2 Make autodetection a WARNING
57a3777 Use tar -tf to check artifact, instead of -xf
b3751ff Do not cull things we need for this run of ybd
9be72d2 Change default min-gigabytes config to 0
6 years, 3 months
YBD Aboriginal status
by Tristan Van Berkom
This will be a shorter status update message than last week...
What progress has been made:
o Worked out build errors up to coreutils.morph
Nothing except for build-essential has been modified in
my definitions branch, some errors I needed to work through
here are that I had omitted busybox from build-essential
and was relying on toybox instead.
Before the virtfs change, busybox was not running (as it
could not be setuid 0, for some reason busybox wants this)
Now that the virtfs stuff is in place, busybox builds and
we use our installed busybox to implement stuff (some
automake builds use mdate-sh script which parses the output
of ls for the name of the month in order to construct
a release date, yeah...)
o Overhaul of artifact creation and staging using a virtfs
conversion layer, most recent version is viewable here:
https://github.com/gtristan/ybd/blob/aboriginal/ybd/virtfs.py
A short explanation of what this is for:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When using the security model "mapped" or "mapped-file" as we
now use, this allows the qemu guest to see the virtfs mounted
system in a way that it believes it can create anything as root.
The aboriginal guest system thus can create device files (mknod)
and give root ownership to anything it wants, BUT, on the host
side, QEMU creates additional metadata, and creates all files
under the credentials of the user running QEMU.
Our virtfs conversion layer performs the following functions:
o Collects the build results of an aboriginal build and
creates the deterministic tarball (artifact)
This is done by interpreting the QEMU virtfs metadata
and using that to set attributes on the archived files.
Interestingly, some artifacts just cannot be untarred
by a regular user (like the fhs-dirs artifact which
contains device files) However, these artifacts should
be the "real thing", if you log into a running system
that YBD/Aboriginal produced and untar as root, it will
just do the right thing.
o Stages an artifact to a directory in such a way that
the QEMU guest will understand it.
Of course, the reverse of the above needs to be done
when staging a real tarball on a shared FS as a regular
user.
This is done by reading a 'real tarball' and translating
the content while extracting, creating the virtfs metadata
files as necessary
o Stages files which need to be read by the aboriginal build
When we stage a git repository to be built, we need to also
provide the metadata for qemu, this is especially true for
symbolic links which the emulator wont understand unless
they are properly translated.
So the result of this overhaul is that we now build as root in
the emulator, and we install everything as root, we end up with
tarballs containing files which clearly belong to root, and we
do this all without being root.
o Improving the IPC
Some progress was made here but I did not get as far as pushing
the results, with Rob Taylor's help I was able to figure out a
better way to spawn the emulator and expose the serial ports so
that on the host they are viewed as named pipes (fifos).
This seems to have buffering and work much better with bash
builtin read / echo than the current solution which uses a socket
on the host and tries to communicate with calls to 'socat'
The reason this has to be improved, is that sometimes the IPC
breaks, so a build breaks because of a messup in trying to tell
the slaved guest to build.
o YBD integration in my aboriginal branch is getting better in
general.
As per the first point, it would seem that we have built enough to
build a minimal system, so it's time to look into deploying a system.
However, to deploy a system, we of course have to build more stuff :)
So the next steps involve:
o Building a high level enough stratum which contains all the tooling
required to run the extensions involved in performing a deployment
Right now I have a blocker which is the coreutils I'm building,
which replaces most of busybox, does not work correctly, it causes
the builds of diff, tar and sed to break: To be investigated.
o Once we have built the tooling required to perform a deployment,
we'll have to teach YBD how to stage the extension scripts along
with the "deployment-essential" strata, along with the actual
data to deploy, and have the emulator run the deployment scripts.
I say "deployment-essential" to convey the idea, I dont think
a new stratum needs to be created, deployment-essential is
most probably already the "foundation" stratum.
Summary of current state
~~~~~~~~~~~~~~~~~~~~~~~~
If you follow the steps to build which I provided in my last status
report, you will be able to completely build core.morph for armv5l,
as a regular user, and you will have those artifacts which contain
files that are root uid/gid.
Cheers,
-Tristan
6 years, 3 months
Proposal: a new repo to hold the Baserock definitions format specification
by Sam Thursfield
Hi
I'd like to propose a new repo that would be the canonical location of
the Baserock definitions format specification.
Here is what I think it should contain: https://www.github.com/ssssam/spec
And it would live at git://git.baserock.org/baserock/baserock/spec
The repo contains a Makefile that converts the text part to HTML. I
don't have time to immediately do this last part, but it should be
less than a day's faff to set up an hourly cron job on the webserver
that hosts http://download.baserock.org to build 'master' of the spec
and serve the results at http://docs.baserock.org.
I want to do this mainly because I think it makes it more obvious how
to contribute improvements to the format. Everything that needs to be
updated for a new version of the format is in this repo, and so each
proposed change is basically a patch to this repo and can go through
our usual patch review process (possibly on the mailing list instead
of Gerrit -- I think changes to the format need to have as much
visibility as possible).
The reference system definitions in definitions.git would no longer
contain anything to do with the definitions format. Thus it's clear
that the two things are separate, and we can introduce a new version
without necessarily migrating everything immediately.
We've discussed this in the past, and already agreed that the
migrations need to move out of the definitions.git repo (if you're
updating a fork of our definitions, you need to run the migrations
*before* merging 'master' of definitions.git, so keeping them *in*
'master' of definitions.git is really
stupid). Hopefully keeping everything in one repo makes sense to
others. Let me know what you think.
Oh and if anyone can think of a better name than "the Baserock
definitions format specification" that might be handy. (And not BDFS
:-).
Currently we have no overall license info for definitions.git, I
suggest using the same license as the wiki (Creative Commons) for
spec.git. The migrations all contain license info already, so this
just covers the text and the JSON-Schema schemas.
Sam
6 years, 3 months
YBD 16.10 is released
by Paul Sherwood
No much is new in this release - mainly contributions from @rdale and
@gtristan, and other fixes. Strata artifacts are now uploaded to kbas as
well as chunks.
3678239 Strata artifacts really should be bit-for-bit
e6af454 Many chunks have no 'kind'
493a3b1 Better fix for mis-typed arch
ad8d245 Revert "Log when skipping definition because of arch"
be5c98a Upload strata too
4ed45c9 Separate out hash_factors function
8427490 Another attempt at claim, can ctrl-c
57bb486 Merge pull request #190 from gtristan/defensive-claim-exception
1557381 Merge pull request #187 from gtristan/yaml-parse-errors
b88e6e6 assembly.py: Handle the flock() exception more aggressively
aa41ac1 definitions.py: Log YAML parse errors while loading definitions
c11cb2d Log when skipping definition because of arch
9831880 Merge pull request #186 from
rdale/rdale/fix-artifact-splitting-regressions
c5fe9cd Fix artifact splitting regressions
c3fa823 Merge pull request #181 from gtristan/change-renames-moves
2d943c8 kbas/main.py: Use shutil.move() instead of os.rename()
11b0f7d cache.py: Use shutil.move() instead of os.rename()
35d6e4d repos.py: Use shutil.move() instead of os.rename()
6 years, 4 months