Hosting binary sysroots for BuildStream builds
by Sam Thursfield
Hello,
BuildStream doesn't support the "bootstrap mode" that Morph and YBD use.
This is for the best as building stuff outside a sandbox can lead to
madness[1], but it means that Baserock needs to build and provide
sysroot binaries that we can then use to "seed" our reference system builds.
Currently we use the freedesktop SDK binaries, but this sucks for a few
reasons including that they are huge (1GB+ when we really only need
<300MB of stuff), and they aren't available for many platforms (x86_32,
x86_64 and armv8l64 only I think).
Producing our own binaries is easy enough to do; our
build-essential.morph / gnu-toolchain.bst stack already does everything
we need.
What I'm wondering is how best to host them for posterity. For now I've
been pulling the artifacts from https://ostree.baserock.org/cache/
directly, but this isn't good because things in a cache can be deleted,
and if our reference system builds depend on the existence of these
binaries then they really really shouldn't be deleted ever.
Here's a proposal, which I welcome comments on and will put into
practice next week if nobody disagrees with it.
* We add a bootstrap/stage3-sysroot.bst element, which is just
gnu-toolchain.bst with some splitting rules to remove debug symbols,
docs and locale.
* We set up a new ostree repo at http://ostree.baserock.org/releases
* We push builds of stage3-sysroot.bst for each platform, using the
following naming convention for the refs:
stage3-sysroot/x86_32
stage3-sysroot/x86_64
stage3-sysroot/armv8l64
* In base.bst we put the ref *and* the sha256 hash, so a given commit
of definitions will always pull the same binaries forever, even if a new
version is pushed to the ref `stage3-sysroot/x86_32`
And that's it. We could perhaps push new releases there every time
something changes in gnu-toolchain.bst; it doesn't actually matter
though because we do a whole 3-stage bootstrap from these binaries
anyway so they will very rarely need to change (I'd guess yearly updates
at most?).
I'm not actually sure how to do the push, the `bst push` command will
use the buildstream cache ID as the ref which isn't helpful here, so
that might need some modification to allow pushing to a different ref name.
And that's it. For those who haven't spent the past couple of months
thinking about baserock and buildstream this email might not fully make
sense, feel free to ask questions (although i might not get to them
until next week).
Thanks
Sam
1. for example, I once lost a week debugging random crashes that turned
out to be caused by an ABI break in GLIBC on armv7 -- the stage1-gcc
binary would work on systems with the newer GLIBC but fail on systems
with older GLIBC, but Baserock's cache key didn't include anything about
the host so there was no way to avoid the breakage...
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
6 years, 1 month
- 1
- 2
Baserock infrastructure news
by Sam Thursfield
Hi all,
Firstly, we just found out that the cloud provider we use will no longer
be operating from Friday 6th October 2017. Gary is looking into
alternative OpenStack hosts and will migrate our systems to the new one,
but there will be some amount of downtime over the next couple of weeks.
This is a good opportunity to get rid of services we no longer use.
storyboard.baserock.org
-----------------------
* Issues should be reported at
https://gitlab.com/groups/baserock/issues
* Existing content is available in mysqldump format at:
https://download.baserock.org/baserock/infrastructure/storyboard.20170926...
gerrit.baserock.org
-------------------
* Patches should be submitted through Gitlab
* Existing content is available in mysqldump format at:
https://download.baserock.org/baserock/infrastructure/gerrit.20170926.mys...
openid.baserock.org
-------------------
* This was set up for use with Gerrit and Storyboard; we don't need
to provide our own any more
* If enough people care about it we could keep it running, but...
- It only supports OpenID 2.0, which is kind of superceded by
OpenID Connect nowadays.
- If you need an OpenID provider, try
https://admin.fedoraproject.org/accounts/
- If you need to set up your own OpenID provider, look at
Ipsilon <https://pagure.io/ipsilon/>
Removing these also means we no longer need to host our own MySQL
database, or an Exim4 mail relay.
I have left these services running for now but they will probably not be
migrated to the new hosting provider, so they will disappear at some
point next week.
The remaining Baserock infrastructure is going to remain active and with
ongoing maintainance support from Codethink:
artifacts1.baserock.org
-----------------------
* We will continue to provide a KBAS cache server for use with YBD
download.baserock.org
---------------------
* We will continue to host all this content
docs.baserock.org
-----------------
* This is duplicated at https://baserock.gitlab.io/spec/ already,
it may just redirect there in future
git.baserock.org
-----------------
* This will continue to provide a Git mirror of loads of cool
projects
* It's currently based off a Baserock reference system; we will
move it to Debian so that it's easier to keep up to date. This
should make it easier to deploy and maintain your own Trove too.
irclogs.baserock.org
--------------------
* This provides services for various projects and will continue to
be managed.
* It's currently based off a Baserock reference system; we will
move it to a mainstream distro so that it's easier to keep up to
date
ostree.baserock.org
-------------------
* We will continue to provide an OSTree cache server for use with
BuildStream
paste.baserock.org
------------------
* This will continue to be hosted for the time being to ensure that
the existing pastes don't disappear off the face of the Web.
Let me know if you have any questions, comments or elegies for the
departed.
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
6 years, 2 months
- 1
- 0
- ← Newer
- 1
- Older →