Issues importing GCC 7.2.0 tarball
by Sam Thursfield
Hi all
I got stuck in a timesink today trying to update our gcc-tarball repo.
I don't have time to fix these issues but I will write them up here
for posterity.
Firstly, the GCC 7.2 tarball from https://ftp.gnu.org/gnu/gcc/gcc-7.2.0/
doesn't import correctly with Lorry's `lorry.tar-importer` script, nor
with the `contrib/fast-import/import-tars.perl` script from Git that it
is based on.
The script fails with an error like this:
fatal: Unsupported command: // any later version.
fast-import: dumping crash report to fast_import_crash_10201
The exact message varies, but the issue seems to be that it's sending
source code as `git fast-import` commands, causing `git fast-import` to
raise an error.
GCC publish checksums for their release tarballs and I have verified
that the file didn't get corrupted during transmission. There `tar -t`
command doesn't report any issue with the tarball and `tar -x` can
successfully extract everything.
To add to the fun, Lorry fails to recognise that the import failed. I
have narrowed this down to what looks like a Python 2 bug. If I run the
tarball import command in my shell or using subprocess.Popen from Python
3, I get a non-zero exit code. However if I run it using
subprocess.Popen from Python 2, I get a returncode of 0.
I don't have time to dig further into either of these problems, so
instead I've manually pushed a gcc-7.2.0 tag to our delta/gcc-tarball
repo.
Sam
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
5 years, 6 months
BuildStream builds pushing to the cache
by Sam Thursfield
Hi,
When I integrated BuildStream into the Baserock CI, I made it so that
the results of builds would be pushed to the cache at
https://ostree.baserock.org/cache but only from branches that were
marked as "protected".
There are a couple of downsides to this approach:
* CI on non-protected branches can be appallingly slow. If I update
GCC on a non-protected branch, every single system will rebuild from
GCC upwards.
* Protected branches can't be deleted or force-pushed, which is
annoying to work with
I've now changed the configuration so that builds running on any branch
can push artifacts to the remote cache.
Read on if you are interested in the security concerns around this
decision!
The artifact caching mechanism used by BuildStream, Morph, and YBD is
based on the idea that by hashing all the inputs to a build, we can
download a prebuilt artifact that was built from the same inputs in
place of an artifact that we built ourselves.
There's an attack vector here -- the remote cache could serve an
artifact with some malicious behaviour instead of the real artifact.
There's no evidence that anyone has tried to do this so far, but it's
possible that a rogue cache could serve, for example, a compromised
version of `opensshd` that causes every VM you build to join a botnet.
To protect against this we gate pushes to ostree.baserock.org behind
SSH. In order to push you must have the private part of one these keys:
https://gitlab.com/baserock/infrastructure/tree/master/baserock_ostree/keys
So we assume we can trust the contents of that cache. We then serve
artifacts over HTTPS so that you can be sure you are receiving exactly
the artifact that the cache sent.
In order to push to the cache the GitLab CI runners need to have the
private part of the baserock-gitlab-ci SSH key. We make that available
as a "secret variable"[1]. Anyone with "developer" access can see these
variables, all they need to do is push a branch where the .gitlab-ci.yml
file contains a line like `echo $private_key`. So effectively, everyone
with developer access to the 'definitions' project can now write stuff
to the cache.
There's currently 15 people with "developer" access to the definitions
repo so this isn't a huge risk in practice. But be careful when adding
more people that they are not spammers or whatever ;-)
(Previously the situation was that only people with "managers" or
"owner" access would be able to see the private key, because you need
that level of access in order to work with protected branches).
In future BuildStream is going to grow support for signing artifacts
with a GPG key. That provides a different chain of trust to the SSH +
HTTPS approach described above; but the GitLab runners still need to
have a private key in order to sign artifacts so the security concerns
are the same.
Let me know if you have any questions about this!
Sam
PS. it might seem that i'm overthinking this issue, but bear in mind
it affects any project that wants to run BuildStream builds as part
of a public CI system. Baserock is functioning as a 'guinea pig' here
so it's important to think through the security concerns in detail.
[1]. https://gitlab.com/help/ci/variables/README#secret-variables
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
6 years
Hosting binary sysroots for BuildStream builds
by Sam Thursfield
Hello,
BuildStream doesn't support the "bootstrap mode" that Morph and YBD use.
This is for the best as building stuff outside a sandbox can lead to
madness[1], but it means that Baserock needs to build and provide
sysroot binaries that we can then use to "seed" our reference system builds.
Currently we use the freedesktop SDK binaries, but this sucks for a few
reasons including that they are huge (1GB+ when we really only need
<300MB of stuff), and they aren't available for many platforms (x86_32,
x86_64 and armv8l64 only I think).
Producing our own binaries is easy enough to do; our
build-essential.morph / gnu-toolchain.bst stack already does everything
we need.
What I'm wondering is how best to host them for posterity. For now I've
been pulling the artifacts from https://ostree.baserock.org/cache/
directly, but this isn't good because things in a cache can be deleted,
and if our reference system builds depend on the existence of these
binaries then they really really shouldn't be deleted ever.
Here's a proposal, which I welcome comments on and will put into
practice next week if nobody disagrees with it.
* We add a bootstrap/stage3-sysroot.bst element, which is just
gnu-toolchain.bst with some splitting rules to remove debug symbols,
docs and locale.
* We set up a new ostree repo at http://ostree.baserock.org/releases
* We push builds of stage3-sysroot.bst for each platform, using the
following naming convention for the refs:
stage3-sysroot/x86_32
stage3-sysroot/x86_64
stage3-sysroot/armv8l64
* In base.bst we put the ref *and* the sha256 hash, so a given commit
of definitions will always pull the same binaries forever, even if a new
version is pushed to the ref `stage3-sysroot/x86_32`
And that's it. We could perhaps push new releases there every time
something changes in gnu-toolchain.bst; it doesn't actually matter
though because we do a whole 3-stage bootstrap from these binaries
anyway so they will very rarely need to change (I'd guess yearly updates
at most?).
I'm not actually sure how to do the push, the `bst push` command will
use the buildstream cache ID as the ref which isn't helpful here, so
that might need some modification to allow pushing to a different ref name.
And that's it. For those who haven't spent the past couple of months
thinking about baserock and buildstream this email might not fully make
sense, feel free to ask questions (although i might not get to them
until next week).
Thanks
Sam
1. for example, I once lost a week debugging random crashes that turned
out to be caused by an ABI break in GLIBC on armv7 -- the stage1-gcc
binary would work on systems with the newer GLIBC but fail on systems
with older GLIBC, but Baserock's cache key didn't include anything about
the host so there was no way to avoid the breakage...
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
6 years, 1 month