Hi,
When I integrated BuildStream into the Baserock CI, I made it so that
the results of builds would be pushed to the cache at
https://ostree.baserock.org/cache but only from branches that were
marked as "protected".
There are a couple of downsides to this approach:
* CI on non-protected branches can be appallingly slow. If I update
GCC on a non-protected branch, every single system will rebuild from
GCC upwards.
* Protected branches can't be deleted or force-pushed, which is
annoying to work with
I've now changed the configuration so that builds running on any branch
can push artifacts to the remote cache.
Read on if you are interested in the security concerns around this
decision!
The artifact caching mechanism used by BuildStream, Morph, and YBD is
based on the idea that by hashing all the inputs to a build, we can
download a prebuilt artifact that was built from the same inputs in
place of an artifact that we built ourselves.
There's an attack vector here -- the remote cache could serve an
artifact with some malicious behaviour instead of the real artifact.
There's no evidence that anyone has tried to do this so far, but it's
possible that a rogue cache could serve, for example, a compromised
version of `opensshd` that causes every VM you build to join a botnet.
To protect against this we gate pushes to
ostree.baserock.org behind
SSH. In order to push you must have the private part of one these keys:
https://gitlab.com/baserock/infrastructure/tree/master/baserock_ostree/keys
So we assume we can trust the contents of that cache. We then serve
artifacts over HTTPS so that you can be sure you are receiving exactly
the artifact that the cache sent.
In order to push to the cache the GitLab CI runners need to have the
private part of the baserock-gitlab-ci SSH key. We make that available
as a "secret variable"[1]. Anyone with "developer" access can see
these
variables, all they need to do is push a branch where the .gitlab-ci.yml
file contains a line like `echo $private_key`. So effectively, everyone
with developer access to the 'definitions' project can now write stuff
to the cache.
There's currently 15 people with "developer" access to the definitions
repo so this isn't a huge risk in practice. But be careful when adding
more people that they are not spammers or whatever ;-)
(Previously the situation was that only people with "managers" or
"owner" access would be able to see the private key, because you need
that level of access in order to work with protected branches).
In future BuildStream is going to grow support for signing artifacts
with a GPG key. That provides a different chain of trust to the SSH +
HTTPS approach described above; but the GitLab runners still need to
have a private key in order to sign artifacts so the security concerns
are the same.
Let me know if you have any questions about this!
Sam
PS. it might seem that i'm overthinking this issue, but bear in mind
it affects any project that wants to run BuildStream builds as part
of a public CI system. Baserock is functioning as a 'guinea pig' here
so it's important to think through the security concerns in detail.
[1].
https://gitlab.com/help/ci/variables/README#secret-variables
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575