Distributed CI and threat modelling
by Lars Wirzenius
Also avilable at:
https://ick.liw.fi/blog/2019/08/11/distributed_ci_and_threat_modelling/
---
Some thoughts about making a distributed CI system secure.
* outline of system
* version control system hold source code
* IDP authenticates and authorizes users, system components
* controller co-ordinates builds, collects build logs
* artifact store holds build artifacts
* workers (many) do the actual building, are told by controller
what to do, fetch source from version control system, upload
artifacts to artifact store
* entitites in the system that need to be protected:
* the person using CI
* the person running the IDP, controller, and artifact store (for
simplicity, assume they're all run by the same person, although
they could each be run by separate people)
* the people running runners
* threats to person using CI
* malicious workers, which embeds unwanted code in build artifact
* mitigation: use reproducible builds and build on at least
two workers to detect unwanted changes in artifacts; this
would work OK, if there are relatively few malicious workers
* many malicious workers, or workers that become malicious after a
long period of working fine
* mitigation: have at least one trusted worker, which might be
slow, but whose output is required for a build to be trusted
* artifacts from maybe-trusted workers can't be used for
deployment, but could be used with sufficient isolation
to speed things up, e.g., to do heavy testing: if the
trusted worker later confirms the binaries are
trustworthy (bitwise identical), then the test results
can be trusted, too
* variant of mitigation: require at least N maybe-trusted
workers to produce bitwise identical build artifacts, where
N is set by the person running the CI or whose project is
being built
* rejected: a karma or reputation system based on past
behaviour: this makes long-lived workers valuable targets,
and years of good behaviour won't protect if the worker gets
hijacked
* threats to person running IDP, controller, artifact store
* there are no new threats to these that come due to the
distributed nature of CI
* all the usual threats apply, of course
* threats to those running workers
* build uses too much CPU or RAM
* mitigation: enable person running worker to set limits and
priorities so that the build doesn't use resources needed by
other things
* build attacks remote hosts (e.g., DDoS)
* mitigation: prevent build from accessing any network hosts,
except version control server, controller, artifact store
* build attacks host where worker runs
* mitigation: run build in a VM, using the best avilable
isolation techniques, such as carefully configured qemu/KVM
to implement the VM, and keeping all related software up to
date
--
I want to build worthwhile things that might last. --joeyh
3 years, 7 months