[RFC] Software Updates
richard.maw at gmail.com
Wed Feb 11 16:06:54 GMT 2015
In Baserock we do atomic updates by copying the new version onto the target
with rsync over ssh. We rsync on top of a snapshot of the current system so
unchanged files don't need to be rewritten. After the update is wrtitten, we
instruct the bootloader to use the new version in future and tell it to reboot.
This is not satisfactory for everything we want to do with Baserock, so at the
Baserock meetup last week we collected some requirements of what Baserock needs
to do that it currently doesn't.
1. Be able to perform kernel updates without a full POST cycle.
This isn't the same as needing Live kernel updates. There's
already existing projects attempting to be able to provide
that. kgraft/ksplice are attempting to life-patch the kernel as
it's running. CRIU intends to reach this goal by dumping the state
of all userland processes and kexecing.
Live updates aren't such a hard requirement, as load balancers allow
updates to machines across a large cluster to be handled without
an interruption in service, but they still need to be reasonably
fast so a staged update of a cluster doesn't leave un-updated nodes
operating with old versions for too long.
To this end, a plain kexec would be suitable, but this needs testing
on sufficiently enterprise hardware, as kexec is not always well
supported, either by the hardware not being restartable without a
power cycle, or the drivers making too many assumptions about the
hardware's initial state.
2. When updates are made to security critical services, they are restarted.
Changes to libraries that services depend on also imply the service
needs to be restarted.
In a package-based update mechanism it would be sufficient to ask
the service to restart, and some services are able to be asked to
gracefully restart without service interruptions.
Our current Atomic update mechanism doesn't allow for Live updates
of services, as we need to restart.
3. Minimise downtime
The ultimate goal of this is to be able to apply an update which
includes a kernel upgrade without any down-time.
This would require a CRIU+kexec or kgraft/ksplice approach, but
for non-kernel updates we also need to be able to handle the update
The CRIU+kexec approach is not yet possible, as CRIU is not yet able to
dump and restore the state of every process that is required to run.
4. Prompt, rolling or continuous updates
We should be able to keep a large fleet of Baserock based systems
up to date.
To this end we need both infrastructure for making these updates
available, and a notification mechanism for devices to discover when
they should update.
CoreOS has a tool called Omaha, which we believe to be an OMADM
implementation. There is concern that it is too complicated, as it
needs to be a rootly service, and requires an XML parser. We need to
confirm this to be able to rule it out, as this may be one of the few
cases where rolling our own is more appropriate than reusing someone
else's work, since it needs to be trivially simple to be secure.
It was jokingly suggested that NNTP may be an appropriate base for
this work if we need to implement our own mechanism.
The updates should be as compact as possible. Btrfs/ZFS send/receive
are able to use filesystem-level knowledge to generate efficient
packs of changes, but we can also do some work to minimise the
differences between binaries by the reproducible build work done by
the TOR project, and ChromeOS' Courgette tool.
5. Non-snapshot-based updates for tiny systems.
There was doubt that our btrfs-based approach scales down as far as
we would like it to.
It was proposed that the traditional A/B partitioning scheme, or
even something OSTree based would be appropriate.
1. Try kexec on enterprise hardware to see how much of a difference it
makes on update cycle times, and whether it is reliable.
2. Investigate tooling for detecting when services need to be restarted,
there already exists tooling for looking at /proc/$pid/maps for
deleted binaries that are still being used.
3. Take the proof-of-concept for Live Atomic Updates to completion.
4. Investigate the state of kgraft/ksplice, and whether we can use them.
5. Investigate a mechanism that allows both pull and push of
notifications for software updates.
Zookeeper may also be appropriate in some circumstances.
6. Investigate using Btrfs/ZFS packs for the transmission of updates,
and whether Courgette can reduce our deltas.
7. Investigate the block churn characteristics when using OSTree on UBIFS.
More information about the baserock-dev