[RFC] Software Updates

Richard Maw richard.maw at gmail.com
Wed Feb 11 16:06:54 GMT 2015

In Baserock we do atomic updates by copying the new version onto the target
with rsync over ssh. We rsync on top of a snapshot of the current system so
unchanged files don't need to be rewritten. After the update is wrtitten, we
instruct the bootloader to use the new version in future and tell it to reboot.

This is not satisfactory for everything we want to do with Baserock, so at the
Baserock meetup last week we collected some requirements of what Baserock needs
to do that it currently doesn't.

1.  Be able to perform kernel updates without a full POST cycle.

    This isn't the same as needing Live kernel updates. There's
    already existing projects attempting to be able to provide
    that. kgraft/ksplice are attempting to life-patch the kernel as
    it's running. CRIU intends to reach this goal by dumping the state
    of all userland processes and kexecing.

    Live updates aren't such a hard requirement, as load balancers allow
    updates to machines across a large cluster to be handled without
    an interruption in service, but they still need to be reasonably
    fast so a staged update of a cluster doesn't leave un-updated nodes
    operating with old versions for too long.

    To this end, a plain kexec would be suitable, but this needs testing
    on sufficiently enterprise hardware, as kexec is not always well
    supported, either by the hardware not being restartable without a
    power cycle, or the drivers making too many assumptions about the
    hardware's initial state.

2.  When updates are made to security critical services, they are restarted.

    Changes to libraries that services depend on also imply the service
    needs to be restarted.

    In a package-based update mechanism it would be sufficient to ask
    the service to restart, and some services are able to be asked to
    gracefully restart without service interruptions.

    Our current Atomic update mechanism doesn't allow for Live updates
    of services, as we need to restart.

3.  Minimise downtime

    The ultimate goal of this is to be able to apply an update which
    includes a kernel upgrade without any down-time.

    This would require a CRIU+kexec or kgraft/ksplice approach, but
    for non-kernel updates we also need to be able to handle the update
    without downtime.

    The CRIU+kexec approach is not yet possible, as CRIU is not yet able to
    dump and restore the state of every process that is required to run.

4.  Prompt, rolling or continuous updates

    We should be able to keep a large fleet of Baserock based systems
    up to date.

    To this end we need both infrastructure for making these updates
    available, and a notification mechanism for devices to discover when
    they should update.

    CoreOS has a tool called Omaha, which we believe to be an OMADM
    implementation. There is concern that it is too complicated, as it
    needs to be a rootly service, and requires an XML parser. We need to
    confirm this to be able to rule it out, as this may be one of the few
    cases where rolling our own is more appropriate than reusing someone
    else's work, since it needs to be trivially simple to be secure.

    It was jokingly suggested that NNTP may be an appropriate base for
    this work if we need to implement our own mechanism.

    The updates should be as compact as possible. Btrfs/ZFS send/receive
    are able to use filesystem-level knowledge to generate efficient
    packs of changes, but we can also do some work to minimise the
    differences between binaries by the reproducible build work done by
    the TOR project, and ChromeOS' Courgette tool.

5.  Non-snapshot-based updates for tiny systems.

    There was doubt that our btrfs-based approach scales down as far as
    we would like it to.

    It was proposed that the traditional A/B partitioning scheme, or
    even something OSTree based would be appropriate.


1.  Try kexec on enterprise hardware to see how much of a difference it
    makes on update cycle times, and whether it is reliable.
2.  Investigate tooling for detecting when services need to be restarted,
    there already exists tooling for looking at /proc/$pid/maps for
    deleted binaries that are still being used.
3.  Take the proof-of-concept for Live Atomic Updates to completion.
4.  Investigate the state of kgraft/ksplice, and whether we can use them.
5.  Investigate a mechanism that allows both pull and push of
    notifications for software updates.
    Zookeeper may also be appropriate in some circumstances.
6.  Investigate using Btrfs/ZFS packs for the transmission of updates,
    and whether Courgette can reduce our deltas.
7.  Investigate the block churn characteristics when using OSTree on UBIFS.

More information about the baserock-dev mailing list