This set of patches adds functionality to Lorry Controller to notice
jobs whose MINION (worker process running the job) has disappeared;
the patches introduce the term "ghost job" for this. This allows such
jobs to be cleaned up automatically, and stops them from preventing
new jobs from running once the table of jobs is filled with ghosts.
Lars Wirzenius (5):
Update ARCH about new API call (remove-ghost-jobs)
Add yarn tests for removing ghost jobs
Add --ghost-timeout option to WEBAPP
Add systemd units to remove ghost jobs automatically
ARCH | 5 ++
lorry-controller-webapp | 10 +++
lorrycontroller/__init__.py | 1 +
lorrycontroller/removeghostjobs.py | 65 +++++++++++++++++++
units/lorry-controller-remove-ghost-jobs.service | 9 +++
units/lorry-controller-remove-ghost-jobs.timer | 6 ++
yarns.webapp/040-running-jobs.yarn | 79 ++++++++++++++++++++++++
7 files changed, 175 insertions(+)
create mode 100644 lorrycontroller/removeghostjobs.py
create mode 100644 units/lorry-controller-remove-ghost-jobs.service
create mode 100644 units/lorry-controller-remove-ghost-jobs.timer
This replaces the `git ls-remote` with inspecting git's remote refs.
This cache gets updated when you run appropriate git commands, so I
think it's a suitable replacement.
It's just a shame that the clone in your workspace will not get updated
when morph has to update the version in its repository cache to do builds.
It is possible with some magic to have the clones in your
workspace use the same repository as your repo cache (see
but it's potentially dangerous, as a user could mess up the branches
In another tanentially related musing, it may make more sense to have the
check for whether a branch has been pushed check whether our local version
is not ahead of the remote version, rather than being the same version.
Richard Maw (1):
Use git's remotes to determine unpushed branches
morphlib/buildbranch.py | 17 +++++++----------
morphlib/gitdir.py | 16 ++++++++++++++++
2 files changed, 23 insertions(+), 10 deletions(-)
I ran `morph build systems/devel-system-x86_64-generic.morph` for a
version of that system that I held a cached build of. Without this
patch Morph produced a 3.2MB log file. With this patch, it produced a
572KB log file.
The new messages look like this:
2014-08-29 08:59:48 DEBUG computed cache key 361c850612961f9fd269d29cf0b44925ab9ad0bc7d4660175800cb5d44b8cb60 for artifact stage1-binutils-bins from source upstream:binutils-redhat|987eb543a034cddc4eeb103f08f566f24a5f964a|strata/build-essential/stage1-binutils.morph
Sam Thursfield (1):
Remove some cache key related log messages
morphlib/cachekeycomputer.py | 20 +++++---------------
1 file changed, 5 insertions(+), 15 deletions(-)
The first two patches can be considered separately from the last two.
Sam Thursfield (4):
Clarify that multiple images can be deployed at once by `morph deploy`
Add `morph upgrade` command, deprecate `morph deploy --upgrade`
deploy: Allow extensions to write to Morph log file
deploy: Make Python extensions log debug messages to MORPH_LOG_FD by
morphlib/exts/rawdisk.check | 4 +-
morphlib/plugins/deploy_plugin.py | 155 +++++++++++++++++++++++++++++++-------
morphlib/writeexts.py | 25 +++++-
yarns/deployment.yarn | 2 +-
yarns/implementations.yarn | 2 +-
5 files changed, 157 insertions(+), 31 deletions(-)
Through its use of cliapp, Morph logs the whole environment as debug
messages on startup.
If this patch is accepted into the Baserock fork of cliapp I shall
forward it to Lars to see if it is wanted upstream. too. However, we'll
still have other unmerged patches compared to the upstream version such
as our changes to how the help works.
Sam Thursfield (1):
Don't log environment variables with 'PASSWORD' in their name.
cliapp/app.py | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
as discussed on irc with Richard, it seems `morph deploy` checks things
I think there are two issues here
1) g.b.o is currently rejecting ls-remote requests, so any user's
2) deploy shouldn't need to do this at all, afaict
cpython.morph was modified in definitions but this changes
were not updated to its repository.
Changing the reference to a one which does not contain the
morph file would be enough to remove the duplicity on
Francisco Redondo Marchena (1):
core.morph: Change cpython ref to point to a non morph ref
strata/core.morph | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
The current approach of using file:// URIs means taht our test suite is
lacking coverage in repository caching, since morph has special handling
of file URIs.
So this patch adds a helper script, that allows a git daemon to be
launched on a random port, and changes the test repo alias to use a git://
URI on localhost with the port of the just spawned git daemon.
Richard Maw (2):
Add script for launching git daemon on random port
yarns: Use a more realistic simulation of a git server
scripts/git-daemon-wrap | 46 ++++++++++++++++++++++++++++++++++++++++++
yarns/architecture.yarn | 2 ++
yarns/branches-workspaces.yarn | 13 ++++++++++++
yarns/building.yarn | 1 +
yarns/deployment.yarn | 7 +++++++
yarns/implementations.yarn | 25 ++++++++++++++++++++++-
yarns/regression.yarn | 4 ++++
yarns/splitting.yarn | 2 ++
8 files changed, 99 insertions(+), 1 deletion(-)
create mode 100755 scripts/git-daemon-wrap