[PATCH 0/5] Lorry Controller: get rid of "ghost jobs"
by Lars Wirzenius
repo: git://git.baserock.org/baserock/baserock/lorry-controller
branch: baserock/liw/de-ghost
commit: b946a00ae92f1a573d51714b0a6b8b075fb5e792
land: master
card: 11305
This set of patches adds functionality to Lorry Controller to notice
jobs whose MINION (worker process running the job) has disappeared;
the patches introduce the term "ghost job" for this. This allows such
jobs to be cleaned up automatically, and stops them from preventing
new jobs from running once the table of jobs is filled with ghosts.
Lars Wirzenius (5):
Update ARCH about new API call (remove-ghost-jobs)
Add yarn tests for removing ghost jobs
Add --ghost-timeout option to WEBAPP
Implement /1.0/remove-ghost-jobs
Add systemd units to remove ghost jobs automatically
ARCH | 5 ++
lorry-controller-webapp | 10 +++
lorrycontroller/__init__.py | 1 +
lorrycontroller/removeghostjobs.py | 65 +++++++++++++++++++
units/lorry-controller-remove-ghost-jobs.service | 9 +++
units/lorry-controller-remove-ghost-jobs.timer | 6 ++
yarns.webapp/040-running-jobs.yarn | 79 ++++++++++++++++++++++++
7 files changed, 175 insertions(+)
create mode 100644 lorrycontroller/removeghostjobs.py
create mode 100644 units/lorry-controller-remove-ghost-jobs.service
create mode 100644 units/lorry-controller-remove-ghost-jobs.timer
--
1.9.1
8 years, 9 months
[PATCH] Stop doing a ls-remote before we build/deploy
by Richard Maw
This replaces the `git ls-remote` with inspecting git's remote refs.
This cache gets updated when you run appropriate git commands, so I
think it's a suitable replacement.
It's just a shame that the clone in your workspace will not get updated
when morph has to update the version in its repository cache to do builds.
It is possible with some magic to have the clones in your
workspace use the same repository as your repo cache (see
http://git.baserock.org/cgi-bin/cgit.cgi/delta/git.git/tree/contrib/workd...),
but it's potentially dangerous, as a user could mess up the branches
pretty badly.
In another tanentially related musing, it may make more sense to have the
check for whether a branch has been pushed check whether our local version
is not ahead of the remote version, rather than being the same version.
Richard Maw (1):
Use git's remotes to determine unpushed branches
morphlib/buildbranch.py | 17 +++++++----------
morphlib/gitdir.py | 16 ++++++++++++++++
2 files changed, 23 insertions(+), 10 deletions(-)
--
1.9.1
8 years, 9 months
[PATCH] morph: Remove some cache key related log messages
by Sam Thursfield
Repository: git://git.baserock.org/baserock/baserock/morph
Ref: sam/less-cache-key-logging
Sha1: cf24999b40b2877dc35f4a77ebc43f29b1dbc960
I ran `morph build systems/devel-system-x86_64-generic.morph` for a
version of that system that I held a cached build of. Without this
patch Morph produced a 3.2MB log file. With this patch, it produced a
572KB log file.
The new messages look like this:
2014-08-29 08:59:48 DEBUG computed cache key 361c850612961f9fd269d29cf0b44925ab9ad0bc7d4660175800cb5d44b8cb60 for artifact stage1-binutils-bins from source upstream:binutils-redhat|987eb543a034cddc4eeb103f08f566f24a5f964a|strata/build-essential/stage1-binutils.morph
Sam Thursfield (1):
Remove some cache key related log messages
morphlib/cachekeycomputer.py | 20 +++++---------------
1 file changed, 5 insertions(+), 15 deletions(-)
--
1.9.3
8 years, 9 months
[PATCH 0/4] `morph deploy` improvements
by Sam Thursfield
Repository: git://git.baserock.org/baserock/baserock/morph
Ref: sam/deploy-improvements
Sha1: 9846f276e00e5f8f3e520e940ea8b78a44fec150
The first two patches can be considered separately from the last two.
Sam Thursfield (4):
Clarify that multiple images can be deployed at once by `morph deploy`
Add `morph upgrade` command, deprecate `morph deploy --upgrade`
deploy: Allow extensions to write to Morph log file
deploy: Make Python extensions log debug messages to MORPH_LOG_FD by
default
morphlib/exts/rawdisk.check | 4 +-
morphlib/plugins/deploy_plugin.py | 155 +++++++++++++++++++++++++++++++-------
morphlib/writeexts.py | 25 +++++-
yarns/deployment.yarn | 2 +-
yarns/implementations.yarn | 2 +-
5 files changed, 157 insertions(+), 31 deletions(-)
--
1.9.3
8 years, 9 months
[PATCH] cliapp: Don't write passwords to log files
by Sam Thursfield
Repository: git://git.baserock.org/delta/cliapp
Ref: baserock/sam/avoid-logging-passwords-from-env
Sha1: 19f3885a8b36249231dbcfdcf065f3512546106c
Through its use of cliapp, Morph logs the whole environment as debug
messages on startup.
If this patch is accepted into the Baserock fork of cliapp I shall
forward it to Lars to see if it is wanted upstream. too. However, we'll
still have other unmerged patches compared to the upstream version such
as our changes to how the help works.
Sam Thursfield (1):
Don't log environment variables with 'PASSWORD' in their name.
cliapp/app.py | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
--
1.9.3
8 years, 9 months
[PATCH] morph: Don't write passwords to log files
by Sam Thursfield
Repository: git://git.baserock.org/baserock/baserock/morph
Ref: sam/avoid-logging-passwords-from-env
Sha1: 3569e362dd882cce3de7383276ac1bfe3830b207
Sam Thursfield (1):
Don't log environment variables with 'PASSWORD' in their name.
morphlib/app.py | 2 +-
morphlib/plugins/deploy_plugin.py | 3 +--
morphlib/util.py | 39 +++++++++++++++++++++------------------
3 files changed, 23 insertions(+), 21 deletions(-)
--
1.9.3
8 years, 9 months
morph deploy seems to visit g.b.o
by Paul Sherwood
Hi
as discussed on irc with Richard, it seems `morph deploy` checks things
on g.b.o.
http://fpaste.org/129700/93473451/
I think there are two issues here
1) g.b.o is currently rejecting ls-remote requests, so any user's
deployments fail
2) deploy shouldn't need to do this at all, afaict
br
Paul
8 years, 9 months
[PATCH] Set cpython ref to a sha1 with no morph file
by Francisco Redondo Marchena
cpython.morph was modified in definitions but this changes
were not updated to its repository.
Changing the reference to a one which does not contain the
morph file would be enough to remove the duplicity on
cpython.morph definition.
REPO: ssh://git@git.baserock.org/baserock/baserock/definitions
BRANCH: baserock/franred/cpython-points-to-no-morph-ref
COMMIT: 6cad1f3c97db3966266b9b2f191d856bdc6db9e8
Francisco Redondo Marchena (1):
core.morph: Change cpython ref to point to a non morph ref
strata/core.morph | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
1.8.4
8 years, 9 months
[PATCH 0/2] Use a better fake git server in yarns
by Richard Maw
The current approach of using file:// URIs means taht our test suite is
lacking coverage in repository caching, since morph has special handling
of file URIs.
So this patch adds a helper script, that allows a git daemon to be
launched on a random port, and changes the test repo alias to use a git://
URI on localhost with the port of the just spawned git daemon.
Richard Maw (2):
Add script for launching git daemon on random port
yarns: Use a more realistic simulation of a git server
scripts/git-daemon-wrap | 46 ++++++++++++++++++++++++++++++++++++++++++
yarns/architecture.yarn | 2 ++
yarns/branches-workspaces.yarn | 13 ++++++++++++
yarns/building.yarn | 1 +
yarns/deployment.yarn | 7 +++++++
yarns/implementations.yarn | 25 ++++++++++++++++++++++-
yarns/regression.yarn | 4 ++++
yarns/splitting.yarn | 2 ++
8 files changed, 99 insertions(+), 1 deletion(-)
create mode 100755 scripts/git-daemon-wrap
--
1.9.1
8 years, 9 months
