1:23 p.m.
repo: git@git.baserock.org:baserock/baserock/definitions
ref: baserock/richardipsum/distbuild-with-setup2
head: 84cb06607e2530757e8ab545b10de5fae3b6862a
Hi all,
These patches include the definitions and deployment stuff for distbuild.
I have branched from baserock/sam/distbuild-with-setup and squashed the
series.
My only comment is that we're using ssh-keyscan to generate the known-hosts
file. This tool prints stuff to stderr for some reason, so you can see its
output during deployment, there doesn't seem to be a flag to disable it either.
Richard Ipsum (3):
Add distbuild stratum
Add armv7lhf distbuild system
Add distbuild config extension
Sam Thursfield (2):
Import distbuild configuration files into definitions.git
distbuild: Add example cluster for deployment
distbuild-system-armv7lhf-highbank.morph | 20 +++
distbuild.configure | 161 ++++++++++++++++++++
distbuild.morph | 20 +++
.../system-generators/ccache-nfs-mount-generator | 16 ++
distbuild/manifest | 19 +++
.../lib/systemd/system/morph-cache-server.service | 9 ++
.../systemd/system/morph-controller-helper.service | 10 ++
.../lib/systemd/system/morph-controller.service | 9 ++
.../lib/systemd/system/morph-worker-helper.service | 10 ++
.../usr/lib/systemd/system/morph-worker.service | 10 ++
.../morph-cache-server.service | 1 +
.../morph-controller-helper.service | 1 +
.../morph-controller.service | 1 +
.../morph-worker-helper.service | 1 +
.../multi-user.target.wants/morph-worker.service | 1 +
example-distbuild-cluster.morph | 37 +++++
16 files changed, 326 insertions(+)
create mode 100644 distbuild-system-armv7lhf-highbank.morph
create mode 100644 distbuild.configure
create mode 100644 distbuild.morph
create mode 100755 distbuild/lib/systemd/system-generators/ccache-nfs-mount-generator
create mode 100644 distbuild/manifest
create mode 100644 distbuild/usr/lib/systemd/system/morph-cache-server.service
create mode 100644 distbuild/usr/lib/systemd/system/morph-controller-helper.service
create mode 100644 distbuild/usr/lib/systemd/system/morph-controller.service
create mode 100644 distbuild/usr/lib/systemd/system/morph-worker-helper.service
create mode 100644 distbuild/usr/lib/systemd/system/morph-worker.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-cache-server.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller-helper.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker-helper.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker.service
create mode 100644 example-distbuild-cluster.morph
--
1.7.10.4
1:23 p.m.
New subject: [PATCH 1/5] Import distbuild configuration files into definitions.git
From: Sam Thursfield <sam.thursfield(a)codethink.co.uk>
These files should be installed into distbuild worker and controller
systems using the install-files configure extension, by adding the
following line in the cluster morphology.
INSTALL_FILES: distbuild/manifest
These files were previously in the internal Codethink chunk repo
'distbuild-setup', and were installed by a Makefile.
---
.../system-generators/ccache-nfs-mount-generator | 16 ++++++++++++++++
distbuild/manifest | 19 +++++++++++++++++++
.../lib/systemd/system/morph-cache-server.service | 9 +++++++++
.../systemd/system/morph-controller-helper.service | 10 ++++++++++
.../usr/lib/systemd/system/morph-controller.service | 9 +++++++++
.../lib/systemd/system/morph-worker-helper.service | 10 ++++++++++
.../usr/lib/systemd/system/morph-worker.service | 10 ++++++++++
.../morph-cache-server.service | 1 +
.../morph-controller-helper.service | 1 +
.../morph-controller.service | 1 +
.../morph-worker-helper.service | 1 +
.../multi-user.target.wants/morph-worker.service | 1 +
12 files changed, 88 insertions(+)
create mode 100755 distbuild/lib/systemd/system-generators/ccache-nfs-mount-generator
create mode 100644 distbuild/manifest
create mode 100644 distbuild/usr/lib/systemd/system/morph-cache-server.service
create mode 100644 distbuild/usr/lib/systemd/system/morph-controller-helper.service
create mode 100644 distbuild/usr/lib/systemd/system/morph-controller.service
create mode 100644 distbuild/usr/lib/systemd/system/morph-worker-helper.service
create mode 100644 distbuild/usr/lib/systemd/system/morph-worker.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-cache-server.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller-helper.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker-helper.service
create mode 120000
distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker.service
diff --git a/distbuild/lib/systemd/system-generators/ccache-nfs-mount-generator
b/distbuild/lib/systemd/system-generators/ccache-nfs-mount-generator
new file mode 100755
index 0000000..127bc84
--- /dev/null
+++ b/distbuild/lib/systemd/system-generators/ccache-nfs-mount-generator
@@ -0,0 +1,16 @@
+#!/bin/sh
+read trove_host </etc/trove-host
+while read line; do echo "$line"; done
>"$1/srv-distbuild-ccache.mount" <<EOF
+[Unit]
+Requires=rpcbind.service
+After=rpcbind.service
+SourcePath=/etc/trove-host
+
+[Mount]
+Type=nfs
+What=$trove_host:/home/cache/ccache
+Where=/srv/distbuild/ccache
+
+[Install]
+WantedBy=morph-worker.service
+EOF
diff --git a/distbuild/manifest b/distbuild/manifest
new file mode 100644
index 0000000..eba7c9f
--- /dev/null
+++ b/distbuild/manifest
@@ -0,0 +1,19 @@
+0040755 0 0 /lib
+0040755 0 0 /lib/systemd
+0040755 0 0 /lib/systemd/system-generators
+0100755 0 0 /lib/systemd/system-generators/ccache-nfs-mount-generator
+0040755 0 0 /usr
+0040755 0 0 /usr/lib
+0040755 0 0 /usr/lib/systemd
+0040755 0 0 /usr/lib/systemd/system
+0100755 0 0 /usr/lib/systemd/system/morph-cache-server.service
+0100755 0 0 /usr/lib/systemd/system/morph-controller.service
+0100755 0 0 /usr/lib/systemd/system/morph-controller-helper.service
+0100755 0 0 /usr/lib/systemd/system/morph-worker.service
+0100755 0 0 /usr/lib/systemd/system/morph-worker-helper.service
+0040755 0 0 /usr/lib/systemd/system/multi-user.target.wants
+0120755 0 0 /usr/lib/systemd/system/multi-user.target.wants/morph-cache-server.service
+0120755 0 0 /usr/lib/systemd/system/multi-user.target.wants/morph-controller.service
+0120755 0 0
/usr/lib/systemd/system/multi-user.target.wants/morph-controller-helper.service
+0120755 0 0 /usr/lib/systemd/system/multi-user.target.wants/morph-worker.service
+0120755 0 0 /usr/lib/systemd/system/multi-user.target.wants/morph-worker-helper.service
diff --git a/distbuild/usr/lib/systemd/system/morph-cache-server.service
b/distbuild/usr/lib/systemd/system/morph-cache-server.service
new file mode 100644
index 0000000..7ee5e16
--- /dev/null
+++ b/distbuild/usr/lib/systemd/system/morph-cache-server.service
@@ -0,0 +1,9 @@
+[Unit]
+Description=Morph cache server
+Requires=local-fs.target network.target
+After=local-fs.target network.target
+ConditionPathExists=/etc/morph-cache-server.conf
+
+[Service]
+ExecStart=/usr/bin/morph-cache-server
+Restart=always
diff --git a/distbuild/usr/lib/systemd/system/morph-controller-helper.service
b/distbuild/usr/lib/systemd/system/morph-controller-helper.service
new file mode 100644
index 0000000..9e5bd3c
--- /dev/null
+++ b/distbuild/usr/lib/systemd/system/morph-controller-helper.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=Morph distributed build controller helper
+Requires=morph-controller.service
+After=morph-controller.service
+ConditionPathExists=/etc/morph-controller.conf
+ConditionPathExists=/etc/morph-controller-helper.conf
+
+[Service]
+ExecStart=/usr/bin/distbuild-helper --config /etc/morph-controller-helper.conf
+Restart=always
diff --git a/distbuild/usr/lib/systemd/system/morph-controller.service
b/distbuild/usr/lib/systemd/system/morph-controller.service
new file mode 100644
index 0000000..b3ba93b
--- /dev/null
+++ b/distbuild/usr/lib/systemd/system/morph-controller.service
@@ -0,0 +1,9 @@
+[Unit]
+Description=Morph distributed build controller
+Requires=local-fs.target network.target
+After=local-fs.target network.target
+ConditionPathExists=/etc/morph-controller.conf
+
+[Service]
+ExecStart=/usr/bin/morph controller-daemon --config /etc/morph-controller.conf
+Restart=always
diff --git a/distbuild/usr/lib/systemd/system/morph-worker-helper.service
b/distbuild/usr/lib/systemd/system/morph-worker-helper.service
new file mode 100644
index 0000000..14c082c
--- /dev/null
+++ b/distbuild/usr/lib/systemd/system/morph-worker-helper.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=Morph distributed build worker helper
+Requires=morph-worker.service
+After=morph-worker.service
+ConditionPathExists=/etc/morph-worker.conf
+ConditionPathExists=/etc/morph-worker-helper.conf
+
+[Service]
+ExecStart=/usr/bin/distbuild-helper --config /etc/morph-worker-helper.conf
+Restart=always
diff --git a/distbuild/usr/lib/systemd/system/morph-worker.service
b/distbuild/usr/lib/systemd/system/morph-worker.service
new file mode 100644
index 0000000..26dcb9a
--- /dev/null
+++ b/distbuild/usr/lib/systemd/system/morph-worker.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=Morph distributed build worker
+Requires=local-fs.target network.target
+Wants=srv-distbuild-ccache.mount
+After=local-fs.target network.target srv-distbuild-ccache.mount
+ConditionPathExists=/etc/morph-worker.conf
+
+[Service]
+ExecStart=/usr/bin/morph worker-daemon --config /etc/morph-worker.conf
+Restart=always
diff --git
a/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-cache-server.service
b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-cache-server.service
new file mode 120000
index 0000000..e624a31
--- /dev/null
+++ b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-cache-server.service
@@ -0,0 +1 @@
+../morph-cache-server.service
\ No newline at end of file
diff --git
a/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller-helper.service
b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller-helper.service
new file mode 120000
index 0000000..1554df7
--- /dev/null
+++
b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller-helper.service
@@ -0,0 +1 @@
+../morph-controller-helper.service
\ No newline at end of file
diff --git
a/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller.service
b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller.service
new file mode 120000
index 0000000..9d92814
--- /dev/null
+++ b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-controller.service
@@ -0,0 +1 @@
+../morph-controller.service
\ No newline at end of file
diff --git
a/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker-helper.service
b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker-helper.service
new file mode 120000
index 0000000..298c582
--- /dev/null
+++
b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker-helper.service
@@ -0,0 +1 @@
+../morph-worker-helper.service
\ No newline at end of file
diff --git a/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker.service
b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker.service
new file mode 120000
index 0000000..3f8d58a
--- /dev/null
+++ b/distbuild/usr/lib/systemd/system/multi-user.target.wants/morph-worker.service
@@ -0,0 +1 @@
+../morph-worker.service
\ No newline at end of file
--
1.7.10.4
1:23 p.m.
New subject: [PATCH 2/5] Add distbuild stratum
---
distbuild.morph | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
create mode 100644 distbuild.morph
diff --git a/distbuild.morph b/distbuild.morph
new file mode 100644
index 0000000..896b911
--- /dev/null
+++ b/distbuild.morph
@@ -0,0 +1,20 @@
+name: distbuild
+kind: stratum
+description: Morph distributed build software
+build-depends:
+- morph: foundation
+- morph: core
+- morph: tools
+chunks:
+- name: bottle
+ repo: upstream:bottle
+ ref: baserock/morph
+ build-depends: []
+- name: morph-cache-server
+ repo: baserock:baserock/morph-cache-server
+ ref: master
+ build-depends: []
+- name: flup
+ repo: upstream:flup
+ ref: baserock/morph
+ build-depends: []
--
1.7.10.4
1:23 p.m.
New subject: [PATCH 3/5] Add armv7lhf distbuild system
---
distbuild-system-armv7lhf-highbank.morph | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
create mode 100644 distbuild-system-armv7lhf-highbank.morph
diff --git a/distbuild-system-armv7lhf-highbank.morph
b/distbuild-system-armv7lhf-highbank.morph
new file mode 100644
index 0000000..fdad7cc
--- /dev/null
+++ b/distbuild-system-armv7lhf-highbank.morph
@@ -0,0 +1,20 @@
+arch: armv7lhf
+configuration-extensions:
+- set-hostname
+- add-config-files
+- simple-network
+- nfsboot
+- install-files
+- distbuild
+- fstab
+description: Morph distributed build node for armv7lhf highbank
+kind: system
+name: distbuild-system-armv7lhf-highbank
+strata:
+- morph: build-essential
+- morph: core
+- morph: foundation
+- morph: bsp-armv7-highbank
+- morph: tools
+- morph: nfs
+- morph: distbuild
--
1.7.10.4
1:23 p.m.
New subject: [PATCH 4/5] Add distbuild config extension
---
distbuild.configure | 161 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 161 insertions(+)
create mode 100644 distbuild.configure
diff --git a/distbuild.configure b/distbuild.configure
new file mode 100644
index 0000000..31afc81
--- /dev/null
+++ b/distbuild.configure
@@ -0,0 +1,161 @@
+#!/bin/sh
+# Copyright (C) 2013-2014 Codethink Limited
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# This is a "morph deploy" configure extension to configure a Baserock
+# build node, as part of a distributed building cluster. It uses the
+# following variables from the environment:
+#
+# * DISTBUILD_CONTROLLER: if 'yes', machine is set up as the controller.
+# * DISTBUILD_WORKER: if 'yes', machine is set up as a worker.
+# * TROVE_ID: hostname and Trove prefix of the server to pull source
+# from and push built artifacts to.
+#
+# The following variable is optional:
+#
+# * ARTIFACT_CACHE_SERVER: by default artifacts are pushed to the same
+# Trove that served the source, but you can use a different one.
+#
+# The following variable is required for worker nodes only:
+#
+# * CONTROLLERHOST: hostname or IP address of distbuild controller machine.
+# * WORKER_SSH_KEY: identity used to authenticate with Trove
+#
+# The following variable is required for the controller node only:
+#
+# * WORKERS: hostnames or IP address of worker nodes, comma-separated.
+
+set -e
+
+
+# If ARTIFACT_CACHE_SERVER isn't set, default to $TROVE_ID.
+if [ "x$ARTIFACT_CACHE_SERVER" = x ]
+then
+ ARTIFACT_CACHE_SERVER="$TROVE_ID"
+fi
+
+set -u
+
+
+# Create the mount point for extra disk space.
+install -d -o 0 -g 0 -m 0755 "$1/srv/distbuild"
+
+
+# Configuration shared by all instances of Morph. For reasons of
+# convenience, this sets the controller-initiator-address, so
+# that when users invoke "morph", they don't need to manually
+# specify it, or to manually add another configuration file to
+# use. However, the setting may only be set for the distbuild
+# initiator (which is what the user invokes), so in ALL other
+# configuration files we unset it.
+#
+# We disable the disk space requirement checks on non-worker nodes.
+# They don't make sense unless you're building locally.
+
+cat <<EOF > "$1/etc/morph.conf"
+[config]
+log = /var/log/morph.log
+log-max = 100M
+cachedir = /srv/distbuild
+tempdir = /srv/distbuild/tmp
+trove-host = $TROVE_ID
+controller-initiator-address = $CONTROLLERHOST
+tempdir-min-space = 0
+cachedir-min-space = 0
+build-ref-prefix = $TROVE_ID
+artifact-cache-server = http://${ARTIFACT_CACHE_SERVER}:8080/
+git-resolve-cache-server = http://${TROVE_ID}:8080/
+EOF
+
+
+# Configuration for a distbuild controller. This configuration
+# file gets used by the systemd unit that runs the controller.
+# The existence of this configuration file triggers the systemd
+# unit, so we only create it on nodes that are meant to act as
+# the controller.
+
+if [ "$DISTBUILD_CONTROLLER" = True ]
+then
+ cat <<EOF > "$1/etc/morph-controller.conf"
+[config]
+log = /var/log/morph-controller.log
+log-max = 100M
+writeable-cache-server = http://${ARTIFACT_CACHE_SERVER}:8081/
+worker = $WORKERS
+controller-helper-address = 127.0.0.1
+EOF
+
+ # Configuration for the controller's helper process. This
+ # gets used by the systemd unit that starts the helper.
+
+cat <<EOF > "$1/etc/morph-controller-helper.conf"
+[config]
+log = /var/log/morph-controller-helper.log
+log-max = 100M
+parent-port = 5656
+parent-address = 127.0.0.1
+EOF
+
+fi
+
+# Configuration for a distbuild worker. This gets reference
+# by the systemd unit that starts the worker. Again, the
+# existence of morph-worker.conf triggers the systemd unit
+# to starts the worker, so we only create it on systems
+# that are meant to be a worker.
+
+if [ "$DISTBUILD_WORKER" = True ]
+then
+ cat <<EOF > "$1/etc/morph-worker.conf"
+[config]
+log = /var/log/morph-worker.log
+log-max = 100M
+controller-initiator-address =
+EOF
+
+ # This will be used for a systemd generator which
+ # nfs mounts the ccache from the trove
+ echo "$TROVE_ID" > "$1/etc/trove-host"
+
+ # Configuration for the controller's helper process. This
+ # gets used by the systemd unit that starts the helper.
+
+cat <<EOF > "$1/etc/morph-worker-helper.conf"
+[config]
+log = /var/log/morph-worker-helper.log
+log-max = 100M
+parent-address = 127.0.0.1
+EOF
+
+
+ # Configuration for the Morph cache server daemon. We
+ # only run the cache server on workers.
+
+cat <<EOF > "$1/etc/morph-cache-server.conf"
+[config]
+port = 8080
+artifact-dir = /srv/distbuild/artifacts
+direct-mode = True
+fcgi-server = False
+EOF
+
+echo "Copy distbuild worker SSH key to system"
+mkdir -p "$1/root/.ssh"
+install -m 0600 "$WORKER_SSH_KEY" "$1/root/.ssh/id_rsa"
+install -m 0644 "${WORKER_SSH_KEY}.pub" "$1/root/.ssh/id_rsa.pub"
+
+# Add trove's host key
+ssh-keyscan -t dsa,ecdsa,rsa "$TROVE_ID" >>
"$1/root/.ssh/known_hosts"
+fi
--
1.7.10.4
1:23 p.m.
New subject: [PATCH 5/5] distbuild: Add example cluster for deployment
From: Sam Thursfield <sam.thursfield(a)codethink.co.uk>
Since the nodes are deployed as NFS root file systems, there's no
VERSION_LABEL field in this cluster, as there is no upgrade method.
The build nodes are stateless and deployed fresh each time.
---
example-distbuild-cluster.morph | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
create mode 100644 example-distbuild-cluster.morph
diff --git a/example-distbuild-cluster.morph b/example-distbuild-cluster.morph
new file mode 100644
index 0000000..736a5be
--- /dev/null
+++ b/example-distbuild-cluster.morph
@@ -0,0 +1,37 @@
+name: example-distbuild-cluster
+kind: cluster
+description: |
+ This is an example cluster morph that can be adapted to set up a
+ Baserock distributed build network.
+
+ You will need to deploy a Trove for the distributed build network
+ to use before deploying this cluster. The worker SSH key file should
+ be generated as part of the Trove deployment. It is the key used by
+ workers to authenticate with the Trove to give them read access to
+ all source repositories.
+systems:
+- morph: distbuild-system-x86_64-generic
+ deploy-defaults:
+ TROVE_ID: $MY_TROVE
+ CONTROLLERHOST: build-controller
+ DISTBUILD_CONTROLLER: no
+ DISTBUILD_WORKER: yes
+ FSTAB_SRC: LABEL=src /srv/distbuild auto defaults,rw,noatime 0 2
+ INSTALL_FILES: distbuild/manifest
+ NFSBOOT_CONFIGURE: yes
+ WORKER_SSH_KEY: ssh-keys/worker.key
+ deploy:
+ build-controller:
+ type: nfsboot
+ location: $MY_TROVE
+ DISTBUILD_CONTROLLER: yes
+ HOSTNAME: build-controller
+ WORKERS: build-node-1, build-node-2
+ build-node-1:
+ type: nfsboot
+ location: $MY_TROVE
+ HOSTNAME: build-node-1
+ build-node-2:
+ type: nfsboot
+ location: $MY_TROVE
+ HOSTNAME: build-node-2
--
1.7.10.4
1:28 p.m.
On Fri, Mar 28, 2014 at 06:23:37PM +0000, Richard Ipsum wrote:
repo: git@git.baserock.org:baserock/baserock/definitions
ref: baserock/richardipsum/distbuild-with-setup2
head: 84cb06607e2530757e8ab545b10de5fae3b6862a
Looks fine to me, +2. (Obviously I didn't have time to test, but I
know Richard's been using these and I've written some of it, so I view
this only as a dump of presumed-working files, and if it fails, it
doesn't break existing stuff and we find and fix problems faster if
more people have access to this.)
--
http://www.codethink.co.uk/ http://wiki.baserock.org/ http://www.baserock.com/
3283
days inactive
3283
days old
6 comments
2 participants
participants (2)
-
Lars Wirzenius -
Richard Ipsum