On 17/12/14 13:29, Lauren Perry wrote: Installing scripts in /usr/bin with a .sh extension is weird, the script should just be called `firehose` or something if possible. A better way would be to rewrite this program in Python and use `pkg_resources.get_provider('firehose').module_path`. for clarity, 'it' here refers to the 'firehose' Python module -- Sam Thursfield, Codethink Ltd. Office telephone: +44 161 236 5575

--- morph | 17 ----------------- 1 file changed, 17 deletions(-) delete mode 100755 morph diff --git a/morph b/morph deleted file mode 100755 index d92fe4c..0000000 --- a/morph +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/sh - -MORPH="${MORPH:-$(which morph)}" - -BASE="$(dirname $0)" -BASE="${BASE:-.}" -BASE="$(realpath ${BASE})" - -MORPH_PLUGIN_PATH="${BASE}/plugin" - -export MORPH_PLUGIN_PATH - -PYTHONPATH="${BASE}${PYTHONPATH:+:${PYTHONPATH}}:" - -export PYTHONPATH - -exec ${MORPH} "$@" -- 1.7.10.4

On 17/12/14 13:29, Lauren Perry wrote: These values should be configurable (with sensible defaults provided) because they need to tie in with the bot's Gerrit credentials, and different deployments may have different constraints on Gerrit user accounts. -- Sam Thursfield, Codethink Ltd. Office telephone: +44 161 236 5575

On 06/01/15 09:51, Lauren Perry wrote: This is a bit of an awkward question ... here's how I picture an operations team member deploying a new Firehose instance. It's not the nicest way, but I think it's the most easily achievable with the tools we have today. 1. Op has a Gerrit instance running already. 2. Op has a 'infrastructure-definitions' git repo cloned from baserock:baserock/definitions 3. Op writes a 'firehose.morph' cluster morphology and commits it to their infrastructure-definitions.git. It specifies some flags like: GERRIT_URL, FIREHOSE_FULL_NAME, FIREHOSE_EMAIL, FIREHOSE_GERRIT_USERNAME 4. Op runs 'morph deploy firehose.morph' and gets a new Firehose instance. 5. Op creates the necessary user accounts on their Gerrit instance (ideally, we provide an Ansible script to help with automating this). I think that having the firehose.configure extension in stage 4 fail if the account details are invalid would be annoying. It would be possible, if we swapped stage 4 and stage 5. But the operator should be able to easily see error logs from the Firehose machine any time they want (there is more that can go wrong with it than just invalid credentials, anyway), so if they fail to do step 5 and their Firehose instance doesn't work, it should be easy for them to find out why. Let me know if this makes sense! Sam -- Sam Thursfield, Codethink Ltd. Office telephone: +44 161 236 5575

On 17/12/14 13:29, Lauren Perry wrote: This URL also needs to be configurable (given 'gerrit_username' and 'gerrit_url' options it can be worked out). -- Sam Thursfield, Codethink Ltd. Office telephone: +44 161 236 5575

On 17/12/14 13:29, Lauren Perry wrote: Add a note to say that this a temporary hack, and really firehose.git should be added as a chunk in baserock's definitions.git repo (or better, actually add firehose as a chunk :) ... Here's another bit which needs to be configurable. -- Sam Thursfield, Codethink Ltd. Office telephone: +44 161 236 5575

On 05/01/15 12:11, Pedro Alvarez wrote: Yes, looking back now, the .configure file needs more work. At the moment this is more detailing the commands required to get firehose working from inside the firehose machine, so I assume if it is ran from inside the deployed firehose system it works, however we require the system be deployed in the configuration if I remember correctly, so as you say this would only perform configuration in the development system and not where we want it. I will revise this patch so that it can be ran from development to configure the deployed firehose.

--- firehose.sh | 21 +++++++++++++++++++++ {plugin => firehose/plugin}/firehose_plugin.py | 0 setup.py | 16 ++++++++++++++++ 3 files changed, 37 insertions(+) create mode 100755 firehose.sh rename {plugin => firehose/plugin}/firehose_plugin.py (100%) create mode 100644 setup.py diff --git a/firehose.sh b/firehose.sh new file mode 100755 index 0000000..4d272d3 --- /dev/null +++ b/firehose.sh @@ -0,0 +1,21 @@ +#!/bin/sh + +MORPH="${MORPH:-$(which morph)}" + +BASE="$(dirname $0)" +BASE="${BASE:-.}" +BASE="$(realpath ${BASE})" + +# Currently the only way to get this to work is to hardcode the path of the plugin. +# Obviously this is not ideal, but until a working method is obtained this remains +# our best option at present. + +MORPH_PLUGIN_PATH="/usr/lib/python2.7/site-packages/firehose-0.1-py2.7.egg/firehose/plugin" + +export MORPH_PLUGIN_PATH + +PYTHONPATH="${BASE}${PYTHONPATH:+:${PYTHONPATH}}:" + +export PYTHONPATH + +exec ${MORPH} firehose "$@" diff --git a/plugin/firehose_plugin.py b/firehose/plugin/firehose_plugin.py similarity index 100% rename from plugin/firehose_plugin.py rename to firehose/plugin/firehose_plugin.py diff --git a/setup.py b/setup.py new file mode 100644 index 0000000..418f4c6 --- /dev/null +++ b/setup.py @@ -0,0 +1,16 @@ +from setuptools import setup +setup( + name = "firehose", + version = "0.1", + packages = ["firehose", "debian"], + package_data={ + "firehose": + ["plugin/*.py"] + }, + scripts = [ + "firehose.sh" + ], + # This is because morph won't find it as a plugin if it is + # installed as a .egg file + zip_safe = False +) -- 1.7.10.4

--- morph | 17 ----------------- 1 file changed, 17 deletions(-) delete mode 100755 morph diff --git a/morph b/morph deleted file mode 100755 index d92fe4c..0000000 --- a/morph +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/sh - -MORPH="${MORPH:-$(which morph)}" - -BASE="$(dirname $0)" -BASE="${BASE:-.}" -BASE="$(realpath ${BASE})" - -MORPH_PLUGIN_PATH="${BASE}/plugin" - -export MORPH_PLUGIN_PATH - -PYTHONPATH="${BASE}${PYTHONPATH:+:${PYTHONPATH}}:" - -export PYTHONPATH - -exec ${MORPH} "$@" -- 1.7.10.4

--- firehose/plugin/firehose_plugin.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/firehose/plugin/firehose_plugin.py b/firehose/plugin/firehose_plugin.py index fba1db3..dd6f29b 100644 --- a/firehose/plugin/firehose_plugin.py +++ b/firehose/plugin/firehose_plugin.py @@ -81,6 +81,7 @@ class FirehosePlugin(cliapp.Plugin): self.base_path = tfs.getsyspath("/") self.make_workspace() self.make_branch(confs[0].landing) + self.insert_githook() self.reset_to_tracking(confs[0].landing) self.load_morphologies() for c in confs: @@ -106,6 +107,11 @@ class FirehosePlugin(cliapp.Plugin): repopath = root['repo'].replace(':', '/') self.gitpath = self.make_path("ws", root['myref'], repopath) + def insert_githook(self): + scp_cmd = ('scp', '-p', '-P', '29418', 'firehose@testgerrit.baserock.org:hooks/commit-msg', + self.gitpath+'/.git/hooks/commit-msg') + os.system(' '.join(scp_cmd)) + def reset_to_tracking(self, root): branch_head_sha = self.app.runcmd(['git', 'rev-parse', 'HEAD'], cwd=self.gitpath).strip() -- 1.7.10.4

--- README | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/README b/README index d22719b..425a1fd 100644 --- a/README +++ b/README @@ -14,6 +14,55 @@ where that holds true, Firehose can provide a valuable service pre-integrating and running those integrations through a CD pipeline to gain confidence in the proposed change. +Presently Required Instructions to Run Firehose +=============================================== + +Firehose deployment isn't automated yet. Here are instructions on how to do it +manually: + +In a 'build' machine in the cloud: + + morph build systems/build-system-x86_64.morph + morph deploy baserock-firehose/baserock-firehose.morph + +Instatiate this image and SSH into it. + + git clone git://git.baserock.org/baserock/baserock/firehose + git checkout baserock/lauren/firehose + python ./setup.py install + adduser firehose -s /bin/false -D + sudo -u firehose -- ssh-keygen -C 'firehose(a)baserock.org' -N '' + sudo chown -R firehose:firehose /var/lib/firehose + +Create an account on the Gerrit instance where Firehose will be pushing: + + ssh -p 29418 you(a)gerrit.baserock.org gerrit \ + create-account --group "'Non-Interactive Users'" \ + --ssh-key xxxx --full-name "'Firehose integration tool'" \ + --email='firehose(a)firehose.baserock.org' \ + firehose + +Test that the Firehose user can connect to Gerrit (this should give you a help +message if successful, or a 'Permission denied' error otherwise): + + sudo -u firehose -- ssh -p 29418 firehose(a)gerrit.baserock.org gerrit + +Move firehose.service and firehose.timer files to /etc/systemd/system/ + +Run: + + systemctl enable firehose.timer + systemctl start firehose.timer + systemctl enable firehose.service + +As Firehose, ensure that the git config matches what gerrit expects W.R.T email address, i.e. + + git config --global user.email firehose(a)firehose.baserock.org + +If Firehose fails due to an error in /tmp/morph_tmp/, edit /etc/morph.conf to include the following: + + [config] + repo-alias=baserock=git://git.baserock.org/baserock/#ssh://firehose@testg... Debian Version Comparison ========================= -- 1.7.10.4

--- firehose.configure | 33 ++++++++++++++++++++++++++------- firehose.sh | 2 +- firehose/plugin/firehose_plugin.py | 24 +++++++++--------------- 3 files changed, 36 insertions(+), 23 deletions(-) diff --git a/firehose.configure b/firehose.configure index c17fe59..27a5fe6 100644 --- a/firehose.configure +++ b/firehose.configure @@ -3,7 +3,7 @@ set -e ROOT="$1" -# Clone the Firehose repo +# Clone the Firehose repo (temporary hack) git clone git://git.baserock.org/baserock/baserock/firehose # Move Firehose arguments to specific directory and timer/service @@ -28,13 +28,32 @@ systemctl enable firehose.timer systemctl start firehose.timer systemctl enable firehose.service +# Check necessary parameters were filled in firehose.morph before +# the system was deployed, and exit if not +if + [ -z "$GERRIT_USERNAME" -a \ + -z "$GERRIT_URL" -a \ + -z "$GIT_USERNAME" -a \ + -z "$GIT_EMAIL" -a]; then + exit 0 +fi + +# Create firehose.conf to store parameters obtained from .morph file +cat >>"$ROOT"/etc/firehose.conf <<EOF +[config] +gerrit_username = $GERRIT_USERNAME +gerrit_url = $GERRIT_URL +git_username = $GIT_USERNAME +git_email = $GIT_EMAIL +EOF + # Add firehose user, then become firehose user -adduser firehose -s /bin/false -D -sudo -u firehose --ssh-keygen -C &quot;firehose(a)baserock.org&quot; -N "" -sudo chown -R firehose:firehose /var/lib/firehose/ -sudo -u firehose -s +adduser "$GERRIT_USER" -s /bin/false -D +sudo -u "$GERRIT_USER" --ssh-keygen -C &quot;$GERRIT_USER(a)baserock.org&quot; -N "" +sudo chown -R "$GERRIT_USER":"$GERRIT_USER" /var/lib/firehose +sudo -u "$GERRIT_USER" -s # Configure a git identity for Firehose -git config --global user.name "Firehose" -git config --global user.email firehose(a)firehose.baserock.org +git config --global user.name "$GIT_USER" +git config --global user.email "$GIT_EMAIL" diff --git a/firehose.sh b/firehose.sh index 4d272d3..a61ba6c 100755 --- a/firehose.sh +++ b/firehose.sh @@ -18,4 +18,4 @@ PYTHONPATH="${BASE}${PYTHONPATH:+:${PYTHONPATH}}:" export PYTHONPATH -exec ${MORPH} firehose "$@" +exec ${MORPH} firehose "$@" --config=/etc/firehose.conf diff --git a/firehose/plugin/firehose_plugin.py b/firehose/plugin/firehose_plugin.py index 66116bf..5e9a55b 100644 --- a/firehose/plugin/firehose_plugin.py +++ b/firehose/plugin/firehose_plugin.py @@ -26,23 +26,13 @@ import morphlib from debian.debian_support import Version @contextmanager -def firehose_git(app): - try: - username = (app.runcmd_unchecked(["git", "config", "--global", "user.name"]))[1].strip() - email = (app.runcmd_unchecked(["git", "config", "--global", "user.email"]))[1].strip() - app.runcmd(["git", "config", "--global", "user.name", "Firehose merge bot"]) - app.runcmd(["git", "config", "--global", "user.email", &quot;firehose(a)firehose.baserock.org&quot;]) - yield () - finally: - app.runcmd(["git", "config", "--global", "user.name", username]) - app.runcmd(["git", "config", "--global", "user.email", email]) - - class FirehosePlugin(cliapp.Plugin): def enable(self): self.app.add_subcommand('firehose', self.firehose_cmd, arg_synopsis='some-firehose.yaml...') + self.app.settings(['gerrit_username'], 'gerrit_username') + self.app.settings(['gerrit_url'], 'gerrit_url') def disable(self): pass @@ -120,7 +110,9 @@ class FirehosePlugin(cliapp.Plugin): log.write('{\"sha\":\"'+sha+'\"}') def insert_githook(self): - scp_cmd = ('scp', '-p', '-P', '29418', 'firehose@testgerrit.baserock.org:hooks/commit-msg', + gerrit_username = self.app.settings['gerrit_username'] + gerrit_url = self.app.settings['gerrit_url'] + scp_cmd = ('scp', '-p', '-P', '29418', gerrit_username+'@'+gerrit_url+':hooks/commit-msg', self.gitpath+'/.git/hooks/commit-msg') os.system(' '.join(scp_cmd)) @@ -249,12 +241,14 @@ class FirehosePlugin(cliapp.Plugin): return True def commit_and_push(self, name): + gerrit_username = self.app.settings['gerrit_username'] + gerrit_url = self.app.settings['gerrit_url'] commit_msg = 'Update to '+name+' ref in definitions.git' branch_name = 'HEAD:refs/for/master/firehose/'+name (code, out, err) = self.app.runcmd_unchecked( ['git', 'commit', '-a', '-m', commit_msg], cwd=self.gitpath) if code == 0: - self.app.runcmd(['git', 'push', 'ssh://firehose@testgerrit.baserock.org:29418/baserock/baserock/definitions', - branch_name], cwd=self.gitpath) + self.app.runcmd(['git', 'push', 'ssh://'+gerrit_username+'@'+gerrit_url+ + ':29418/baserock/baserock/definitions', branch_name], cwd=self.gitpath) -- 1.7.10.4