Gitweb links: ...log http://git.netsurf-browser.org/netsurf-wiki.git/shortlog/da0510b2c6a1e9a1... ...commit http://git.netsurf-browser.org/netsurf-wiki.git/commit/da0510b2c6a1e9a191... ...tree http://git.netsurf-browser.org/netsurf-wiki.git/tree/da0510b2c6a1e9a1910a... The branch, master has been updated via da0510b2c6a1e9a1910a4ec7b8bbb7d5feb7dfde (commit) from 6f50fa1169888494aca906918be866a6de490943 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commitdiff http://git.netsurf-browser.org/netsurf-wiki.git/commit/?id=da0510b2c6a1e9... commit da0510b2c6a1e9a1910a4ec7b8bbb7d5feb7dfde Author: Vincent Sanders <vince(a)kyllikki.org&gt; Commit: Vincent Sanders <vince(a)kyllikki.org&gt; add buster ci install page diff --git a/continuous_integration.mdwn b/continuous_integration.mdwn index e494d59..fadaeeb 100644 --- a/continuous_integration.mdwn +++ b/continuous_integration.mdwn @@ -34,6 +34,8 @@ The majority of the build slaves are provided by our own [[virtual host server|virtual_host_server]] excepting operating systems which cannot be virtualised (principally MAC OS X). +### [[Debian 10 (Buster) Setup|continuous_integration/debian_buster_setup]] + ### [[Debian 9 (Stretch) Setup|continuous_integration/debian_stretch_setup]] ### [[Fedora 26 Setup|continuous_integration/fedora_26_setup]] diff --git a/continuous_integration/debian_buster_setup.mdwn b/continuous_integration/debian_buster_setup.mdwn new file mode 100644 index 0000000..b216b14 --- /dev/null +++ b/continuous_integration/debian_buster_setup.mdwn @@ -0,0 +1,349 @@ +[[!meta title="Continuous Integration Debian Buster Setup"]] +[[!meta author="Kyllikki"]] +[[!meta date="2018-12-21T12:46:14Z"]] + + +[[!toc]] + +## Debian 9 (Buster) OS install + +### amd64 VDS install from media + +[[Virtual server setup|virtual_host_server]] + +Install minimal system from netinst CD (attached when VDS is created on +phoenix) Config options: + +- In the "role" selection select "ssh server" and "system utilities" only. +- The whole disc default partitioning is fine +- The base user the install insists on creating should be the netsurf user. +- Boot loader in MBR + +Once installed: + +- install sudo package and add netsurf user to sudo group +- edit /etc/default/grub + + GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8" + +as root update grub + + update-grub2 + +### arm64 VDS install + +[[ARM64 virtual server setup|virtual_host_server_arm64]] + +Once installed: + +- create netsurf user +- install sudo package and add netsurf user to sudo group +- edit /etc/inittab comment pty 3 through 6 and uncomment serial T0 +- enable backports + +`echo "deb `[`http://http.debian.net/debian`](http://http.debian.net/debian)` buster-backports main" > /etc/apt/sources.list.d/backports.list` +`apt-get update` + + +## Packaged CI worker install + +Do a base OS install + +The recommended hostname for CI workers is "nsciworker17" this allows us +to clearly identify CI worker nodes. Note historically we have used +"cislave1" which has been objected to by several users. See +[[changing hostname|https://wiki.debian.org/howto/changehostname_changing_hostname]] on how to achive this. + +On master jenkins use "manage nodes" to create new node. Ensure "remote +fs root" is set to /var/lib/jenkins add variable JENKINS\_HOME set to +/var/lib/jenkins + +Note: replace arm64 with architecture name as required (armhf etc.) + +When configuring a scaleway instance the "Launch method" should be set +to "start and stop this node on-demand" with the "Start script" set to +`perl /usr/bin/pscw.pl start netsurf-ciworker17` and "Stop script" set +to `perl /usr/bin/pscw.pl stop netsurf-ciworker17`. The Availability +set to "Take this slave on-line when in demand and off-line when idle" +and "in demand delay" set to 0 and "idle delay" set to 15. This will +mean the node is turned off and not charged for when idle. + +As superuser: + +create jenkins user + + adduser --system --group --home /var/lib/jenkins/ --disabled-login jenkins + +Install https transport + + apt-get install apt-transport-https + +Add CI server repo to slave apt sources + + echo "deb https://ci.netsurf-browser.org/debian/ buster/amd64/" >> /etc/apt/sources.list.d/netsurf-browser.list + +update repos + + apt-get update + +install ci worker package. accept the large package list and the + unsigned package install for ns-ci-slave + + apt-get install ns-ci-worker + +edit /etc/default/ns-ci-worker to set the correct url and secret parameters + +ensure /opt is setup correctly to allow toolchains to be built on the node + + mkdir -p /opt/netsurf + chown jenkins:jenkins /opt/netsurf + +become jenkins user + + su -s/bin/bash - jenkins + +create ssh keypair (accept defaults - no password) + + ssh-keygen -t rsa -C &quot;netsurf(a)nsciworker12.netsurf-browser.org&quot; + +copy .ssh/id\_rsa.pub from worker to jenkins master node and append + to /home/netsurf/.ssh/authorized\_keys + + scp /home/jenkins/.ssh/id_rsa.pub netsurf@ci.netsurf-browser.org:nsciworker12_id_rsa.pub + +exit jenkins user shell + +start CI worker daemon + + /etc/init.d/ns-ci-worker start + + + +## Pbuilder setup + +This allows a worker to build Debian packages. The worker should be +installed as a normal CI worker node and then: + +as superuser on node: + + # apt-get install pbuilder + # addgroup pbuilder + # addgroup jenkins pbuilder + +create /etc/sudoers.d/pbuilder + + jenkins ALL = NOPASSWD:/usr/sbin/pbuilder + +`visudo` and alter Defaults + + Defaults env_reset,env_keep="DIST ARCH" + +replace `/etc/pbuilderrc` + + # this is your configuration file for pbuilder. + # the file in /usr/share/pbuilder/pbuilderrc is the default template. + # /etc/pbuilderrc is the one meant for overwriting defaults in + # the default template + # + # read pbuilderrc.5 document for notes on specific options. + + # List of Debian suites. + DEBIAN_SUITES=("sid", "buster", "jessie", "wheezy", "squeeze") + + # List of Ubuntu suites. + UBUNTU_SUITES=("vivid" "utopic" "trusty" "saucy" "raring" "quantal" "precise" "oneiric" "natty" "lucid" "hardy") + + # Mirrors to use. Update these to your preferred mirror. + DEBIAN_MIRROR="ftp.uk.debian.org" + UBUNTU_MIRROR="mirrors.kernel.org" + + # set a default distribution if none is used. + : ${DIST:="$(lsb_release --short --codename)"} + + # set the architecture to the host architecture if none set. + : ${ARCH:="$(dpkg --print-architecture)"} + + NAME="$DIST" + if [ -n "${ARCH}" ]; then + NAME="$NAME-$ARCH" + DEBOOTSTRAPOPTS=("--arch" "$ARCH" "${DEBOOTSTRAPOPTS[@]}") + fi + BASETGZ="/var/cache/pbuilder/$NAME-base.tgz" + DISTRIBUTION="$DIST" + BUILDRESULT="/var/cache/pbuilder/$NAME/result/" + APTCACHE="/var/cache/pbuilder/$NAME/aptcache/" + BUILDPLACE="/var/cache/pbuilder/build/" + + if $(echo ${DEBIAN_SUITES[@]} | grep -q $DIST); then + # Debian configuration + MIRRORSITE="http://$DEBIAN_MIRROR/debian/" + COMPONENTS="main contrib non-free" + if $(echo "$STABLE_CODENAME stable" | grep -q $DIST); then + OTHERMIRROR="$OTHERMIRROR | deb $MIRRORSITE $STABLE_BACKPORTS_SUITE $COMPONENTS" + fi + elif $(echo ${UBUNTU_SUITES[@]} | grep -q $DIST); then + # Ubuntu configuration + MIRRORSITE="http://$UBUNTU_MIRROR/ubuntu/" + COMPONENTS="main restricted universe multiverse" + else + echo "Unknown distribution: $DIST" + exit 1 + fi + +The architecture is assumed to be the native one from + + dpkg --print-architecture + +This can be set by passing ARCH to pbuilder (useful for i386 maybe?) + +for each distribution this node will build for: + +create pbuilder result directory and set ownership permissions + + # mkdir -p /var/cache/pbuilder/buster-armhf/result + # chown root:pbuilder /var/cache/pbuilder/buster-armhf/result + # chmod g+w /var/cache/pbuilder/buster-armhf/result + +become jenkins user + + su -s/bin/bash - jenkins + +create pbuilder base for distribution + + sudo DIST=buster pbuilder create + +if desired additional packages and config can be made to the base with + +`sudo DIST=buster pbuilder login --save-after-login` + +## distcc worker node + +Do a basic OS install but \*not\* a CI worker setup. + +A recommended hostname for distcc worker is something like "cicpu0" this +allows us to use systems as processing node for other purposes than just +distcc in future. See debians +[[changing hostname|https://wiki.debian.org/howto/changehostname_changing_hostname]] on how to achive this. + +The Netsurf repository has necessary updated packages in it and can be +accessed by doing the following: + +Add CI server repo to worker apt sources + + echo "deb https://ci.netsurf-browser.org/builds/debian/ buster/amd64/" >> /etc/apt/sources.list + +update repos + + apt-get update + +use apt to install these packages: + + build-essential + gcc + clang + distcc + +edit /etc/default/distcc + + STARTDISTCC="true" + ALLOWEDNETS="192.168.211.0/24" + LISTEN="0.0.0.0" + JOBS="8" + +start the service + + service distcc start + +ensure the client has hosts set to use the new worker + + + +## Manual CI worker install + +Caution these instructions may not be up to date. + +### required packages + +The Netsurf repository has necessary updated packages in it and can be +accessed by doing the following: + +Add CI server repo to slave apt sources + +`echo "deb `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)` buster/amd64/" >> /etc/apt/sources.list` + +update repos + + apt-get update + +use apt to install these packages: + +`openjdk-7-jre-headless ` +`screen ` +`build-essential` +`ccache` +`clang` +`git` +`pkg-config` +`check` +`doxygen` +`libjson0-dev (from our repo - needs bugfixes `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)`)` +`libexpat1-dev` +`libxml-perl` +`libxml-xpath-perl` +`lcov` +`gcovr (from our repo)` +`gperf` +`flex` +`bison` +`libpng-dev` +`libjpeg-dev` +`libmozjs185-dev` +`libglib2.0-dev` +`libcurl4-openssl-dev` +`liblcms1-dev` +`libxml2-dev` +`librsvg2-dev` +`libmng-dev` +`libgtk2.0-dev` +`libmozjs-dev` + +### config + +- on master jenkins use "manage nodes" to create new node. Ensure + "remote fs root" is set to /home/netsurf/jenkins +- create netsurf user +- as netsurf user: + - wget <http://ci.netsurf-browser.org/jenkins/jnlpJars/slave.jar> + - run screen + - create jenkins-slave.sh + + #!/bin/bash + + java -Djava.awt.headless=true -jar slave.jar -jnlpUrl https://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp -secret 0123456789abcdef01234567890abcdef + + - run jenkins-slave.sh + - create new screen tab + - create ssh keypair (accept defaults - no password) + + ssh-keygen -t rsa -C &quot;netsurf(a)cislave0.netsurf-browser.org&quot; + +- - copy .ssh/id\_rsa.pub from slave to jenkins master node and + append to /home/netsurf/.ssh/authorized\_keys + + scp ci.netsurf-browser.org:.ssh/id_rsa.pub .id_rsa.pub + cat id_rsa.pub >> .ssh/authorized_keys + +- - copy .ssh/id\_rsa.pub from master node to slave and append to + /home/netsurf/.ssh/authorized\_keys + - create reverse-ssh.sh (change tunnel port number!) + + #!/bin/sh + + ssh -R 22224:localhost:22 netsurf(a)ci.netsurf-browser.org 'bash -c "while true; do echo .; sleep 60; done" + +- - run reverse-ssh.sh + - on the master create a shell script to use the ssh tunnel + connection, thus firewalls etc are moot as long as the slave can + connect to the master + + ssh netsurf@localhost -p 22223 diff --git a/continuous_integration/debian_stretch_setup.mdwn b/continuous_integration/debian_stretch_setup.mdwn index 2333455..557472e 100644 --- a/continuous_integration/debian_stretch_setup.mdwn +++ b/continuous_integration/debian_stretch_setup.mdwn @@ -309,10 +309,10 @@ Add CI server repo to slave apt sources update repos - apt-getupdate + apt-get update install ci worker package. accept the large package list and the - unsigned package install for gcovr and ns-ci-slave + unsigned package install for ns-ci-slave apt-get install ns-ci-worker @@ -361,9 +361,9 @@ create /etc/sudoers.d/pbuilder `visudo` and alter Defaults -`Defaults        env_reset,env_keep="DIST ARCH"` + Defaults env_reset,env_keep="DIST ARCH" -replace /etc/pbuilderrc +replace `/etc/pbuilderrc` # this is your configuration file for pbuilder. # the file in /usr/share/pbuilder/pbuilderrc is the default template. @@ -417,7 +417,7 @@ replace /etc/pbuilderrc The architecture is assumed to be the native one from -`dpkg --print-architecture` + dpkg --print-architecture This can be set by passing ARCH to pbuilder (useful for i386 maybe?) @@ -431,11 +431,11 @@ create pbuilder result directory and set ownership permissions become jenkins user -`su -s/bin/bash - jenkins` + su -s/bin/bash - jenkins create pbuilder base for distribution -`sudo DIST=stretch pbuilder create` + sudo DIST=stretch pbuilder create if desired additional packages and config can be made to the base with @@ -455,18 +455,18 @@ accessed by doing the following: Add CI server repo to worker apt sources -`echo "deb `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)` stretch/amd64/" >> /etc/apt/sources.list` + echo "deb https://ci.netsurf-browser.org/builds/debian/ stretch/amd64/" >> /etc/apt/sources.list update repos -`apt-get update` + apt-get update use apt to install these packages: -`build-essential` -`gcc` -`clang` -`distcc` + build-essential + gcc + clang + distcc edit /etc/default/distcc @@ -477,7 +477,7 @@ edit /etc/default/distcc start the service -`service distcc start` + service distcc start ensure the client has hosts set to use the new worker @@ -492,13 +492,13 @@ Caution these instructions may not be up to date. The Netsurf repository has necessary updated packages in it and can be accessed by doing the following: -- Add CI server repo to slave apt sources +Add CI server repo to slave apt sources `echo "deb `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)` stretch/amd64/" >> /etc/apt/sources.list` -- update repos +update repos -`apt-get update` + apt-get update use apt to install these packages: @@ -542,33 +542,33 @@ use apt to install these packages: - run screen - create jenkins-slave.sh -`#!/bin/bash` - -`java -Djava.awt.headless=true -jar slave.jar -jnlpUrl `[`http://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp`](http://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp)` -secret 0123456789abcdef01234567890abcdef` + #!/bin/bash + + java -Djava.awt.headless=true -jar slave.jar -jnlpUrl https://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp -secret 0123456789abcdef01234567890abcdef -- - run jenkins-slave.sh + - run jenkins-slave.sh - create new screen tab - create ssh keypair (accept defaults - no password) -`ssh-keygen -t rsa -C &quot;netsurf(a)cislave0.netsurf-browser.org&quot;` + ssh-keygen -t rsa -C &quot;netsurf(a)cislave0.netsurf-browser.org&quot; - - copy .ssh/id\_rsa.pub from slave to jenkins master node and append to /home/netsurf/.ssh/authorized\_keys -`scp ci.netsurf-browser.org:.ssh/id_rsa.pub .id_rsa.pub` -`cat id_rsa.pub >> .ssh/authorized_keys` + scp ci.netsurf-browser.org:.ssh/id_rsa.pub .id_rsa.pub + cat id_rsa.pub >> .ssh/authorized_keys - - copy .ssh/id\_rsa.pub from master node to slave and append to /home/netsurf/.ssh/authorized\_keys - create reverse-ssh.sh (change tunnel port number!) -`#!/bin/sh` - -`ssh -R 22224:localhost:22 netsurf(a)ci.netsurf-browser.org 'bash -c "while true; do echo .; sleep 60; done"'` + #!/bin/sh + + ssh -R 22224:localhost:22 netsurf(a)ci.netsurf-browser.org 'bash -c "while true; do echo .; sleep 60; done" - - run reverse-ssh.sh - on the master create a shell script to use the ssh tunnel connection, thus firewalls etc are moot as long as the slave can connect to the master -`ssh netsurf@localhost -p 22223` + ssh netsurf@localhost -p 22223 ----------------------------------------------------------------------- Summary of changes: continuous_integration.mdwn | 2 + continuous_integration/debian_buster_setup.mdwn | 349 ++++++++++++++++++++++ continuous_integration/debian_stretch_setup.mdwn | 56 ++-- 3 files changed, 379 insertions(+), 28 deletions(-) create mode 100644 continuous_integration/debian_buster_setup.mdwn diff --git a/continuous_integration.mdwn b/continuous_integration.mdwn index e494d59..fadaeeb 100644 --- a/continuous_integration.mdwn +++ b/continuous_integration.mdwn @@ -34,6 +34,8 @@ The majority of the build slaves are provided by our own [[virtual host server|virtual_host_server]] excepting operating systems which cannot be virtualised (principally MAC OS X). +### [[Debian 10 (Buster) Setup|continuous_integration/debian_buster_setup]] + ### [[Debian 9 (Stretch) Setup|continuous_integration/debian_stretch_setup]] ### [[Fedora 26 Setup|continuous_integration/fedora_26_setup]] diff --git a/continuous_integration/debian_buster_setup.mdwn b/continuous_integration/debian_buster_setup.mdwn new file mode 100644 index 0000000..b216b14 --- /dev/null +++ b/continuous_integration/debian_buster_setup.mdwn @@ -0,0 +1,349 @@ +[[!meta title="Continuous Integration Debian Buster Setup"]] +[[!meta author="Kyllikki"]] +[[!meta date="2018-12-21T12:46:14Z"]] + + +[[!toc]] + +## Debian 9 (Buster) OS install + +### amd64 VDS install from media + +[[Virtual server setup|virtual_host_server]] + +Install minimal system from netinst CD (attached when VDS is created on +phoenix) Config options: + +- In the "role" selection select "ssh server" and "system utilities" only. +- The whole disc default partitioning is fine +- The base user the install insists on creating should be the netsurf user. +- Boot loader in MBR + +Once installed: + +- install sudo package and add netsurf user to sudo group +- edit /etc/default/grub + + GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8" + +as root update grub + + update-grub2 + +### arm64 VDS install + +[[ARM64 virtual server setup|virtual_host_server_arm64]] + +Once installed: + +- create netsurf user +- install sudo package and add netsurf user to sudo group +- edit /etc/inittab comment pty 3 through 6 and uncomment serial T0 +- enable backports + +`echo "deb `[`http://http.debian.net/debian`](http://http.debian.net/debian)` buster-backports main" > /etc/apt/sources.list.d/backports.list` +`apt-get update` + + +## Packaged CI worker install + +Do a base OS install + +The recommended hostname for CI workers is "nsciworker17" this allows us +to clearly identify CI worker nodes. Note historically we have used +"cislave1" which has been objected to by several users. See +[[changing hostname|https://wiki.debian.org/howto/changehostname_changing_hostname]] on how to achive this. + +On master jenkins use "manage nodes" to create new node. Ensure "remote +fs root" is set to /var/lib/jenkins add variable JENKINS\_HOME set to +/var/lib/jenkins + +Note: replace arm64 with architecture name as required (armhf etc.) + +When configuring a scaleway instance the "Launch method" should be set +to "start and stop this node on-demand" with the "Start script" set to +`perl /usr/bin/pscw.pl start netsurf-ciworker17` and "Stop script" set +to `perl /usr/bin/pscw.pl stop netsurf-ciworker17`. The Availability +set to "Take this slave on-line when in demand and off-line when idle" +and "in demand delay" set to 0 and "idle delay" set to 15. This will +mean the node is turned off and not charged for when idle. + +As superuser: + +create jenkins user + + adduser --system --group --home /var/lib/jenkins/ --disabled-login jenkins + +Install https transport + + apt-get install apt-transport-https + +Add CI server repo to slave apt sources + + echo "deb https://ci.netsurf-browser.org/debian/ buster/amd64/" >> /etc/apt/sources.list.d/netsurf-browser.list + +update repos + + apt-get update + +install ci worker package. accept the large package list and the + unsigned package install for ns-ci-slave + + apt-get install ns-ci-worker + +edit /etc/default/ns-ci-worker to set the correct url and secret parameters + +ensure /opt is setup correctly to allow toolchains to be built on the node + + mkdir -p /opt/netsurf + chown jenkins:jenkins /opt/netsurf + +become jenkins user + + su -s/bin/bash - jenkins + +create ssh keypair (accept defaults - no password) + + ssh-keygen -t rsa -C &quot;netsurf(a)nsciworker12.netsurf-browser.org&quot; + +copy .ssh/id\_rsa.pub from worker to jenkins master node and append + to /home/netsurf/.ssh/authorized\_keys + + scp /home/jenkins/.ssh/id_rsa.pub netsurf@ci.netsurf-browser.org:nsciworker12_id_rsa.pub + +exit jenkins user shell + +start CI worker daemon + + /etc/init.d/ns-ci-worker start + + + +## Pbuilder setup + +This allows a worker to build Debian packages. The worker should be +installed as a normal CI worker node and then: + +as superuser on node: + + # apt-get install pbuilder + # addgroup pbuilder + # addgroup jenkins pbuilder + +create /etc/sudoers.d/pbuilder + + jenkins ALL = NOPASSWD:/usr/sbin/pbuilder + +`visudo` and alter Defaults + + Defaults env_reset,env_keep="DIST ARCH" + +replace `/etc/pbuilderrc` + + # this is your configuration file for pbuilder. + # the file in /usr/share/pbuilder/pbuilderrc is the default template. + # /etc/pbuilderrc is the one meant for overwriting defaults in + # the default template + # + # read pbuilderrc.5 document for notes on specific options. + + # List of Debian suites. + DEBIAN_SUITES=("sid", "buster", "jessie", "wheezy", "squeeze") + + # List of Ubuntu suites. + UBUNTU_SUITES=("vivid" "utopic" "trusty" "saucy" "raring" "quantal" "precise" "oneiric" "natty" "lucid" "hardy") + + # Mirrors to use. Update these to your preferred mirror. + DEBIAN_MIRROR="ftp.uk.debian.org" + UBUNTU_MIRROR="mirrors.kernel.org" + + # set a default distribution if none is used. + : ${DIST:="$(lsb_release --short --codename)"} + + # set the architecture to the host architecture if none set. + : ${ARCH:="$(dpkg --print-architecture)"} + + NAME="$DIST" + if [ -n "${ARCH}" ]; then + NAME="$NAME-$ARCH" + DEBOOTSTRAPOPTS=("--arch" "$ARCH" "${DEBOOTSTRAPOPTS[@]}") + fi + BASETGZ="/var/cache/pbuilder/$NAME-base.tgz" + DISTRIBUTION="$DIST" + BUILDRESULT="/var/cache/pbuilder/$NAME/result/" + APTCACHE="/var/cache/pbuilder/$NAME/aptcache/" + BUILDPLACE="/var/cache/pbuilder/build/" + + if $(echo ${DEBIAN_SUITES[@]} | grep -q $DIST); then + # Debian configuration + MIRRORSITE="http://$DEBIAN_MIRROR/debian/" + COMPONENTS="main contrib non-free" + if $(echo "$STABLE_CODENAME stable" | grep -q $DIST); then + OTHERMIRROR="$OTHERMIRROR | deb $MIRRORSITE $STABLE_BACKPORTS_SUITE $COMPONENTS" + fi + elif $(echo ${UBUNTU_SUITES[@]} | grep -q $DIST); then + # Ubuntu configuration + MIRRORSITE="http://$UBUNTU_MIRROR/ubuntu/" + COMPONENTS="main restricted universe multiverse" + else + echo "Unknown distribution: $DIST" + exit 1 + fi + +The architecture is assumed to be the native one from + + dpkg --print-architecture + +This can be set by passing ARCH to pbuilder (useful for i386 maybe?) + +for each distribution this node will build for: + +create pbuilder result directory and set ownership permissions + + # mkdir -p /var/cache/pbuilder/buster-armhf/result + # chown root:pbuilder /var/cache/pbuilder/buster-armhf/result + # chmod g+w /var/cache/pbuilder/buster-armhf/result + +become jenkins user + + su -s/bin/bash - jenkins + +create pbuilder base for distribution + + sudo DIST=buster pbuilder create + +if desired additional packages and config can be made to the base with + +`sudo DIST=buster pbuilder login --save-after-login` + +## distcc worker node + +Do a basic OS install but \*not\* a CI worker setup. + +A recommended hostname for distcc worker is something like "cicpu0" this +allows us to use systems as processing node for other purposes than just +distcc in future. See debians +[[changing hostname|https://wiki.debian.org/howto/changehostname_changing_hostname]] on how to achive this. + +The Netsurf repository has necessary updated packages in it and can be +accessed by doing the following: + +Add CI server repo to worker apt sources + + echo "deb https://ci.netsurf-browser.org/builds/debian/ buster/amd64/" >> /etc/apt/sources.list + +update repos + + apt-get update + +use apt to install these packages: + + build-essential + gcc + clang + distcc + +edit /etc/default/distcc + + STARTDISTCC="true" + ALLOWEDNETS="192.168.211.0/24" + LISTEN="0.0.0.0" + JOBS="8" + +start the service + + service distcc start + +ensure the client has hosts set to use the new worker + + + +## Manual CI worker install + +Caution these instructions may not be up to date. + +### required packages + +The Netsurf repository has necessary updated packages in it and can be +accessed by doing the following: + +Add CI server repo to slave apt sources + +`echo "deb `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)` buster/amd64/" >> /etc/apt/sources.list` + +update repos + + apt-get update + +use apt to install these packages: + +`openjdk-7-jre-headless ` +`screen ` +`build-essential` +`ccache` +`clang` +`git` +`pkg-config` +`check` +`doxygen` +`libjson0-dev (from our repo - needs bugfixes `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)`)` +`libexpat1-dev` +`libxml-perl` +`libxml-xpath-perl` +`lcov` +`gcovr (from our repo)` +`gperf` +`flex` +`bison` +`libpng-dev` +`libjpeg-dev` +`libmozjs185-dev` +`libglib2.0-dev` +`libcurl4-openssl-dev` +`liblcms1-dev` +`libxml2-dev` +`librsvg2-dev` +`libmng-dev` +`libgtk2.0-dev` +`libmozjs-dev` + +### config + +- on master jenkins use "manage nodes" to create new node. Ensure + "remote fs root" is set to /home/netsurf/jenkins +- create netsurf user +- as netsurf user: + - wget <http://ci.netsurf-browser.org/jenkins/jnlpJars/slave.jar> + - run screen + - create jenkins-slave.sh + + #!/bin/bash + + java -Djava.awt.headless=true -jar slave.jar -jnlpUrl https://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp -secret 0123456789abcdef01234567890abcdef + + - run jenkins-slave.sh + - create new screen tab + - create ssh keypair (accept defaults - no password) + + ssh-keygen -t rsa -C &quot;netsurf(a)cislave0.netsurf-browser.org&quot; + +- - copy .ssh/id\_rsa.pub from slave to jenkins master node and + append to /home/netsurf/.ssh/authorized\_keys + + scp ci.netsurf-browser.org:.ssh/id_rsa.pub .id_rsa.pub + cat id_rsa.pub >> .ssh/authorized_keys + +- - copy .ssh/id\_rsa.pub from master node to slave and append to + /home/netsurf/.ssh/authorized\_keys + - create reverse-ssh.sh (change tunnel port number!) + + #!/bin/sh + + ssh -R 22224:localhost:22 netsurf(a)ci.netsurf-browser.org 'bash -c "while true; do echo .; sleep 60; done" + +- - run reverse-ssh.sh + - on the master create a shell script to use the ssh tunnel + connection, thus firewalls etc are moot as long as the slave can + connect to the master + + ssh netsurf@localhost -p 22223 diff --git a/continuous_integration/debian_stretch_setup.mdwn b/continuous_integration/debian_stretch_setup.mdwn index 2333455..557472e 100644 --- a/continuous_integration/debian_stretch_setup.mdwn +++ b/continuous_integration/debian_stretch_setup.mdwn @@ -309,10 +309,10 @@ Add CI server repo to slave apt sources update repos - apt-getupdate + apt-get update install ci worker package. accept the large package list and the - unsigned package install for gcovr and ns-ci-slave + unsigned package install for ns-ci-slave apt-get install ns-ci-worker @@ -361,9 +361,9 @@ create /etc/sudoers.d/pbuilder `visudo` and alter Defaults -`Defaults        env_reset,env_keep="DIST ARCH"` + Defaults env_reset,env_keep="DIST ARCH" -replace /etc/pbuilderrc +replace `/etc/pbuilderrc` # this is your configuration file for pbuilder. # the file in /usr/share/pbuilder/pbuilderrc is the default template. @@ -417,7 +417,7 @@ replace /etc/pbuilderrc The architecture is assumed to be the native one from -`dpkg --print-architecture` + dpkg --print-architecture This can be set by passing ARCH to pbuilder (useful for i386 maybe?) @@ -431,11 +431,11 @@ create pbuilder result directory and set ownership permissions become jenkins user -`su -s/bin/bash - jenkins` + su -s/bin/bash - jenkins create pbuilder base for distribution -`sudo DIST=stretch pbuilder create` + sudo DIST=stretch pbuilder create if desired additional packages and config can be made to the base with @@ -455,18 +455,18 @@ accessed by doing the following: Add CI server repo to worker apt sources -`echo "deb `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)` stretch/amd64/" >> /etc/apt/sources.list` + echo "deb https://ci.netsurf-browser.org/builds/debian/ stretch/amd64/" >> /etc/apt/sources.list update repos -`apt-get update` + apt-get update use apt to install these packages: -`build-essential` -`gcc` -`clang` -`distcc` + build-essential + gcc + clang + distcc edit /etc/default/distcc @@ -477,7 +477,7 @@ edit /etc/default/distcc start the service -`service distcc start` + service distcc start ensure the client has hosts set to use the new worker @@ -492,13 +492,13 @@ Caution these instructions may not be up to date. The Netsurf repository has necessary updated packages in it and can be accessed by doing the following: -- Add CI server repo to slave apt sources +Add CI server repo to slave apt sources `echo "deb `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)` stretch/amd64/" >> /etc/apt/sources.list` -- update repos +update repos -`apt-get update` + apt-get update use apt to install these packages: @@ -542,33 +542,33 @@ use apt to install these packages: - run screen - create jenkins-slave.sh -`#!/bin/bash` - -`java -Djava.awt.headless=true -jar slave.jar -jnlpUrl `[`http://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp`](http://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp)` -secret 0123456789abcdef01234567890abcdef` + #!/bin/bash + + java -Djava.awt.headless=true -jar slave.jar -jnlpUrl https://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp -secret 0123456789abcdef01234567890abcdef -- - run jenkins-slave.sh + - run jenkins-slave.sh - create new screen tab - create ssh keypair (accept defaults - no password) -`ssh-keygen -t rsa -C &quot;netsurf(a)cislave0.netsurf-browser.org&quot;` + ssh-keygen -t rsa -C &quot;netsurf(a)cislave0.netsurf-browser.org&quot; - - copy .ssh/id\_rsa.pub from slave to jenkins master node and append to /home/netsurf/.ssh/authorized\_keys -`scp ci.netsurf-browser.org:.ssh/id_rsa.pub .id_rsa.pub` -`cat id_rsa.pub >> .ssh/authorized_keys` + scp ci.netsurf-browser.org:.ssh/id_rsa.pub .id_rsa.pub + cat id_rsa.pub >> .ssh/authorized_keys - - copy .ssh/id\_rsa.pub from master node to slave and append to /home/netsurf/.ssh/authorized\_keys - create reverse-ssh.sh (change tunnel port number!) -`#!/bin/sh` - -`ssh -R 22224:localhost:22 netsurf(a)ci.netsurf-browser.org 'bash -c "while true; do echo .; sleep 60; done"'` + #!/bin/sh + + ssh -R 22224:localhost:22 netsurf(a)ci.netsurf-browser.org 'bash -c "while true; do echo .; sleep 60; done" - - run reverse-ssh.sh - on the master create a shell script to use the ssh tunnel connection, thus firewalls etc are moot as long as the slave can connect to the master -`ssh netsurf@localhost -p 22223` + ssh netsurf@localhost -p 22223 -- NetSurf Developer Wiki Backing Store