Optimal settings for max-jobs (-j for make)
by Paul Sherwood
TL; DR max-jobs should probably be set to number-of-cores or somewhere
in the 10 - 20 range, whichever is less
For a while we've been parallelising make with -j flags =
int(number-of-cores * 1.5)
Recently I got around to checking this assumption while tuning ybd to
minimise wallclock time on some cloud machines, including
- AWS mx4.10xlarge (40-cores, biggest memory-optimised machine)
- AWS cx4.8xlarge (36 cores, biggest compute-optimised machine)
- Scaleways S1 (cloud-based ARMv7)
AFAICT for the builds I looked at in detail (qtwebkit and gcc) there's
a steep drop-off in performance gain from 10 cores onwards.
For example building Qt Webkit on AWS mx4.10xlarge
-j Seconds
60 435
30 448
25 464
20 465
15 549
10 708
5 1203
1 5247
As a chart:
https://docs.google.com/spreadsheets/d/149jS3VVJ7jA14dSJybeOr4otlEFrzCVJj...
And for gcc
-j mx4.10 cx4.8
60 181 155
30 172 150
20 167 147
10 214 188
5 291 255
3 387 337
2 562 481
1 1214 903
Chart:
https://docs.google.com/spreadsheets/d/149jS3VVJ7jA14dSJybeOr4otlEFrzCVJj...
And for gcc on Scaleway
-j seconds
6 3208
4 3008
3 3617
So in general it looks better to start with number-of-cores for
machines with 8 cores or less, and consider the trade-off for building
more things in parallel vs hogging the whole cpu for a single make on
the bigger machines.
Given ybd can parallelise to do multiple components at once, for the
AWS instances I've tested building ci.morph with a range of settings. so
far it seems that
- instances = 5 max-jobs = 10 is fastest on the mx4.10xlarge
- instances = 4, max-jobs = 9 is fastest on the cx4.8xlarge
- cx is around 10% faster than the mx in general (and 30% cheaper per
hour)
br
Paul
Message 1 of 12950
6 years, 9 months
Research on exporting build definitions from other build tools
by Sam Thursfield
Hi all
I spent the last couple of weeks, along with Pedro, doing some research
on exporting from
This ties in with the idea that it would be great if build/integration
instructions weren't all tied to specific tools. We have part of the
puzzle solved already: the current Baserock reference definitions are
usable by 2 totally separate tools. There's still lots of room for
improvement in the Baserock definitions format ... and one way of
improving it is to try to represent existing instructions from *other*
build/integration efforts, using our supposedly general-purpose
definitions format.
We won't be able to produce working, useful, readable, maintainable
definitions this way, in most cases. But it can still inform our data
model; for example, many build/integration tools track runtime
dependency information in their data model, which suggests that any
generic data model for build/integration will need to track them
This all sounds quite grand, but actually what I have to show you is
some wiki links and a really nasty patch for BitBake :-)
BitBake recipes
---------------
Turns out you can turn BitBake recipes into working shell + Python
scripts, which effectively 'flattens' the data and makes it easier to
represent them generically. There is *lots* of context and many
hardcoded paths, so consider this a 'view' onto the data, rather than
something you could work on and maintain.
It would be interesting to take this further, and try to 'distill' key
information like source code location (SRC_URI), or the configure
commandline. This would be best done by post-processing the exported
shell & Python files, I think ... perhaps tweaking the base .bbclass
files to add special markers would be useful, too.
For example, in base.bbclass, base_do_fetch() could add some commentted
marker that would expand to '#@#@# source-code-location: git://foo/bar"
in the generated .py script, which a post-processing script could then
easily find when scannning all the generated output. And any package
that had overridden the base 'do_fetch' task wouldn't have that marker,
highlighting that it was doing something non-standard to fetch its
source code.
Read more about that here:
http://wiki.baserock.org/projects/importing-from-bitbake/
There is a (rather dry) video of me doing this export:
https://www.youtube.com/watch?v=gW1NKZPomjI
I also sent the patch to bitbake-devel, pointing out that it's just a
proof of concept / ugly hack at this point, to see if readers of that
list could give any feedback or pointers:
http://lists.openembedded.org/pipermail/bitbake-devel/2015-August/006297....
Nix derivations
---------------
Pedro's NixOS research is here:
<http://wiki.baserock.org/projects/importing-from-nixos/>. It seems that
he got as far as discovering there are lots of circular dependencies,
which prevented actually extracting much information so far.
I had a quick look for how NixOS bootstraps itself; it seems to be
defined here:
<https://github.com/NixOS/nixpkgs/blob/6a3b25dbd3e1f3a7257bc34e97d51254392...>.
Looks like a 5 stage bootstrap process, that rebuilds some packages at
each stage, so perhaps the key to resolving the circular deps is to
distinguish between the different instances of these packages?
Debian/Ubuntu packages
----------------------
I was pointed towards Germinate, a tool to deal with Debian 'seeds'. It
returns dependency information, but not much else -- no build instructions.
I also tried 'dgit', which provides a neat wrapper over the Debian
archive. However, it seems to require SSH access to a machine at
debian.org which I don't have, so I guess the tool is only useful for
people who actually maintain packages, at least for now.
In general I think Debian and Ubuntu will be the least fruitful area for
this kind of research, because there is so much diversity in the way
packages are built that it would really timeconsuming to try to extract
information from all the different places it might be.
I'm away for the next two weeks. Please feel free to ask questions about
this research on this mailing list, and I'll answer once I'm back.
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
6 years, 9 months
YBD tempdir locking review
by Richard Maw
I promised to review Paul's locking code for cleaning up his tempdir on start.
Since I couldn't reply in-line on
https://github.com/devcurmudgeon/ybd/compare/clean-tmp-with-locks I've
drafted my response as an e-mail.
I wouldn't normally post a review to the list,
especially since to my knowledge, Paul had no prior experience with file locking,
but he pre-emptively OK'd it, and it may serve useful for others,
since the mistakes made are those a beginner would make,
and highlight how tricky it can be to get it right.
> diff --git a/app.py b/app.py
> index 88d8327..0c6baa8 100644
> --- a/app.py
> +++ b/app.py
> @@ -141,6 +142,22 @@ def load_configs(config_files):
> log('SETUP', 'Configuration from %s:\n\n' % config_file, text)
>
>
> +def cleanup(tmpdir):
> + try:
> + with open(os.path.join(config['base'], 'lock'), 'w') as lockfile:
You don't need a separate file to be the lock.
You can lock the tmpdir directory itself if you use the low level
`os.open(path, os.O_RDONLY)` syscall.
However you'd need to write your own context manager for the directory file descriptor,
as there's no way to use the basic `open()` function to open a directory,
and python2.7 has a bug that prevents you using `os.fdopen()`
on a file descriptor you opened directly which happens to be a directory file descriptor.
Ideally, rather than closing the lock file after cleaning up the tempdir,
you should use flock to convert it to a shared lock,
so you can't end up with another YBD process starting in the mean-time
and causing *your* YBD process to block while that process tries to clean up the tempdir.
> + to_delete = os.listdir(tmpdir)
> + fcntl.flock(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB)
I generally prefer the try-except around the statement I expect to fail, so I
can know what's causing me to have the exception, as anything else in your try:
block could be giving you the IOError.
However since you're using a context manager you're not going to leak
resources, so the only information you lose is why you can't do any cleanup.
> + fcntl.flock(lockfile, fcntl.LOCK_UN)
If you release a lock while you are still using the resource,
you may as well not take the lock.
You should unlock after you have cleaned up your temporary directory.
Because you're unlocking too early,
you can end up with multiple YBD processes thinking that they should clean up the tempdir,
which will cause at least all but one of them to crash,
when they try to remove a directory that doesn't exist.
However, because you fork before cleaning up, you could easily miss this.
> + if os.fork() == 0:
I'm not sure why you are forking here,
if it's to allow you to clean it up in the background,
then I fear it's missing the point,
since because you released the exclusive lock,
your other process will now take the shared lock,
and think that it can safely use the tempdir,
while actually you are still cleaning it up.
> + for dirname in to_delete:
> + if os.path.isdir(dirname):
> + shutil.rmtree(dirname)
> + log('SETUP', 'Cleanup successful for', tmpdir)
> + sys.exit(0)
> + except IOError:
> + log('SETUP', 'No cleanup for', tmpdir)
> +
> +
> @contextlib.contextmanager
> def chdir(dirname=None):
> currentdir = os.getcwd()
> diff --git a/ybd.py b/ybd.py
> index a3e632e..c7d5b8c 100755
> --- a/ybd.py
> +++ b/ybd.py
> @@ -19,6 +19,7 @@
>
> import os
> import sys
> +import fcntl
> import app
> from assembly import assemble
> from deployment import deploy
> @@ -31,6 +32,12 @@ import sandboxlib
> print('')
> with app.timer('TOTAL'):
> app.setup(sys.argv)
> +
> + app.cleanup(app.config['tmp'])
> +
> + lockfile = open(os.path.join(app.config['base'], 'lock'), 'r')
> + fcntl.flock(lockfile, fcntl.LOCK_SH | fcntl.LOCK_NB)
> +
A shared lock can't be taken while there's an exclusive lock being held,
so if you modify your cleanup code to appropriately keep the exclusive lock,
you will see YBD commands fail if they are started while the tempdir is
being cleaned up.
If you trust that the other YBD process will either release the lock or
terminate, then you can do a blocking lock here by removing the
`|fcntl.LOCK_NB`.
If you don't, then your options are:
1. Have a retry loop attempting to do a non-blocking lock.
This has the unfortunate property that you could miss the window where the
lock isn't held exclusively with a bit of bad luck.
2. Spawn a thread to do the blocking flock,
and sleep in your own thread until the timeout period,
so you can interrupt the lock on timeout.
3. Use `signal.alarm(timeout)` to have the kernel be the one doing (2.),
at which point the `flock()` call will be interrupted.
Though to make this work you need to in the very least register the
SIG_IGN signal handler for SIGALRM.
4. Shell out to /usr/bin/flock, which does the timeout, and has a mode to pass it an existing file.
You'd call it with something like:
r = subprocess.call(['flock', '--shared', '--wait', '--timeout', '10', str(lockfile.fileno())])
if r != 0:
log("Failed to obtain shared lock on tempdir, check if any running YBD processes are stuck")
sys.exit(1)
If you only expect YBD to be called interactively, then I'd normally suggest
just putting a message up before attempting to take the lock, so that the user
can decide that it must be stuck and kill it.
However there's some projects around calling YBD in batch jobs,
so shelling out to flock may be the simplest answer to having a blocking lock with timeout.
> target = os.path.join(app.config['defdir'], app.config['target'])
> app.log('TARGET', 'Target is %s' % target, app.config['arch'])
> with app.timer('DEFINITIONS', 'parsing %s' % app.config['def-version']):
In summary:
0. Locking is *difficult* even when you know all the caveats, review is essential.
1. It's possible to use the directory itself as the lock file,
but I don't blame you for not trying to.
2. Hold the exclusive lock while cleaning up the tempdir
3. Convert the lock to a shared lock, rather than closing, re-opening and re-locking.
4. Don't fork before cleaning up the tempdir, you need to wait to finish cleaning it up anyway.
5. Taking the shared lock is best done while blocking with a timeout,
surprisingly one of the least awful ways to do this is to shell out to /usr/bin/flock.
6 years, 9 months
Generic cache server: the revenge
by Sam Thursfield
Hi
I was discussing what the quickest route would be to get
cache.baserock.org to accept artifacts submissions from the wider world,
and not just from the private network that it's on. With specifically an
atomic 'push' method that would work well for the YBD 'controllerless'
distributed build approach that Paul has been investigating.
For historical reasons, cache.baserock.org is currently a Trove instance
that isn't really a Trove. But in order to get something running as
quickly but neatly as possible, with new features, I think this would be
the approach:
- Add a '/2.0/submit' method to morph-cache-server that accepts artifact
as HTTP PUT data.
- Make this method put the artifact in /home/cache/artifacts in a
subdirectory named for the artifact, which is a neat way of making the
PUT be atomic: subsequent pushes will fail because they can't overwrite
the directory
- Teach the /files method to look inside these directories for files as
well as in /home/cache/artifacts itself
- Pick a base OS to deploy this on (debian, fedora or a baserock 'build'
system would probably all work)
- Write an Ansible script to install morph-cache-server from Git, and
HTTPS proxy tool (haproxy), configure morph-cache-server and the proxy
to run at boot. See trove-setup.git for how to do most of this (but it
can be a lot simpler than that).
- Set up the proxy to authenticate access to the /2.0/submit method. I'm
no expert on how to do this, but I'd suggest requiring client-side SSL
certificates. This might help:
https://raymii.org/s/tutorials/haproxy_client_side_ssl_certificates.html
These instructions are incomplete, it's just an idea of an approach for
solving this that would hopefully be quick, and make the infrastructure
team (or at least, me) very happy.
I would be happy to replace the current cache.baserock.org with
something like the thing described above. We already have some machines
at baserock.org which are defined in infrastructure.git simply as
Ansible scripts to run on top of a given distribution. For example..
..the mail relay:
http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/infrastructure...
..the frontend haproxy system:
http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/infrastructure...
Lauren already did some work on morph-cache-server recently, I don't
know where it is pushed to, but it would probably be worth starting from
there.
Hope this helps anyone who is interested in doing this :-)
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
6 years, 9 months