Distributed CI
by Lars Wirzenius
We've had a short disucssion on IRC about maybe adding support to Gitano (a
git server) and ick for ActivityPub instead of having explicit triggers
configured in Gitano to trigger builds in ick. I thought I'd expand on that
a little bit.
The thing I'm grasping at is that currently, each git server instance needs
to notify each CI instance separetely, creating a fairly tight coupling.
Alternatively, each CI instance needs to poll each git server, which is
unworkable at scale.
So currently, if there are three users, one using github.com, one using
gitlab.com, and one using the Debian gitlab instance (salsa), and each user
is developing their own software, then each user needs to configure their
own git server to notify their own CI server. So far, so good.
However, if each of the three also wants to build and deploy each other's
software, so that if user A makes a change, it gets put through CI by A's,
B', and C's CI, and then deployed to A's, B's, and C's production server,
and likewise for B's and C's software, then each of the three needs to add
a trigger hook on each of th three git repositories. That's three
repositories, three CI servers, and a total of nine trigger hooks.
That's still manageable, but there are problems. Not all git servers allow
random strangers to add trigger hooks on each repository. Also, three
repositories and three CI servers is small potatoes. Imagine having fifty
thousand repositories! Debian has about that many binary packages.
Also imagine that those fifty thousand repositories are used by a million
users. Not all of them by all users, but many repositories by each of the
million. Having millions of hooks does not scale. Having that much polling
also doesn't scale.
So here's the solution I'm thinkin of: instead of having the git server
notify each CI server, have the git server send a message to an ActivityPub
server, and have each CI server listen on such messages on their own
ActivityPub server. The AP servers federate (send messages to each other),
so each git server and each CI server only needs to post to and listen on
one AP server.
With this design, many-to-lots communication becomes feasible.
A makes a change, and pushes to their git server. The git server posts a
message to its own AP server, saying "this branch on this repo on this git
server has changed, new commit it CAFEBEEF." The AP server knows who's
"following" A's repository, and so the message flows out to the AP server
network to every AP server where a CI server is interested in that
repository. Each of those CI servers can then trigger a build and deploy of
A's software.
I don't actually know much about ActivityPub yet. It's the protocol behind
the Mastodon system (https://joinmastodon.org/), and it works fine for a
Twitter clone. I don't know yet if it would work for what I'm talking about
above, but I think so.
Of course, doing distributed CI at this scale might not be something that
anyone actually cares about. We'll see. But even at a small scale, I'd like
to break the necessity of the git server having to know about each CI
server that is interested in each of its repositories.
Also, this needs thinking for non-public repositories.
4 years, 8 months
Code of conduct for the ick project
by Lars Wirzenius
I am proposing to adopt the Contributor Covenant Code of Conduct for the
ick project. I've made the following changes:
* added a copy to ick.liw.fi
* added a copy to the ick source tree
* added an email alias ick-conduct(a)liw.fi
The alias points currently only at myself. Once ick gains more
contributors, I'd be happy to add someone else to the "conduct team". What
this means is that the people responsible for handling any misbehavior can
be reached via that address.
I've asked around and it seems the Covenant is held in high regard.
However, if anyone whom this email reaches has objections to it, please
tell me. If you persuade me, changing the code of conduct for ick is still
easy.
4 years, 9 months
Ick and checking out source from git
by Lars Wirzenius
(Long mail, question at end.)
I set up a demo instance of git and invited Daniel to try it. (If anyone
else wants to have a go, let me know privately.)
One of the issues Daniel found is that ick does not yet support well is
building a source tree from a stack of git repositories. This is something
Daniel does to keep his upstream code and its Debian packaging separate.
Basically, the upstream code lives at:
git://git.example.com/foo.git
The Debian packaging lives at:
git://git.example.com/foo-debian.git
These are meant to be stacked:
git clone git://git.example.com/foo.git
cd foo
git clone git://git.example.com/foo-debian.git
This actuall does work with ick right now, but only for public repositories
that can be accessed without authentication.
Ick has a "git" action that can be used when using the ssh protocol, which
does require authentication. This uses the worker's ssh key, but only
supports one URL:
action: git
where: host
This uses the project parmaeters git_url, git_dir, and git_ref:
parameters:
git_url: ssh://git@git.example.com/foo.git
git_ref: master
git_dir: foo
The above parametgers would tell the git action to clone the foo.git
repository to the foo directory in the workspace (checking out the master
branch). This works, but only for one repository. Daniel needs at least
two (or would need, if his repositories weren't public).
I'd like to support this for any number of repositories. I suggest the
following:
parameters:
git:
- url: ssh://git@git.example.com/foo.git
ref: master
dir: foo
- url: ssh://git@git.example.com/foo-debian.git
ref: master
dir: foo/debian
The above would do the same things as the two git clones above.
What do you all think? Especially you, Daniel? Would that be an acceptable
design (at least for now)?
4 years, 9 months
Ick installaion: feedback?
by Lars Wirzenius
I've written up some ick installation instructions:
https://ick.liw.fi/install/
I'd welcome any feedback on them. It would be very valuable to me if you
could try to install ick following these instructions.
(If you've ever read the INSTALL file in the source tree, it's gone now,
and fully replaced with a new way to install ick.)
4 years, 9 months