1:42 a.m.
TL;DR: The Ick controller can be changed in a straightforward manner
to use Muck for persistent data storage, except for log files. It
doesn't seem worthwhile to support an in-place conversion. Instead,
the few people affected can re-create their projects and pipelines on
a new, fresh demo Ick instance, which will replace the current one.
TS;RF (too short; rambling follows):
Currently, the Ick controller stores various resources directly in its
local filesystem. I wish to change the controller to use Muck instead.
The main motivation for this is to have better access control: users
of the same controller shouldn't see or be able to change projects or
pipelines for other users.
Muck is a JSON data store with access control. Part of the access
control is that every JSON object ("resource") is owned by a specific
user. Every user can only access their own objects, for now (although
this will become more flexible later).
Where the controller currently creates, say, a resource for the
project foo, by storing it as YAML in the file
"/var/lib/ick/state/projects/foo", I plan to change that so that it
creates a JSON resource in Muck like this:
{
"_type": "project",
"_name": "foo",
... # all other fields as in the current YAML file
}
Muck invents a unique identifier for each object, and guarantees no
other object has that identifier. The controller will not use this,
and will instead use a search on the "_type" and "_name" fields to
find the right object. This is so that users may refer to projects and
pipelines using more humane names: "ick.liw.fi" instead of
"48053f4f-71d9-42a1-b3ca-8574cbb788aa" for example.
Muck allows arbitrary JSON objects to be stored. For the Ick
controller, the following approach seems like a reasonable first
attempt:
* Each object will have a "_type" field, which specifies the type of
the object: project, pipeline, build, log, or worker. This is needed
so that one can search for a "project foo", as opposed to "pipeline
foo".
* Each object will have a user-assigned "_name" field, which the
controller makes sure is unique and prefixed with the object owner's
username.
Muck does not have transactions that span multiple HTTP requests.
The controller will do its best to ensure a name is unique, but it
can't guarantee that. However, if it notices a name clash later, it
will treat that as an error. For example, if there are two projects
named "liw/ick.liw.fi", the controller will refuse to trigger
either. (This is a limitation in Muck and will be fixed later,
possibly by teaching Muck about user-defined names for resources,
and having it make sure they're unique. But that will have to wait
for a later version of Muck.)
* Depending on the type of the object, it may contain other fields as
well, see below.
* The controller creates objects in Muck based on API calls to the
controller, and passes on the access token it gets. Muck uses the
access token's "sub" field to assign ownership of the object; some
access tokens do not have such a field, and can thus not be
accressed by any user; this will be the case for workers that
register themselves.
The following object types will be supported initially:
* project - projects defined by the user
* pipeline - pipelines defined by the user
* build - a build that's been triggered, is running, or is finished
* log - a build log
* worker - a worker
The contents of these object types are as they are in the controller
now. Switching to Muck does not change that, except for logs, which
need special handling to avoid very bad performance.
Muck stores all incoming resources to a "change log". A build log may
get thousands of updates: each line of output may become an update.
Doing a new update of the log object would result in thousands of
nearly identical copies in the change log, which is quite wasteful:
each new version of the log resource would be identical to the
previous one, except with a line of new text. When a build produces a
thousand lines of output, Muck would store a thousand copies of the
log object in its change log.
To avoid this waste, the controller will be changed to store the build
logs as follows:
* The worker-manager will send each build log snippet as a separate
update, as before, but it will also add a sequence number to the
updates.
* The controller will create a separate log object for reach update in
Muck. This object will contain only the new snippet. The new log
object will have the log snippet, plus a reference to the build for
which it is part of, and the sequence number.
* The controller will reconstruct the whole build log when it's
requested for a complete log, by fetching all log objects that refer
to the specified build, and catenating the log output snippets in
order of the sequence number.
I think this will work, but I haven't written any code yet. Comments?
I plan on starting work on this next week, when I get back from gallivanting
in SF.
I don't plan on converting existing projects and pipeline on the demo Ick
instance. That seems like work that would be useful, but it's also work
that's maybe not worthwhile yet, given the demo Ick instance only has a few
users. I'm lazy, sorry.
9:07 a.m.
* The controller will reconstruct the whole build log when it's
requested for a complete log, by fetching all log objects that refer
to the specified build, and catenating the log output snippets in
order of the sequence number.
Assume you're building a kernel, which might spit out 400,000 lines
of build-log. Assume each new update, due to buffering, or sizing,
is 10 lines.
To get the whole build-log you're going to need to:
* Get the first piece.
* Get the next child.
* Get the next child.
* Get the next child, 39,997 times more.
That seems like it will scale terrible. The only potential win I can
see here is imagining that you might want to view the output via a
brower and you'll probably only care about the LAST 100 lines, not the
FIRST. (i.e. The bit which will typically contain "built blah", or
"build failed".)
I'd be tempted to write the log locally, then upload as one massive
object when complete. Sure it might be big, but it's more efficient
to stream a 1Mb HTTP transfer than to keep repeating HTTP-calls to
get the next bit.
(For a web-case you can use AJAX to stream on-scroll. But for email
output you'll want the whole thing to send..)
Steve
--
https://www.steve.org.uk/
6:40 p.m.
(Sorry about taking so long to answer: I was travelling for work.)
On Sat, Feb 09, 2019 at 05:07:55AM +0000, Steve Kemp wrote:
Assume you're building a kernel, which might spit out 400,000
lines
of build-log. Assume each new update, due to buffering, or sizing,
is 10 lines.
To get the whole build-log you're going to need to:
* Get the first piece.
* Get the next child.
* Get the next child.
* Get the next child, 39,997 times more.
That seems like it will scale terrible. The only potential win I can
see here is imagining that you might want to view the output via a
brower and you'll probably only care about the LAST 100 lines, not the
FIRST. (i.e. The bit which will typically contain "built blah", or
"build failed".)
You're right, this scales badly to long build logs, because of the
large number of HTTP requests required. (The size of the logs is so
much less of an issue I'm going to ignore that for now.)
For now, I think I'm OK with that, if it's fast enough for
not-very-long build logs. From my personal Ick instance, the longest
log files are about 14000 lines. I added a little benchmark script to
the muck-poc repository to test this: benchmark-log. I generates N
snippets, stores them in Muck, then searches for the snippets, and
retrieves each, and reconstructs the whole log by catenating the
snippets in order (i.e., as outlined in my previous mail). It also
reports how long each stage takes.
Note that muck-poc and the benchmark are both running sequentially.
Running with N=14000, on my laptop, the script outputs:
creating snippets
getting list of snippets
reconstructing full log
OK
68 creation time
0 list snippets
67 assemble log
This is not excellent. Is it tolerable? Not very tolerable for
interactive use. I'm not much worried, for now, about the time it
takes to create the log file snippets, that's going to be drowned by
the actual build. The time to get the list of snippets is fast enough,
I think, even in the current prototype written in Python without
indexes.
The full log assembly time is bad. As you say, it's because there's a
lot of HTTP requests. It might be tolerable for a short while, while
we improve things, just to get the Ick controller to use Muck instead
of having local state, but let's discuss how we can improve it.
I don't like the idea of keeping the log file locally, either on the
worker-manager or the controller, as it means that one can't do the
equivalent of "tail -f" while the build is running. Updating the
controller's view of the log while the build is running, without
lagging much from actual output, is a thing I'd really like to keep.
I also don't want to rely on websockets or similar, for now. The Muck
archtecture doesn't currently suit that, and I don't want to change
that, at least for now.
The controller could coalesce small snippets while the build is
running. It could notice that there's a lot of small snippets, and
combine them into a bigger snippet, deleting the small ones. This
would radically reduce the number of snippets. If, say, the controller
combined 1000 small snippets to one larger one, that would reduce the
number of snippets so much reconstructing a complete log is tolerably
fast - order of zero seconds at the moment on my laptop. It'd mean
some duplication of log data in the Muck changelog, but I think that'd
also be acceptable. There's a fair bit of back and forth of log data,
of course.
Thus, the controller would do this:
* incoming short snippets would arrive from the worker-manager as
outlined in my previous email
* the controller would create a short snippet object in Muck for each
incoming snippet
* new step: if there are more than M short snippet objects for a
build, the controller would construct a combined snippet object, and
delete the short snippet objects
* new step: at the end of the build, the controller would comine all
short snippet objects
* new: when returning the whole log, the controller would first
catenate all the combined snippets, and then the short snippets
Does this seem reasonable? I think it'd have tolerably good
performance.
Another approach, some day in the future, might be to have a "log
server" instead of having the controller do that, but that is a whole
new component, and I don't want to go there yet.
--
I want to build worthwhile things that might last. --joeyh
7:54 p.m.
(Sorry about taking so long to answer: I was travelling for work.)
No worries.
You're right, this scales badly to long build logs, because of
the
large number of HTTP requests required. (The size of the logs is so
much less of an issue I'm going to ignore that for now.)
For now, I think I'm OK with that, if it's fast enough for
not-very-long build logs. From my personal Ick instance, the longest
log files are about 14000 lines.
I suspect this depends upon what you're building; for complicated
reasons at work I'm creating images which build nginx, php, curl,
and similar things from source. The build output from such jobs
is pretty volumous. At home I tend to build linux-kernels which
also have a good few thousand lines of output.
This is not excellent. Is it tolerable? Not very tolerable for
interactive use. I'm not much worried, for now, about the time it
takes to create the log file snippets, that's going to be drowned by
the actual build. The time to get the list of snippets is fast enough,
I think, even in the current prototype written in Python without
indexes.
That's probably fair.
The full log assembly time is bad. As you say, it's because
there's a
lot of HTTP requests. It might be tolerable for a short while, while
we improve things, just to get the Ick controller to use Muck instead
of having local state, but let's discuss how we can improve it.
That's probably a good approach. When it comes to builds in my
current system I have to say that I tend to scroll straight to
the bottom to look for an "All OK", or "ERROR/FAIL" message.
The stuff in the middle I don't care about so much very often.
I'd not suggest that having a ring-buffer was a good idea, but if
it were possible to keep track of the last update that might be
a compromise worth making.
I don't like the idea of keeping the log file locally, either on
the
worker-manager or the controller, as it means that one can't do the
equivalent of "tail -f" while the build is running. Updating the
controller's view of the log while the build is running, without
lagging much from actual output, is a thing I'd really like to keep.
Sure.
I suspect when everything is done the build-log belongs with the
output artifacts, in the object-store. That might be personal
bias at play of course.
* new step: if there are more than M short snippet objects for a
build, the controller would construct a combined snippet object, and
delete the short snippet objects
* new step: at the end of the build, the controller would comine all
short snippet objects
That seems like a good idea.
Steve
--
https://www.steve.org.uk/
9:59 p.m.
On Wed, Feb 20, 2019 at 03:54:23PM +0000, Steve Kemp wrote:
> (Sorry about taking so long to answer: I was travelling for
work.)
No worries.
> You're right, this scales badly to long build logs, because of the
> large number of HTTP requests required. (The size of the logs is so
> much less of an issue I'm going to ignore that for now.)
> For now, I think I'm OK with that, if it's fast enough for
> not-very-long build logs. From my personal Ick instance, the longest
> log files are about 14000 lines.
I suspect this depends upon what you're building; for complicated
reasons at work I'm creating images which build nginx, php, curl,
and similar things from source. The build output from such jobs
is pretty volumous. At home I tend to build linux-kernels which
also have a good few thousand lines of output.
Yup. My personal projects are fairly small and simple. I do want to
build Ick to be able to handle quite large projects, though. The log
file handling will be part of that.
I think we have a sufficiently good design for now, and can progress
with making the Ick controller not have any local, persistent state,
and to use Muck for persistence instead.
The stuff in the middle I don't care about so much very often.
I'd not suggest that having a ring-buffer was a good idea, but if
it were possible to keep track of the last update that might be
a compromise worth making.
Getting the tail of the current log, say, the last 100 lines, should
be pretty fast.
I suspect when everything is done the build-log belongs with the
output artifacts, in the object-store. That might be personal
bias at play of course.
I suspect you're right. If nothing else, the log from one build is
quite similar to the log of the next build, most of the time, and
having a way to store them that takes care of the similarities to
avoid using so much space may well make it worthwhile to have a
special "log service" of some sort. We'll see. That's a "maybe in
the
future" idea, not something that's worth doing now, I think.
--
I want to build worthwhile things that might last. --joeyh
1497
days inactive
1513
days old
4 comments
2 participants
participants (2)
-
Lars Wirzenius -
Steve Kemp