(Sorry about taking so long to answer: I was travelling for work.)
You're right, this scales badly to long build logs, because of
large number of HTTP requests required. (The size of the logs is so
much less of an issue I'm going to ignore that for now.)
For now, I think I'm OK with that, if it's fast enough for
not-very-long build logs. From my personal Ick instance, the longest
log files are about 14000 lines.
I suspect this depends upon what you're building; for complicated
reasons at work I'm creating images which build nginx, php, curl,
and similar things from source. The build output from such jobs
is pretty volumous. At home I tend to build linux-kernels which
also have a good few thousand lines of output.
This is not excellent. Is it tolerable? Not very tolerable for
interactive use. I'm not much worried, for now, about the time it
takes to create the log file snippets, that's going to be drowned by
the actual build. The time to get the list of snippets is fast enough,
I think, even in the current prototype written in Python without
That's probably fair.
The full log assembly time is bad. As you say, it's because
lot of HTTP requests. It might be tolerable for a short while, while
we improve things, just to get the Ick controller to use Muck instead
of having local state, but let's discuss how we can improve it.
That's probably a good approach. When it comes to builds in my
current system I have to say that I tend to scroll straight to
the bottom to look for an "All OK", or "ERROR/FAIL" message.
The stuff in the middle I don't care about so much very often.
I'd not suggest that having a ring-buffer was a good idea, but if
it were possible to keep track of the last update that might be
a compromise worth making.
I don't like the idea of keeping the log file locally, either on
worker-manager or the controller, as it means that one can't do the
equivalent of "tail -f" while the build is running. Updating the
controller's view of the log while the build is running, without
lagging much from actual output, is a thing I'd really like to keep.
I suspect when everything is done the build-log belongs with the
output artifacts, in the object-store. That might be personal
bias at play of course.
* new step: if there are more than M short snippet objects for a
build, the controller would construct a combined snippet object, and
delete the short snippet objects
* new step: at the end of the build, the controller would comine all
short snippet objects
That seems like a good idea.