Discussion of Baserock entity names on IRC
by Richard Dale
There was a lengthy discussion on the #baserock IRC channel of what to
call the various entities in the Baserock development system. Here is
the transcript:
[15:15:05] <paulsherwood> in another place i've been discussing names....
[15:15:16] <paulsherwood> please don't all sigh or cry bikeshed :)
[15:15:39] <paulsherwood> it's been suggested that 'layer' is a much
more useful word than stratum
[15:15:42] <-- paulw (~paulwater(a)82-70-136-246.dsl.in-addr.zen.co.uk)
has quit (Quit: Leaving)
[15:15:54] <paulsherwood> we shied away from this to avoid confusion
with bitbake layers...
[15:16:14] <paulsherwood> but maybe that's wrong, given there's a clear
implication of one thing on top of another
[15:16:14] <rdale_ct> how do they differ from baserock strata then?
[15:16:19] <paulsherwood> any strong opinions?
[15:16:56] <paulsherwood> well, i'm not an expert but bitbake layers are
not described in yaml iiuc, and can have more logic
[15:17:34] <paulsherwood> also 'deployment' has been suggested instead
of 'cluster'
[15:18:00] <paulsherwood> (nb this would initially only be changes in
ybd internal representation, to see if it improves legibility)
[15:21:14] <ssam2> deployment is clearly better than cluster to me
[15:21:29] <paulsherwood> i agree :)
[15:21:31] <ssam2> as for 'layer'... i think it's just as inaccurate as
'stratum'
[15:21:44] <ssam2> 'group' would be accurate... but too generic
[15:22:52] <franred> a stratum is technically a layer (so it won't
change anything) - no strong opinion on the change. I assume that
cluster was referencing that you can deploy multiple systems at the same
time, so once again, Im not against it.
[15:27:17] <paulsherwood> i wanted 'group' for stratum too
[15:27:58] <paulsherwood> level, or tier, maybe :)
[15:28:46] <rdale_ct> a 'level of components' doesn't make sense, a
'tier of components' is better, but i still prefer 'layer'
[15:30:00] <paulsherwood> i still prefer 'group' but fine :)
[15:31:37] <richard_maw> 'layer' works better to describe the mechanism,
since you can only compose a system by layering layers on top of each other
[15:32:04] <richard_maw> if you start allowing dependencies directly on
chunks, then 'group' becomes more appropriate, since then it's just a
convenience for grouping chunks together
[15:32:44] <paulsherwood> i was assuming we would move to dependencies
on chunks at some point
[15:33:26] <paulsherwood> (while still supporting depedency on
strata/group/whatever)
[15:33:37] <richard_maw> then I'd go for group
[15:34:09] <paulsherwood> rdale_ct: i think we have +3 for group here :)
[15:34:44] <pedroalvarez> are we moving to dependencies on chunks then ?? :D
[15:36:06] <paulsherwood> pedroalvarez: afaik ybd would already support
it (or that was my intention) but i've not tested it since early forays
into the data structure while coding the definitions parsing in 2014
[15:37:22] <rdale_ct> 'group' has a linux meaning and using 'layer'
avoids confusion with user/groups
[15:38:40] <paulsherwood> ssam2, richard_maw ^^ are you still preferring
group ?
[15:39:30] <paulsherwood> i know this is bikeshedding, but our names
have been a problem for a long time, would like this to be a clear step
in the right direction if possible
[15:39:57] <richard_maw> yes, group is generic enough that it's obvious
that it requires extra context, in filesystems this is user groups, in
our definitions they would be chunk/package/whatever groups
[15:40:20] <ratmice> the other day was reading:
http://web.mit.edu/Saltzer/www/publications/nbo/nbo.pdf
[15:40:46] <ratmice> a bit surprised by how many of the naming related
terms in that paper are still in use today
[15:41:31] -*- richard_maw throws "atom" into the mix for replacing
chunk names
[15:41:50] <rjek> "package"
[15:42:08] <paulsherwood> ratmice: that looks like a good read for the
train :)
[15:43:23] <paulsherwood> package is already loaded, even moreso than
component?
[15:44:09] -*- rjek thinks the meaning is clearer to newbies
[15:44:24] <paulsherwood> you're probably right
[15:45:20] <richard_maw> package does conflate input source with output
artifact, hence the clumsy term "source package"
[15:45:30] <richard_maw> won't be a problem for us, since source
packages aren't artifacts
[15:45:31] <rdale_ct> chunks can be split, and calling them 'atoms'
would imply that they are, er 'atomic' would 'artifact splitting'
doesn't make much sense with them
[15:45:35] <richard_maw> but could be a conceptual model issue
[15:47:50] <paulsherwood> so far i'm favouring 'deployment', 'system',
'group' and 'component'... last call... any strong objections?
[15:48:26] -*- richard_maw wonders whether systems need to exist
[15:48:33] <richard_maw> long term anyway
[15:48:40] <richard_maw> short term I don't think we can come up with
anything better
[15:49:10] <paulsherwood> if they don't, then it has to be possible to
apply 'system integration commands' to groups?
[15:49:28] <CTtpollard> is group a package group?
[15:49:35] <16WAAO5A6> s/group/collection?
[15:49:41] <ssam2> systems should just be special kinds of groups
[15:49:54] <paulsherwood> lol. group is about to be ybd's internal
representation of the current 'stratum' concept.
[15:49:58] <paulsherwood> ssam2: +1
[15:50:39] <paulsherwood> 16WAAO5A6: more letters, and less popular in
our straw poll :)
[15:50:40] -*- richard_maw thinks systems should not be, and that it's
just convention to have a top-level group defined that isn't depended on
by anything
[15:50:59] <ratmice> rdale_ct: chunks can be split? does not artifact
splitting split 'artifacts', into different chunks?
[15:51:36] <richard_maw> a group is imbued with system-ness by virtue of
the fact that you are selecting it as the build target on the
command-line, either directly, or via a deployment
[15:52:28] <paulsherwood> richard_maw: oh, so just run all implied
s-i-commands in those cases?
[15:52:41] <richard_maw> yep
[15:52:57] <paulsherwood> works for me :)
[15:52:58] <richard_maw> whether a group is a system is just a point of view
[15:53:13] <richard_maw> kind-of philosophical, but cuts out the need
for an explicit concept
[15:53:15] <rdale_ct> you can say 'top layer', but you can't say 'top group'
[15:53:29] <pedroalvarez> hahah
[15:53:32] <paulsherwood> supergroup :)
[15:53:34] <richard_maw> rdale_ct: the top-layer doesn't imply the
system as a whole
[15:53:55] <richard_maw> paulsherwood: a supergroup is just any group
that contains/requires other groups
[15:54:01] <richard_maw> hence supergroups are just groups
[15:54:21] <richard_maw> you could have a convention to call the 'top
group' the system if you wanted
[15:54:43] <ratmice> or the root..
[15:54:54] <SotK> "a group which is not a subgroup"
[15:55:21] <richard_maw> another issue with calling them layers, is that
a layer is divisible from the things it depends on
[15:55:32] <richard_maw> the implication you could peel a layer off
something and put it on top of something else entirely
[15:55:39] <paulsherwood> richard_maw: i was joking, while thinking of
Cream and other 70s supergroups :)
[15:56:20] <richard_maw> mathematically the name isn't without merit
[15:57:02] <paulsherwood> ok, i think for ybd in this step it should be
'deployment', 'system', 'group' and 'component', with the stretch goal
of trying to remove the 'system' as discussed above
[15:58:27] -*- richard_maw is still concerned about the terminology of
how components interact with split artifacts, since that was terminology
we hadn't got right before either, and it's more difficult than just
replacing the name of a concept we were already sure of
[16:00:12] <paulsherwood> i take your point, richard_maw - that needs
more thought
[16:00:23] <ssam2> I had a go at making a generic data model for
build+integration instructions by the way
[16:00:37] <ssam2> here is a clumsy attempt at visualising its current
state:
https://github.com/ssssam/software-integration-ontology/blob/master/docs/...
[16:00:56] <paulsherwood> ybd has one artifact per component. istm that
splitting is an operation that gets performed on artifacts, not components?
[16:01:10] <ssam2> (the "Project" entity is tying in with the existing
DOAP data model and the "Package" entity is tying in with SPDX, you can
ignore them)
[16:01:36] <ssam2> thinking about it, having components and artifacts is
super confusing
[16:01:47] <ssam2> both really generic terms that could mean exactly the
same thing
[16:01:54] <paulsherwood> true
[16:02:18] <ssam2> source-component would be clearest (but probably too
long)
[16:02:22] <paulsherwood> erk :)
[16:02:36] <ratmice> i'm kind of partial to calling things more by what
they contain e.g. 'source heirarchy' vs hrm, ssam2 got there first :)
[16:02:55] <ssam2> Morph has always called these things Source internally
[16:02:59] <richard_maw> paulsherwood: I'm thinking split artifacts
should be another definition type which go in a different file, refer to
a chunk/source/component and select which files that should be part of it
[16:03:25] <rdale_ct> i would call chunks 'components' which can be
split into 'packages'
[16:04:01] <pedroalvarez> this bike is too big and has too many wheels .. :)
[16:04:25] <paulsherwood> pedroalvarez: agreed :)
[16:04:29] <rdale_ct> well, terrible names are a really important
baserock problem
[16:04:41] <richard_maw> and you can depend on a group (includes
everything the group "contains"), a "chunk" which contains all files
produced by the build, or a "split artifact" which contains a subset of
the files produced by the build
[16:05:17] <ssam2> note that in the data model I tried to produce, I
also tried to describe what each entity *is*, which was a useful exercise
[16:05:25] <pedroalvarez> subcomponents
[16:05:31] <richard_maw> rdale_ct: I'm liking the idea of packages for
split artifacts, not sure component is right for that any more, since
component could be a generic term that encompasses packages
[16:05:44] <ssam2> e.g. 'artifact': "A file or set of files that is
interesting for some reason."
[16:08:32] <richard_maw> ssam2: what do you call the combination of
build instructions, sources and dependencies that after building,
produces something that can be used?
[16:08:44] <rdale_ct> a 'build unit'?
[16:10:00] <ratmice> build tree?
[16:10:51] <ssam2> richard_maw: definitions?
[16:11:00] <richard_maw> I like package because it implies the selection
of which files should be included, but I don't think artifact fits in
the conceptual model, as packages are "artifacts" by a definition
[16:11:01] <ssam2> to use established baserock terms
[16:11:34] <ssam2> if I was talking in a context where 'definitions'
wasn't established, I think I'd try to spell out explicitly what I meant
instead of trying to find a word
[16:12:24] <richard_maw> ssam2: I view definitions as the textual
representation from the yaml document tree, rather than the internal
concept they are parsed into
[16:13:19] <richard_maw> so you can have group definitions, being the
text you write to define a group
[16:13:23] <ssam2> ok
[16:13:59] <richard_maw> but what is the equivalent of a chunk definition
[16:14:02] <ssam2> I would call the data structure that a build tool
operates on a 'build graph'
[16:14:12] <ssam2> that's a pretty common term I think
[16:14:21] <richard_maw> we kind of need a word that is indivisible
[16:14:34] <richard_maw> I don't think component necessarily works,
since it's common to see subcomponents
[16:14:46] <ssam2> chunk? :-)
[16:14:47] <richard_maw> you could use component to describe both groups
or chunks
[16:15:27] <richard_maw> ssam2: aye, for purity I think it fits, but
it's not an intuitive name
[16:16:18] -*- paulsherwood hopes for terms that newbies can grok,
without need for a comp sci background
[16:17:22] <richard_maw> hm, rdale's "build unit", while possibly
verbose, is both correct and more self-describing than [build] chunk
[16:17:38] <richard_maw> there's a bit of ambiguity of what a "build" is
[16:18:09] <richard_maw> since build describes both the top-level system
build, and the lower-level component build
[16:19:27] <paulsherwood> richard_maw: i think of top-level as assembly
[16:20:35] <paulsherwood> ssam2: how easy is it to generate that png? i
wonder i'm tempted to suggest that we try for an 'official' resolution
of this as one or more pull-requests to your repo
[16:20:51] <ssam2> it's a total pain to generate it
[16:21:09] <ssam2> involves taking a screenshot of a big Java app called
Protégé, with lots of babysitting to get a good layout first
[16:21:19] <paulsherwood> ouch
[16:21:24] <ssam2> there may be a better way
[16:21:26] <richard_maw> ssam2: ugh, can we just use dot?
[16:21:36] <ssam2> Tracker has some stuff in to convert OWL to dot, I think
[16:22:24] <ssam2> it would definitely be good to have the data model
formally specified somewhere
[16:22:32] <ssam2> probably not on my github though :-)
[16:22:38] <persia> A note on the semantics of "package": many folk seem
to think packages should support install-time integration scripts. If
this term is used, and one wants to be able to have reproducible or
comprehensible images, be very careful.
[16:22:40] <richard_maw> paulsherwood: if we can emphasise assembly as
the operation to convert a system/group selected to be the system into a
system artifact, then yes, I think we could use "build" for the
low-level bit
[16:23:55] <richard_maw> persia: hm, that would be staging area
construction time integration scripts, which we've managed to avoid so
far, by the few things that needed it putting extra stuff in
pre-configure-commands
[16:24:54] <persia> Yes. Also, many people believe you can "install a
package" post-deploy. I'm not saying don't use it, just be careful that
it isn't misunderstood due to heavy overloading.
[16:26:16] <paulsherwood> richard_maw: currently ybd has assembly as the
process of traversing the definitions to create a target, building
necessary components along the way, and combining them as the required
artifact.
[16:26:33] <paulsherwood> i guess you mean assmebly should only be the
combine step?
[16:26:40] <richard_maw> exactly
[16:26:59] <paulsherwood> ok
[16:27:03] <richard_maw> when building lego, you do the assembly part,
someone else has built all the blocks and provided instructions on how
to put them together
[16:27:39] <paulsherwood> good point. so traverse, build, assemble? is
there a better verb than traverse?
[16:28:14] <richard_maw> hm, there's parse, but that's more about
turning the tree of yaml documents into something you understand,
whereas building the graph is a step after that
[16:28:23] <richard_maw> analyse possibly
[16:28:33] <richard_maw> or use graph as a verb maybe
[16:28:43] -*- paulsherwood is finally admitting, to the assembled
internet multitudes, that the ybd code is hard to understand
[16:28:57] <paulsherwood> there, i said it. don't all chortle too loudly :)
[16:29:35] <richard_maw> this is in particular why I found the variables
labelled "this" so difficult
[16:29:40] <pedroalvarez> :)
[16:29:41] <paulsherwood> ack.
[16:29:54] <richard_maw> we have all these concepts floating around, and
if we don't use the proper label, we need an incredible amount of
context to know what's happening
[16:30:04] <ratmice> hrm wrt 'build unit' I was thinking 'origin' might work
[16:30:44] <richard_maw> ratmice: hm, sounds more related to where the
source code came from to me, but that may just be because the default
git remote is labelled origin
[16:31:35] <-- 16WAAO5A6
(~tiagogome(a)82-70-136-246.dsl.in-addr.zen.co.uk) has quit (Ping timeout:
260 seconds)
[16:32:48] <paulsherwood> i must admit i'm being tempted by 'unit' over
component (it's shorter, for a start)
[16:34:27] <rdale_ct> i'm not sure 'unit' works by itself 'build unit'
is about the granularity of the build at the bottom level
[16:36:42] -*- richard_maw is favouring 'build'
[16:37:34] <richard_maw> given the top-level operation is assembly, if
we were using unit or component, it would be build unit or build component
[16:37:46] <paulsherwood> whatever chunk moves to, it has more
significance than just being a 'build unit' - from an outside
perspective, it's the name of a specific project/program
[16:38:31] <paulsherwood> i would hate to have to rever to every project
as a 'build unit' as much as i've hated using 'stratum/strata'
[16:38:54] <rdale_ct> yes, you might be right
[16:39:11] <richard_maw> paulsherwood: I'm not sure project is right
either, since that's more about how upstream organises themselves,
rather than what we build
[16:39:25] <richard_maw> a project may produce many things we build
[16:39:40] <richard_maw> and a build may produce many programs or libraries
[16:39:46] <paulsherwood> true - i'm not suggesting project, but that's
the common parlance for some aspects of this concept outside baserock
[16:40:47] <richard_maw> aye, and most projects, mostly small ones, are
easily mappable to 1 source code repository and 1 build to produce the 1
program
[16:41:05] <richard_maw> but using it as terminology gets very clunky
when this isn't the case
[16:42:19] <rdale_ct> debian builds 'binary packages' from a 'source
package'
[16:43:31] <richard_maw> so we're back to a "build" for the translation
from source code into artifacts/packages
[16:44:16] <paulsherwood> ack
[16:45:48] --> tiagogomes_
(~tiagogome(a)82-70-136-246.dsl.in-addr.zen.co.uk) has joined #baserock
[16:45:56] <richard_maw> it's not a clean comparison, since "source
packages" are pre-processed to bundle up the build instructions with the
source code, whereas we don't have this bundling step
[16:48:12] <rdale_ct> we have a reference to a git repo, instead of the
actual source - but the thing we now call a 'chunk' as build
instructions plus source just like a source package to me
[16:48:42] <-- tiagogomes_
(~tiagogome(a)82-70-136-246.dsl.in-addr.zen.co.uk) has quit (Client Quit)
[16:48:52] <ssam2> well, 'source package' and 'binary package' don't
seem so bad. and should meet the criterion of being already established
[16:48:54] --> tiagogomes_
(~tiagogome(a)82-70-136-246.dsl.in-addr.zen.co.uk) has joined #baserock
[16:48:59] <richard_maw> rdale_ct: aye, except that a "source package"
is an artifact in debian's build process, while it's an internal detail
in ours
[16:49:10] <ssam2> why not 'source artifact' and 'binary artifact' or
'source component' and 'binary component' ?
[16:49:38] <rdale_ct> i like the component pair
[16:49:48] <paulsherwood> assuming we stay with 'artifact' for the
result of 'build' on a 'component/chunk/build-unit' - do we have a
different name for the result of 'assemble' on a 'group' or is it just
another kind of artifact?
[16:49:49] <richard_maw> for which bits? source artifact implies that at
some point we spit out a tarball of source code
[16:50:39] <richard_maw> paulsherwood: I'm thinking of artifact closer
to its dictionary definition by this point, in that it's anything
produced, hence cacheable
[16:50:46] <paulsherwood> output of build, tarred.
[16:51:03] <paulsherwood> but i'm ok with output of build or assemble,
tarred.
[16:51:39] <richard_maw> I wouldn't bother specifying the format necessarily
[16:52:39] <richard_maw> since it shouldn't matter whether it's a disk
image, loose files, or something else entirely
[16:52:45] <paulsherwood> fair enough, but we continue with artifact for
all kinds of outputs (chunk, strata, system)
[16:53:35] <richard_maw> ack, whether stratum artifacts need to exist is
another matter
[16:53:41] <paulsherwood> agreed
[16:57:03] <rdale_ct> where have we ended up with 'source xxxx' and
'binary xxxx'?
[16:59:48] <rdale_ct> for me xxxx: component > package > artifact
[17:04:41] <paulsherwood> i'm currently assuming 'artifact' is cached.
unclear what splitting generates, or where 'package' fits
[17:06:02] <rdale_ct> a 'source component' builds a 'binary component'
which can be split into 'binary sub components'
[17:06:35] <richard_maw> paulsherwood: I'm thinking a "package" is the
result of a split, but that you can depend on a build directly, which
has a default package containing everything, or the package
[17:07:33] <paulsherwood> why wouldn't 'split' just be something done as
part of 'assemble'?
[17:07:42] <richard_maw> in debian parlance, split artifacts are
"packages" or "binary packages", while "source packages" are packaged up
"build" instructions with the sources
[17:08:41] <paulsherwood> rdale_ct: 'source component' 'binary
component' 'binary sub components' are too complicated, imo
[17:08:48] <richard_maw> paulsherwood: because assemble is the top-level bit
[17:09:20] <paulsherwood> richard_maw: ack. but what's the use-case for
splitting a component on its own?
[17:09:57] <-- faybrocklebank
(~faybrockl(a)cpc44-bagu10-2-0-cust229.1-3.cable.virginm.net) has quit
(Quit: Leaving)
[17:10:12] <rdale_ct> a 'source component' builds a 'binary artifact'
which can be split into 'packages'
[17:10:40] <richard_maw> paulsherwood: you mean that you don't have an
explicit step for producing a "package", rather that you filter the
"build"'s output when using the "package"?
[17:10:55] -*- richard_maw had considered that to be a possible optimisation
[17:10:57] <paulsherwood> richard_maw: yes, i think so
[17:11:16] <richard_maw> yeah, no reason you can't do that
[17:11:58] <richard_maw> I take it your model has assembly as both the
construction of the top level system artifact, and the construction of
the staging areas that builds are run in too?
[17:12:41] <paulsherwood> currently, yes. but i'm happy with the idea of
traverse, build, assemble as above - in which case staging probably gets
created in traverse
[17:13:06] <paulsherwood> (or maybe not, unclear)
[17:13:12] <richard_maw> can't, you need the result of some builds
before you can assemble the staging area AIUI
[17:13:39] <richard_maw> I have no problem with build and assemble being
corecursive
[17:14:12] <richard_maw> but we made system assembly different from
staging area assembly different for optimisation reasons
[17:14:22] <paulsherwood> yes. traverse could deal with that. and build
is actually (check if already built, then check remote cache, then build)
[17:14:25] <-- ctbruce (~bruceunde(a)82-70-136-246.dsl.in-addr.zen.co.uk)
has quit (Quit: Leaving)
[17:15:38] <richard_maw> paulsherwood: I think we might be hitting an
impedence mismatch here, since in my conceptual model traverse builds a
data structure that gets executed, while I think yours has traverse
causing assemblies and builds
[17:15:39] <rdale_ct> i'm not sure that ybd traverses anything because
it doesn't create a build graph
[17:15:50] <paulsherwood> in part i like the traverse/build/assemble
idea because the assembly code is currently almost as hard as the
definitions code to grok. it has to be multi-instance, idempotent, atomic
[17:15:51] <richard_maw> rdale_ct: we've had this discussion before
[17:16:01] <richard_maw> rdale_ct: ybd has an implied one rather than an
explicit one
[17:16:10] <paulsherwood> yes :-)
[17:16:42] <rdale_ct> in terms of naming functions within ybd, i can't
see the word 'traverse' being useful
[17:16:48] <paulsherwood> in the previous discussion i had to admit i
didn't understand what a graph was, because folks seemed to be thinking
i was being wilfully obtuse
[17:16:54] <paulsherwood> rdale_ct: ok
[17:17:12] <paulsherwood> any alternative suggestions?
[17:18:14] <rdale_ct> i think 'assemble' and 'build' are fine, they just
need to include what they are building or assemblying, which they don't
at the moment - eg 'assemble_system'
[17:19:44] <rdale_ct> i don't go a bundle on 'do_' as a prefix either, i
would rather just have 'build()' rather than 'do_build()' - it doesn't
seem to mean anything to me
[17:19:47] <paulsherwood> trouble with that is that we can apply ybd to
any level... and previosly we established that a system is only a subset
of 'group' really :)
[17:20:20] <paulsherwood> rdale_ct: agreed - that's a recent hack, to
wrap build()
[17:21:37] <richard_maw> there's a separate question of whether we
should be able to do that, previously there was contention because we
wanted to have higher-level definitions able to pass down parameters
like the architecture
[17:21:49] <richard_maw> now, I'm happy to be able to build groups directly
[17:22:00] <rdale_ct> can the system-integration commands be run on a
group at any level - or do they only make sense to run them on what we
now call 'systems'?
[17:22:24] <paulsherwood> any level, i think
[17:22:55] <richard_maw> shouldn't be a problem, we want groups to
include all the commands they need to be able to do that
7 years, 4 months
GENIVI reference system
by Sam Thursfield
Hello!
I admit I've not been following GENIVI and Automotive Grade Linux
recently and have got totally confused about GENIVI Baseline vs. GENIVI
Demo Platform vs. Unified Code Base.
Is the GENIVI Baseline reference system in definitions.git obsolete or
is it still useful?
Did AGL and GENIVI agree to standardise on Yocto recently for all future
work? That's the impression I get from here:
<https://lwn.net/Articles/671153/> but I could be misunderstanding.
Is there any ongoing use of Baserock within GENIVI (for the demo
platform perhaps?) or not?
I'm asking just because right now I don't have a clue and I am curious!
(Of course, maybe nobody knows yet :-)
Thanks
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
7 years, 4 months
Baserock StoryBoard has been upgraded!!
by Pedro Alvarez
Hi all!
Last year I spent some time trying to get our StoryBoard deployment
updated, given that we were using a really old version with too many
bugs... Turns out StoryBoard developers have been working hard and we
have some improvements with the new version:
Recent events:
--------------
The Recent Events weren't working before. You can find it on the
Stories Dashboard [1]. On this list you will see now new events and
that happened to the stories you are subscribed to. It's a nice
feature to follow what's going on with some stories.
[1]: https://storyboard.baserock.org/#!/dashboard/stories
Email Notifications:
--------------------
Yes, emails! They are in an early stage, but you can give them a try
to see how useful they are for you, and maybe to give some feedback to
StoryBoard devs. To enable them you only have to go to your Profile
Preferences page [2] and check the box that says "Enable notification
emails". As in Recent Events, you only will get emails for stories
that you are subscribed to.
[2]: https://storyboard.baserock.org/#!/profile/preferences
Boards (Kanban) and Worklists:
------------------------------
Now we can organise what we are doing in Kanban boards or in
Worklists. You can create one using the "Create New..." button located
at the top bar. It's possible to create private and public
Boards/Worklists, and also is possible to set various owners and
users. I've created an example Board that we could use for tracking
work done on the build tools, called Baserock build tool board [3]. If
you want access to it, you can request it to me by email or IRC.
[3]: https://storyboard.baserock.org/#!/board/1
Other improvements:
-------------------
- Markdown descriptions and comments, with preview buttons.
- Pagination fixed in various places. This solves the problem of
"comments not being posted", for example.
- Various UI fixes
I'm sure I'm missing other important improvements, but this is all I
can think of now. I still have to send the patches to
infrastructure.git, but I hope to get to it soon.
I hope people is interested on giving it a try, and using it. I think
it can help us a lot to stay a bit more organised.
Regards,
Pedro.
7 years, 4 months
Impediments in the path of shallow git mirrors
by Richard Maw
Warning: This will be a long mail to read.
I was asked if I could make it possible to mirror gcc with shallow history.
After a bit of investigation, I said yes, but it will be difficult.
I stand by that assessment, but I no longer have the time to do so,
since I hit a problem that increased the work required over my available time.
The original plan
=================
1. Extend gitano to let you configure it to accept shallow history.
Patches to support this were sent upstream.
2. Make lorry-controller enable shallow when creating repositories.
3. Make lorry support fetching from shallow repositories.
4. Add extra configuration for git lorries to:
1. Allow you to specify which refs to fetch.
This would let you only fetch important tags and refs.
2. Allow you to specify a fetch depth.
This would allow you to make a shallow repository out of a deep one.
The flow for performing a shallow import would then be:
1. lorry-controller reads the config, and if it's new, creates the repository
and sets the
2. lorry runs the equivalent of:
`git fetch $upstream_url --depth=$depth --update-shallow $refspecs`
3. lorry runs the equivalent of:
`git push ssh://git@localhost/delta/$repo $refspecs`
Reality
=======
Unfortunately I encountered many issues,
since shallow mirrors are a use-case that git upstream hasn't considered before.
Problems
--------
1. We need to push the new history to the trove,
but shallow push breaks if you push more than one branch at a time.
We can work around this by performing multiple pushes,
which would allow lorry to work without requiring a new version of git.
2. We lose the ability to detect non-fast-forward updates.
We can work around this by setting `--force` when fetching.
See later for why, and how it can be fixed.
3. Converting an existing repository to/from a shallow mirror isn't possible.
The problem is that pushing from source to destination isn't the same
as fetching from destination to source.
This would cause us trouble if we wanted to deepen the history of a shallow.
An exceedingly ugly work-around is possible,
and is detailed later in this document.
4. Old, downstream troves, will be unable to fetch from shallow repositories.
We could hide the repositories from downstream troves,
but gitano upstream doesn't want shallow repositories to be hidden
either by default or by configuration option,
so we'd need to fork or write a plugin.
Fixing shallow fast-forward detection
=====================================
Discussion with upstream can be found at:
http://thread.gmane.org/gmane.comp.version-control.git/283495
1. Have the sender advertise the "ancestry-check" feature.
2. Have the receiver request ancestry before sending "want"s.
See `find_common()` in `fetch-pack.c`
3. Have the sender reply with the relationships.
See `receive_needs()` in `upload-pack.c`
4. Have the receiver augment its `struct ref`
to add the ancestry result.
5. Continue as normal until `update_local_ref()` in `builtin/fetch.c`,
where the `in_merge_bases()` check should use the ancestry result.
If the server doesn't support "ancestry-check",
then it would be convenient to degrade to forcing the fetch,
which would require adding an extra config option for fetch.
Working around the lack of a way to specify depth with push
===========================================================
This is a pretty evil solution, but would work.
It would also work around the bug only allowing one ref push.
The gist of it is to add a plugin to gitano that adds a `reverse-fetch` command,
which boils down to the following, after ensuring the auth hooks support it.
reverse_fetch() {
repo="$1"
shift
cd "$repo"
git fetch fd::0,1 "$@"
}
Then rather than allowing lorry to push its changes,
lorry controller must do the push with reverse-fetch,
since lorry is explicitly unaware of gitano extensions.
Pusing with reverse-fetch involves expanding the push refspec, reversing it,
and plugging a send-pack command into the ssh command.
loony_push() {
# Note, not quoting or whitespace safe
fetchcmd="reverse-fetch "$fetchrepo" --depth=$DEPTH "$refspecs")"
socat EXEC:"git upload-pack $(quote "$GIT_REPO")" \
EXEC:"ssh git@localhost $fetchcmd"
}
`--depth` is required if the repo is shallow.
If before lorry fetched the repo was shallow, but after it is not,
then `--unshallow` must be passed to the fetch, rather than `--depth`.
`--unshallow` may not be specified at any other time.
Preventing downstream trove breakage
====================================
Old downstream troves can't handle shallow repositories.
Unless we can ensure there are no old downstream troves,
we must hide shallow mirrors from gitano's `ls` output,
so that they don't attempt to mirror them.
There is precedent, since gitano supports archiving repositories to hide them.
However gitano upstream won't accept a patch that hides shallow repositories,
either by default or by configuration option.
So instead this would require a plugin to wrap the "ls" command,
so that unless you specify `--show-shallow`, they aren't listed.
Then new versions of lorry-controller could opt into shallow support,
by setting `--show-shallow` after it has been fixed to support shallow.
7 years, 4 months
weirdness in ybd, or sandboxlib, or their environment?
by Paul Sherwood
Hi all,
recently a couple of users have been finding cases where ybd was
failing to build known-good definitions. I'm struggling to debug what's
going on, because the problem doesn't occur on any of my normal test
scenarios (i.e. AWS, Ubuntu VirtualBox on Mac, Ubuntu on HP Moonshot
AARCH64 and ARMv7 chroot). It *does* occur reasonably frequently on one
of our Concourse setups, and has been happening reproducibly for Tristan
on his work deep in the outback with Aboriginal Linux.
I've tweaked master YBD to ensure that uncaught exceptions explode
properly (16.01 and previous releases are hiding them, sadly) and
confirmed that an example of the problem is as follows.
From the exception, it seems somehow sandboxlib is trying to an already
mounted or busy proc directory. It's obviously possible I've broken
something, but I'm wondering if this could be somehow as a result of
whatever container approach Concourse is using?
Any ideas?
br
Paul
16-01-12 03:34:52 [14/121/121] [linux-api-headers] Running command:
case "$MORPH_ARCH" in
armv5l)
ARCH="arm" ;;
armv7b|armv7l|armv7lhf)
ARCH="arm" ;;
armv8l64|armv8b64)
ARCH="arm64" ;;
x86_32)
ARCH="i386" ;;
x86_64)
ARCH="x86_64" ;;
ppc64)
ARCH="powerpc" ;;
*)
echo "Error: unsupported Morph architecture: $MORPH_ARCH" >&2
exit 1
esac
ARCH=$ARCH make INSTALL_HDR_PATH=dest headers_install
WARNING: ['umount', '/root/ybd/tmp/tmpSuZip3/dev/shm'] failed: umount:
tmpfs: mountpoint not found
WARNING: ['umount', '/root/ybd/tmp/tmpSuZip3/proc'] failed: umount:
proc: mountpoint not found
16-01-12 03:38:44 [14/121/121] [linux-api-headers] Running command:
install -d "$DESTDIR${PREFIX-/usr}/include"
Traceback (most recent call last):
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/__main__.py",
line 65, in <module>
assemble(defs, target)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 92, in assemble
preinstall(defs, component, subcomponent)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 154, in preinstall
preinstall(defs, component, it)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 147, in preinstall
preinstall(defs, component, it)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 147, in preinstall
preinstall(defs, component, it)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 154, in preinstall
preinstall(defs, component, it)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 147, in preinstall
preinstall(defs, component, it)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 154, in preinstall
preinstall(defs, component, it)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 147, in preinstall
preinstall(defs, component, it)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 154, in preinstall
preinstall(defs, component, it)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 156, in preinstall
assemble(defs, dependency)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 101, in assemble
do_build(defs, component)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 111, in do_build
build(defs, component)
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/assembly.py",
line 187, in build
allow_parallel=('build' in build_step))
File
"/tmp/build/641c0e3f-a9d8-4e7f-5533-e94ca07e9810/ybd/ybd/sandbox.py",
line 155, in run_sandboxed
env=env, **config)
File "/usr/local/lib/python2.7/dist-packages/sandboxlib/chroot.py",
line 227, in run_sandbox_with_redirection
exit, out, err = run_sandbox(command, **sandbox_config)
File "/usr/local/lib/python2.7/dist-packages/sandboxlib/chroot.py",
line 208, in run_sandbox
with mount_all(filesystem_root, extra_mounts):
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/local/lib/python2.7/dist-packages/sandboxlib/chroot.py",
line 139, in mount_all
mount(source, path, mount_type, mount_options)
File "/usr/local/lib/python2.7/dist-packages/sandboxlib/chroot.py",
line 112, in mount
argv, err.decode('utf-8')))
RuntimeError: ['mount', '-t', 'proc', '-o', '', '',
'/root/ybd/tmp/tmpSuZip3/proc'] failed: mount: is already mounted or
/root/ybd/tmp/tmpSuZip3/proc busy
7 years, 4 months
YBD 16.01 is released
by Paul Sherwood
Hi all,
the main functionality change since the last release is that ybd [1]
now refers to an artifact server [2] by default, so that the
user-experience for building from scratch should be faster for new
users. The artifact server is being populated manually from time to time
for master definitions (ARMv7 and x86_64) - I'm hoping to establish a
cost-effective CD pipeline for this in the near future.
Also, a few annoyances have been fixed - in particular some weird
behaviour that was making YBD blow up in Concourse.
Full list of commits is below:
92727a5 If there's a definitions dir below us, cd into it
2644bcb Log how and where ybd is invoked
350e027 Trap and warn on non-dict definitions
a476e21 Explicitly log what the dirs are set to
f00d7c9 Exit if cache artifact dir is not created properly
ffce239 Beginning Dockerfile and pipeline for concourse
ca52aa5 Exit tidily if sandbox install fails
254a057 Start of concourse pipeline - gets definitions and ybd
31b63a4 Set artifact cache server by default
12d3eac Pep8
c3be0b7 Specify requests as a dependency, fixess #147
6abf30f Erk, fix bugs in 984ed94 and 9a13f50
9a13f50 Don't include clusters in task count
984ed94 Tidyup deployment/timer code
c401fe0 Only do deploys for specified architecture, and time them
br
Paul
[1] https://github.com/devcurmudgeon/ybd/tree/16.01
[2] http://artifacts1.baserock.org:8000
7 years, 4 months
Bootstrapping multi-arch baserock
by Tristan Van Berkom
Happy 2016 hackers,
So in the last week of 2015 I was looking into cross compiling with
Baserock, which just turned out to be a quick study of the kinds of
roadblocks we might face if we were to try such a thing.
At the same time I have been trying to build Baserock on emulated
systems (qemu) with varying success.
The goal of course, is to try out Baserock and see if I can build GNOME
for another architecture, how easy is it ? what is the procedure for
supporting a new arch with Baserock ?
Before I get into the juice of what I'm trying out now, let me just
quickly gloss over the approaches I'm pretty sure to abandon.
Cross-Compiling
~~~~~~~~~~~~~~~
This is pretty much debunked. Yes, some projects successfully cross
compile entire operating systems, such as buildroot and yocto based
builds.
While it is possible to pull off, it represents a huge amount of work
to get off the ground and a constant maintenance cost to upgrade;
upstream modules are just not built to be easily cross compilable,
and that is quite alright.
Raspbian under QEMU
~~~~~~~~~~~~~~~~~~~
Gives you a quite functional debian installation for arm arches,
should be sufficient for compiling a system like GNOME for arm using
current baserock definitions 'as is'.
Unfortunately, this took me over a day to just get started. There is
documentation on the web on how to boot raspbian in qemu and most of
this points to dead links for either the kernel, the raspbian image
or both. After some time I was able to find a raspbian kernel and
image combination which worked.
Using this image, I was unfortunately unable to build stage1-gcc in
Baserock for either armv7lhf or armv5l arches (not that it's not
possible, but it did not work after a few attempts).
Baserock under QEMU
~~~~~~~~~~~~~~~~~~~
I tried to boot the build system for the armv7lhf[0] arch using qemu
and futzing around with the genivi-initial-setup scripts[1] by
mounting the built image and extracting it's kernel, but did not have
much luck with that, it would seem those scripts are also severely
out of date and expect one to be running an (old) ubuntu system.
There is probably a correct voodoo encantation of qemu-system-arm to
boot this image, but I gave up before finding out what it is.
The sad thing about the latter two approaches is that we would need an
image for a given arch to build an image for that arch, but there is
not much in the way of a build story to support a new arch and build
one of these images from scratch the first time around.
This means that if we want to build on a given board, we depend on
third parties to build an environment with a complete enough stack to
run ybd first.
On to the juice
~~~~~~~~~~~~~~~
Enter Aboriginal Linux[2]. As Base Rockers, I expect you are already
familiar enough with what the Aboriginal Linux project is, but I was
not, so here it is in a nutshell:
o Motto is "We cross compile so you dont have to".
o Has a documented procedure for bootstrapping a bootable
environment for 16 different arches (as of this week, so I hear on
#toybox irc). Also, prebuilt images are provided for popular
arches.
o Provides the smallest possible toolchain on all supported arches.
o Has a documented proceedure for extending the image, currently this
is done with tarballs. There is an LFS "control image" which they
build regularly to test that procedure.
o Maintains their own launch scripts to get the image running in
qemu (and these launch scripts "just work", without hours of pain).
o For each arch, there is also a provided cross compiler which can be
run from x86 arches, now here comes the really neat part...
o If setup correctly, uses distcc on the host running QEMU to run the
provided cross compiler, offloading the compilation workload from
the emulator to the host.
This brand of cross compilation is entirely transparent and short
circuits the whole mucking about with host vs target paths and
build script hackery normally involved in cross-building.
It also allows me to run "make -j8" from inside an arm emulator and
have my host cross-compile objects on behalf of the emulator at
high speeds and 8 threads at a time.
Currently I am building an extended toolchain in an armv5l Aboriginal
Linux emulator as a proof of concept, theoretically I just need to
build everything that ybd requires to build the build-essential stratum
and once that is passed the build should truck along (slowly) and
eventually complete. The only stumbling blocks would be some modules
which require special treatment to be built on a given arch.
If this is successful, which I don't see why it should not be, I am
considering Aboriginal Linux as a viable bootstrapping platform for
building systems with Baserock.
So, assuming this plan is a success, this is what I think would be
involved in getting a thoroughly repeatable and documented
bootstrapping process for building Baserock for a given arch:
o Create our own Aboriginal Linux derived platform with an extended
toolchain, just enough to run ybd and get passed build-essential.
This probably means building the bootstrap image using tarballs at
least at first, possibly a repository of them sitting on the trove.
This repository should be able to bootstrap itself into all of the
arches supported by Aboriginal Linux.
It will also serve as documentation about definitions, i.e. it can
be deduced that everything that is built in the bootstrap image, is
in fact a requirement for building Baserock, it is the missing
link.
o Since we are already building a cross compiler for the bootstrap
image to compile with, it would make sense to drop our
[cross]compiler from the build-essential stratum, this way we can
run ybd on definitions but still use the aboriginal distcc "trick",
allowing us to build full systems for embedded targets at
reasonable speeds.
o Drop host dependencies from build extensions
There is no reason to require tools such as mkfs.btrfs from the
host, and indeed, as was outlined in this bug report[3], we need to
avoid it.
There are some concerns about maintaining cross-deployment
capabilities in the build extensions which I admit I don't
completely understand (how can you cross-deploy if you are not
cross-building in the first place ?). In any case this should be
dropped as it would become non-essential to the mission and as
such; only add to the overall maintenance burden of Baserock.
Any thoughts, comments ?
Best Regards,
-Tristan
[0]:http://download.baserock.org/baserock/baserock-current-build-system
-armv7lhf-jetson.img.gz
[1]:git://trove.baserock.org/baserock/baserock/genivi-initial-setup
[2]:http://landley.net/aboriginal/
[3]:https://storyboard.baserock.org/#!/story/65
7 years, 4 months
New year's resolution: improve desktop Linux distributions
by Sam Thursfield
Hi all
If I understand correctly, part of the reason for starting work on
Baserock was to research fixing various problems in existing desktop
Linux distributions.
The Baserock project is 4 years old now, and I thought maybe it's time
to look at that aim again.
I had a go at listing everything that is wrong with the current set of
free software operating systems that I could theoretically run as a
desktop OS. Please let me know if you disagree with any of it or if you
can think of other things to add.
Here's the list.
Arch Linux:
- infinite number of possible variants that cannot all be tested
- upgrades are not atomic
- cannot rollback after upgrade
Baserock reference systems:
- lack of packages
- lack of easy runtime configurability
- lack of momentum
- lack of good documentation
- no serious handling of security updates
- security fix in a single component may require rebuilding 1000s of
other components
Debian/Ubuntu:
- very complex, lots of moving parts, many of which overlap
- sources spread across many repos of different types
-> dgit mitigates this problem: it makes all Debian source packages
available through consistant Git interface
- infinite number of possible variants that cannot all be tested
- upgrades are not atomic
- cannot rollback after upgrade
- packages lag behind upstream versions at times
- packages sometimes have a big delta compared to upstream
Fedora:
- very complex, lots of moving parts, many of which overlap
- infinite number of possible variants that cannot all be tested
-> rpm-ostree project will help with this: you can create a manifest
that is a specific set of tested packages, and provide the
resulting rootfs as a single OSTree commit. Users can then layer
changes on top of the well-tested base image, and those changes
could be reapplied on each upgrade of the base OS.
- upgrades are not atomic
-> rpm-ostree project will help with this: a rootfs managed with
`ostree` can be upgraded atomically
- cannot rollback after upgrade
-> rpm-ostree project will help with this: a rootfs managed with
`ostree`can be rolled back to an old version
Gentoo:
- no way to share prebuilt artifacts
- infinite number of possible configurations that cannot all be tested
- upgrades are not atomic
- cannot rollback after upgrade
GNU Guix Software Distribution:
- lack of packages
- nonstandard filesystem layout makes packaging some software very
difficult
- security fix in a single component may require rebuilding 1000s of
other components
-> experimental 'grafts' feature solves this:
https://www.gnu.org/software/guix/manual/html_node/Security-Updates.html
- no serious handling of security updates
NixOS:
- lack of packages
- nonstandard filesystem layout makes packaging some software very
difficult
- security fix in a single component may require rebuilding 1000s of
other components
- I'm not sure how quickly NixOS handle security updates currently
I'm doing a talk in the distros devroom in FOSDEM 2016 which will relate
to all this a bit...
Happy Christmas & New Year
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
7 years, 4 months
