On Sat, Feb 20, 2016 at 3:58 PM, Paul Sherwood
On 2016-02-20 12:54, Sam Thursfield wrote:
> I'd like to propose a new repo that would be the canonical location of
> the Baserock definitions format specification.
I'm interested in this, but instinctively against the idea of separating the
spec from the content.
My underlying concerns are:
- it's adding another moving part
- it will make things more complicated for users to understand, rather than
- things will get out of step
I think in this case the spec and the reference definitions are two
different things, and it's actually *more* confusing to put them in
the same repo.
What do you think will get out of step, exactly? The definitions.git
repo clearly advertises what version of the format it expects
(VERSION). The build tools don't need to depend on the 'spec.git'
repo, and if they do, they can embed it as a submodule or something so
they track an exact version.
As I said before, it's pretty useless to have the migrations in master
of definitions.git, so there already needs to be a separate repo for
those. Given that, this proposal doesn't actually increase the number
of moving parts at all.
> Here is what I think it should contain:
> And it would live at git://git.baserock.org/baserock/baserock/spec
... why not a spec directory in definitions?
Its history will become tangled with the day-to-day development of the
various reference systems. If the spec is in its own repo, we can have
tags like v7, v8, v9. We can do that with definitions.git, too, but
there are already lots of tags that are unrelated, and lots of commits
that are totally unrelated to the evolution of the spec.
It also marks a particular definitions.git repo as being "special",
which I don't think is sensible. We already moved repos once (from
morphs.git). Putting the spec in with our reference definitions would
be annoying for people who want to write definitions independently of
ours, but following the same spec.
This is kind of like saying that the ANSI C standard should live in
the GCC.git repo, or that the POSIX spec should live in with the code
for BSD UNIX.
> I want to do this mainly because I think it makes it more obvious
> to contribute improvements to the format.
Not to me, I'm afraid. It may make it easier to *change* the format, but
theoretical improvements with no supporting practical examples may not be
improvements at all.
To put it another way, currently definitions + build tool together provide
some proof of validity (testing) of a notional spec. Creating a spec repo
without coupling it to definitions and tooling is effectively divorcing it
The spec repo will be subject to normal patch review. I think we can
deal with the idea that changes to it should only be merged if at
least one build tool already understands the new version. We already
require a migration script, which means it has to be tested on
> Everything that needs to be
> updated for a new version of the format is in this repo, and so each
> proposed change is basically a patch to this repo and can go through
> our usual patch review process (possibly on the mailing list instead
> of Gerrit -- I think changes to the format need to have as much
> visibility as possible).
In general I think we lost some visibility by adopting gerrit, so I'm in
favour of changes on ml.
But if this is just a specification, what would be the basis of review? the
spec is independent of code, tests, examples?
No, it would just be in a different repo. Also, the migrations are
code and would live in with the spec and be a part of any change to
> The reference system definitions in definitions.git would no
> contain anything to do with the definitions format.
Why do you think this is a good thing?
To give an example from last week, I'm in favour of Dan's proposal for
definitions version 8, but I don't think we should migrate the
reference definitions.git to version 8 until Morph supports the new
version. It's hard to express that with everything in the same repo.
If the spec is separate from the definitions then it becomes obvious
that we can change one without immediately changing the other.
> Thus it's clear
> that the two things are separate,
We've previously suffered by not having a spec for definitions (although we
learned quite a lot along the way). But this suggestion creates a spec which
will be independent of definitions - what is the incentive (or mechanism) to
keep them aligned?
The VERSION file. what's the incentive to *not* keep them aligned? I
guess it's possible that people could start proposing changes to the
spec without bothering to implement support for them in any build
tools, but we can easily not merge those patches. The sky will not
> and we can introduce a new version
> without necessarily migrating everything immediately.
Couldn't we do this already? IIUC we've been introducing and applying
migrations at the same time by convention, but it's not actually required?
ie we could introduce a migration as one commit. then later apply the
migration (or a set of them) and update the version.
> We've discussed this in the past, and already agreed that
> migrations need to move out of the definitions.git repo
I genuinely don't remember this agreement, would appreciate a link to the
discussion if you have one?
It was in IRC. Here's an example:
* infrastructure.git is at VERSION 4
* I want to merge 'master' of definitions.git, which is at VERSION 7
* To reduce merge conflicts, I want to run the migrations for version
4 to 7 before running `git merge`
* But, the migrations will only be available *after* I run `git
merge`... so instead I have to clone definitions.git somewhere else,
and copy the migrations/ dir into infrastructure.git, then run them
This isn't hypothetical, this is an experience I have actually had.
It's not the end of the world, but it really feels like a bad design.
IIUC this is your key reason for the proposed separation?
I guess the reasons are:
* we need a separate repo for migrations anyway
* we should encourage use of the definitions format outside our little
world of definitions.git
* we should make easy to spread the introduction of a new version over
a short period, rather than making it seem that the
* it's easier to look at the spec and its history if it isn't a
subdirectory of a much busier repo
I've been pondering how best to handle forking of definitions for
time, and I'm sure I've suggested what follows before, but maybe not
on-list. Here goes:
- definitions are more like data than code
- users will inevitably want to create their own data, even if they're not
'forking' in the normal code sense
- this situation seems to be different from normal forking of a code repo
- in particular
- for code there's a theoretical best practice objective for downstream
users to upstream their stuff to reduce costs
- here, a significant amount of what users care about should never be
upstreamed (their systems, custom strata etc)
IIUC Yocto/bitbake deal with this by recommending users to create new
recipes which derive from the existing upstream ones. There are some
practical implementation details of their approach which lead to
complexity, but I think the core principle is correct.
In my view the simplest way to support users' custom data is for them to
create their definitions in different directories/files from the standard
examples. So based on the current structure, I'd go for:
I think key advantage of this would be that we can expressly tell folks:
- by keeping downstream content in separate files, new versions of upstream
definitions can always be merged without conflict
- if you find you're needing to change the upstream definitions, either
- please consider offering your patches upstream
- if your changes don't go upstream, maybe you should be creating copies
and maintaining those
We could make this more obvious by sticking all of our directories one level
down in a namespace (eg baserock? or master? or examples? or reference?)
Ultimately I'd like to see the format evolved so that names are unique,
meaning that tools could parse without reference to paths, but that's many
steps from where we are now, and may even be undesirable.
I agree with all that.
> Hopefully keeping everything in one repo makes sense to
> others. Let me know what you think.
Keeping everything in the definitions repo makes more sense to me, but as I
said above, I'll be happy if others can expose the gaps in my reasoning.
I meant "keeping all format spec-related stuff in one repo", I guess I
wasn't clear there .