On Mon, 2016-10-10 at 11:57 +0100, Paul Sherwood wrote:
I broadly agree with your summation of these two problems, so will
focus on your proposed solutions...
Great, comments below...
On 2016-10-10 11:13, Tristan Van Berkom wrote:
> TL;DR: Creating a system with baserock means that one has to work
> git repository cluttered with a vast amount of irrelevant "stuff".
> Changes are pushed/forced upon higher level projects/components
> TL;DR: Changes in low level strata are forced upon thier consumers,
> regularly causing breakage and making it impossible for projects
> depending on low level strata to ensure their systems actually
> Proposal to use recursive git submodules to address this issue
> Yes I said it, and I know there are plenty of reasons to hate a
> submodule, obnoxious things as they are. However I have not yet
> able to come up with a better solution.
Well, your basic argument is that keeping everything in one repo is
much clutter, and I can't really disagree with that. There are only
ways to solve this:
- somehow (ab)use git to keep the clutter in separate branches of
- or use multiple repos
I'm willing to accept that the first approach would be too much
trickery and lead to confusion, so we're down to what's the best way
combine multiple repos in one body of work.
Git submodules are not the only one. I'm not a git wizard but I
that 'git subtree' is broadly similar but also less invasive. I
we consider both approaches to determine what's best for our
I'm reading some blogs about git subtrees right now, looks interesting
> So here is how I would propose to address the issues: I would first
> propose the following split of definitions, each shared git repo
> be comprised of one or more strata. Systems would be split out into
> repositories holding their own system specific strata, system
> deployments and extensions.
I'm ok with the general principle - others can consider the details
what goes in which subtree/submodule
> As far as I can see the main caveat in the above proposal is that
> sometimes we will want to reorganize things, for instance something
> from GNOME might become desirable in the freedesktop module in
> be reusable by another system; in this case we have some
> of history.
> I dont think this horribly bad, analogy: it has happened before
> custom widgets (GtkOverlay and GtkInfoBar) have been developed as a
> part of the gedit program, but were later considered desirable for
> other applications and then merged into GTK+ instead.
> The above is IMO a much better scenario than if all of the GNOME
> Desktop were to be maintained in a single git repository.
I'd just note that whatever the grouping that we settle on, there'll
cases where downstream users need only part of a subgroup - so
they'll still have the clutter you've described! ???
I think not, the analogy I made for this specific case was:
"When I write a C program, I strictly do not leave any deadcode
lying around in my own repository, but I dont mind linking against
Qt even if I do not happen to use every function provided by that
I.e. the presence of freedesktop inside the GNOME project module should
be seen as an opaque dependency, not as a part of the GNOME workspace.
If I have a problem with the freedesktop module, I should treat that as
I would a dependency of a C program. I should fork that freedesktop
module to apply my fix and propose a patch for it upstream, once it's
accepted I can depend on a newer version of freedesktop.
However, this presents a technical difficulty which I had overlooked,
which is what directory structure would work.
With submodules, I had envisioned something like this:
./gnome/freedesktop <-- git submodule
./gnome/freedesktop/gnu-linux <-- recursive git submodule
The problems with the above are:
a.) The modules I depend obnoxiously appear inside my own work
directory (although I find this much less disturbing than the
present situation: The only power I have over this submodule is
to decide what version I'm building, I might not even have
commit rights to the freedesktop upstream git).
b.) Definitions needs to support relative path references, because
the project root for building gnu-linux is different when
building from inside freedesktop, and likewise again different
when building inside gnome.
If technically possible, it would be nicer to have a way to setup a
work directory like this:
However it should be the gnome git history which dictates what version of freedesktop is
used, and in turn the freedesktop git history which determines the version of gnu-linux,
so this part indeed requires a bit more thought.
> Thoughts on this proposal ? Improvements on the given proposal ?
> counter proposals to address the same issues in a different way ?
I'm ok in principle with your proposed separation concept in
provide we justify the subtree vs submodule choice.
Just to note, though - one of the things we need to consider in any
change like this is super-long-term reproducibility. Assuming we
down this path, how are users expected to follow?
- do we go with morph's approach of forcing users to upgrade their
definitions and tool at the same time?
Certainly not, or, only in the worst of cases moving forward.
- currently YBD aims to continue building old definitions versions,
could be extended to support the scheme you're proposing (as well as
Daniel's V9 and V10 proposals) but at some point the complexity of
handling all the versions may not make sense.
- we 'lock' a stable version of YBD for all versions prior to
and then move on with a migration and new forward-only tooling?
I think this is somewhat orthogonal to the splitting up of definitions.
It's possible that YBD needs to support relative paths (if it does not
already) to support, for example, build-depends references to:
So as is, I don't think this should really constitute an API change in
YBD, at least not one which would be backwards incompatible.
However to answer that question, with the new changes in V9 and V10 it
may be a desirable thing to drop back compat, if it happens to be an
ideal moment for that, and if there is no critical user base depending
on support for the older formats.
Practically speaking: What is the cost of migrating existing
definitions that are maintained and useful to someone, vs the cost of
maintaining a tool that is backwards compatible down to V1 ? And, is
that cost entirely a cost of our own ? Or is it a cost we impose on
If we do both refactor YBD and drop support for < V10 definitions for
some reason, then that refactor should come with preparation for being
forever backwards compatible in the future down to V10.