Forgot to send this email to baserock-dev.
-------- Forwarded Message --------
Subject: Re: Proposal: Axing the Stratum - Enter Runtime Dependencies
and Build Flavors
Date: Thu, 5 Nov 2015 18:30:46 +0000
From: Tiago Gomes <tiago.gomes(a)codethink.co.uk>
To: Tristan Van Berkom <tristan.vanberkom(a)codethink.co.uk>
On 05/11/15 07:10, Tristan Van Berkom wrote:
I did receive your mail early enough, but needed some time to process
this one :)
On Wed, 2015-11-04 at 11:07 +0000, Tiago Gomes wrote:
> On 03/11/15 08:35, Tristan Van Berkom wrote:
>> On Mon, 2015-11-02 at 16:11 +0000, Tiago Gomes wrote:
>>> Thanks for this interesting read.
>> And thank you for reading it :)
>>> On 02/11/15 04:07, Tristan Van Berkom wrote:
>>>> 2 Proposed Model: Runtime Dependencies and Flavors
>>>> Simply put, the proposed model is to build the entire dependency graph
>>>> by defining each and every chunk separately, where each chunk declares
>>>> it's direct dependencies explicitly.
>>>> One of the goals here is to ensure that there is only ever a single
>>>> chunk definition for a given upstream git module, ever.
>>>> Instead of defining duplicate chunks for various compilation flavors,
>>>> depending on their optional dependencies of configuration/build
>>>> commands, we would encode these as semantics of the chunk definition
>>>> For convenience, we would keep the concept of "stratum" around,
>>>> they would not conceptually "contain" chunks. Strata would
>>>> to existing chunks and strata as dependencies, as such they would be
>>>> allowed to overlap in such a way that a system may include two strata
>>>> which refer to some of the same chunks, but the build mechanics would
>>>> revolve around building an entire dependency tree of chunks, while
>>>> strata would only define which chunks and strata come together as
>>>> logical groups.
>>> It is conceptually simpler to me to think about stratum as a collection
>>> of chunks. If you change the meaning of strata to be a dependency list
>>> for some other chunk or stratum, it doesn't seem logical to me that when
>>> a system includes a stratum, the list of chunks and strata that the
>>> stratum depends on, will be part of the system.
>> In the current situation, it is possible to add a stratum requirement to
>> a stratum, and then forget to include the arguably redundant stratum in
>> the system definition.
>> Is there a purpose to this ? Are we interested in allowing strata to
>> include other strata/chunks which are not supposed to end up in the
>> system build ?
> I can't say if this is a feature or lack of validation in our
> morphologies :)
>> I think we are arguing for a simple matter of taste in this instance, a
>> build tool should always be able to print a build graph or list of what
>> will be included in the target system by following the dependency chain.
>> Is there any practical value to forcing the explicit listing of all
>> chunks which are to be included in a given system, over just using a
>> command line utility to list those chunks if one were interested to
>> know ?
> That's a good question. Maybe we should allow the system morphologies
> to include both strata and chunks. That, together with using flavors
> for archs, would allows us to get rid of some "leaf" strata.
I dont think excluding entire chunks from system assembly, which were
depended on in the build process is the best way to address system
Not to say that there is no need for reducing what we add to our
systems; just that I think I much prefer a hybrid solution along the
lines of Richard Maw's runtime dependency proposal, or, any solution
which addresses the distribution problem at the artifact level rather
than the chunk level.
>>> We could change our systems to include chunks instead of including
>>> strata that would make our systems very large.
>> Your statement, if I am reading it correctly, is that a build-time
>> dependency is required to run in order to build the depending chunk.
>> This is untrue in virtually every case that I can think of. It is only
>> true for the toolchain itself, in which case these build tool chunk
>> dependencies should simply be classified as build dependencies.
> That's not the case for the toolchain only. As an example, sphinx needs
> to run in order to build the documentation for some Python packages. If
> the libraries that sphinx depends on weren't in the staging area, the
> making of the documentation would fail.
This is a good point, it *is* still toolchain strictly speaking, as it
constitutes tooling which is required to run during the build process.
> You could argue that we could put the libraries that sphinx depends on
> its build dependencies, but then we would be loosing the advantages of
> having run-depends.
Yes; as much as we would lose the advantages of run-depends at the
toolchain level, we lose it for everything that we consider to be build
tools, which includes the building of documentation.
A general statement that "we would be losing the advantages of having
run-depends" however is a huge exaggeration, for a few reasons:
o I'm still convinced that even including documentation builders as
"toolchain" we are talking about a very small percentage of a full
o Just because a given tool build-depends on a given module
does not preclude other chunks which may run-depend on the same
module, without build-depending on the said tool.
o Realistically, not all modules/chunks are created equally. A build
time dependency on boost or WebKit is immensely more expensive than
a build dependency on some installed DTDs or metadata, or the
installation of a python script.
Which is to say that, yes, the proposed solution to runtime dependencies
does not save us a rebuild for 100% of the possible cases it could, but
it does save us a rebuild for at least 90% of cases, and more
importantly; it does so with sufficient simplicity.
Still, the question of how many chunks exactly is required to build the
toolchain is enough to give some pause. Certainly it remains well below
10% of a full system build, but lets indulge this thought a little
I was just stating that there wouldn't be advantages of having
runtime-depends for this particular case. Like you said below, it
wouldn't achieve 100% efficiency.
TBH, I don't care much about how runtime-dependencies would be handled.
As Richard says :
We can either include runtime dependencies of artifacts we
on, or leave runtime dependencies to being purely for system artifact
There's arguments for either.
It makes some amount of sense for runtime dependencies of an artifact
you build-depend on to be included in the staging area, as it saves
having to define these build-only-runtime-depends as build-depends,
which would cause unnecessary build-time dependencies for python modules.
However, if you don't include this, then you only need to rebuild the
system artifact when build-dependencies change, which has some appeal.
The important thing is that you don't depend on _your_ runtime
dependencies at build-time, but people who depend on you end up depending
on _your_ runtime dependencies.
This is sufficiently odd that it bakes my noodle a little.
My feeling is that build time and runtime dependency declarations at
chunk level is the right balance of efficiency vs complexity.
Further elaboration in the definitions to declare these dependencies at
the artifact level instead of the chunk level would bring us closer to
100% efficiency (without making the reality of circular dependencies
disappear) - however that 100% efficiency comes at an unreasonably high
cost of complexity on all sides. Implementors would be left with a much
more complex format to digest, and more importantly, users would be left
with a much more complex and error prone definitions format.
Does this sound reasonable to you ?
Why do you think that it would be more complex for the user? The
definitions wouldn't differ from the model that you proposed, it is just
how the runtime depends on a staging area are handled. Again, I am not
advocating one way or another.