1:05 a.m.
We currently handle definitions that need to pull multiple source
repositories together in an ad-hoc way.
For gcc we import the gmp, mpc and mpfr source trees in by checking them
into our delta branch directly.
Some upstreams handle importing the sources by adding submodules.
However, we have to make them fetch sources from our Trove for
reproducibilty and latency concerns, so we end up having to add a delta
on top to change where the submodules fetch their sources from.
This works against one of our goals for minimising the delta, so we
need a way to encode in our builds how to deal with components that need
sources from multiple repositories to function.
Our current approach from submodules introduces a _lot_ of extra work
when we need to update multiple submodules recursively, so we need a
better solution.
Proposal
========
To solve this, I propose we extend the source repository information from
just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
So rather than:
name: base
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
unpetrify-ref: baserock/morph
⋮
build-depends: []
We extend it to be able to take a "sources" field.
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
sources:
- repo: upstream:foo
ref: baserock/morph # used to validate commit is anchored
# properly and as a hint for where
# you ought to merge changes to
commit: cafef00d…
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The `path` field is used to specify that this tree should be placed in
the parent tree at this position.
The `path` field is optional, defaulting to `.`
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Sub-sources can be placed over an empty directory, when there's no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
If there's a file, symlink, non-empty directory or submodule that doesn't
match `submodule-commit` there, then it results in a build-failure
at build-time.
The `submodule-commit` check exists as a safeguard against the parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
If you get a build fail because the submodule isn't appropriate then
you have to check whether the version you specify works, then update
the commit.
Cache key changes
=================
This shouldn't require any cache key changes to existing definitions,
but builds that make use of multiple source repositories will also hash
the commit tree and path.
Alternative solutions for submodules
====================================
We could continue to use the current model, and deal with the pain of
having to make multiple branches in multiple repositories to satisfy
the change to the repository paths.
We could have a magic look-up table to replace repository urls when we
parse the submodules.
We could use git-replace to magically make git do the url correction, but
then we'd have to handle replacements properly, rather than dropping them.
7:55 p.m.
On Wed, Nov 12, 2014 at 16:05:55 +0000, Richard Maw wrote:
We extend it to be able to take a "sources" field.
[snip]
The `path` field is optional, defaulting to `.`
And presumably will be a failure at morphology parse time if it tries
to be above the root of the checkout (e.g. ../../dev/blah)
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Given it's at source creation time, I assume the normalisation algorithm will
involve collapsing sequential /s and then collapsing ..s in some manner?
Sub-sources can be placed over an empty directory, when there's
no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
Since git cannot have empty directories present in the tree, I suspect only the
no path or the submodule option are the relevant ones here.
The `submodule-commit` check exists as a safeguard against the
parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
So the idea of submodule-commit is that it has to match the commit in the git
tree you're cloning on top of, but the commit is what you actually check out?
[snip]
In addition, you're going to need to define the topological sort mechanism
needed to ensure that you check out the multiple sources in the right order.
Cache key changes
=================
This shouldn't require any cache key changes to existing definitions,
but builds that make use of multiple source repositories will also hash
the commit tree and path.
I'd be tempted to introduce a cache key change which causes the current
morphology layout to produce a cache key along the lines of a single source
entry whose path is '.' sooner rather than later. However if you're not
happy
with that, then I'm okay for changes to only occur when we have multi-source
chunks in.
Alternative solutions for submodules
====================================
We could continue to use the current model, and deal with the pain of
having to make multiple branches in multiple repositories to satisfy
the change to the repository paths.
We could have a magic look-up table to replace repository urls when we
parse the submodules.
We could use git-replace to magically make git do the url correction, but
then we'd have to handle replacements properly, rather than dropping them.
None of these are delightful :-( I vote +1 to your concept for multi-source
chunks.
--
Daniel Silverstone http://www.codethink.co.uk/
Software Engineer GPG 4096/R Key Id: 3CCE BABE 206C 3B69
12:12 a.m.
On Fri, Nov 21, 2014 at 10:55:58AM +0000, Daniel Silverstone wrote:
On Wed, Nov 12, 2014 at 16:05:55 +0000, Richard Maw wrote:
> We extend it to be able to take a "sources" field.
[snip]
> The `path` field is optional, defaulting to `.`
And presumably will be a failure at morphology parse time if it tries
to be above the root of the checkout (e.g. ../../dev/blah)
Yep.
> If multiple paths, after normalising, clash, then it results in
a
> build-failure at source creation time.
Given it's at source creation time, I assume the normalisation algorithm will
involve collapsing sequential /s and then collapsing ..s in some manner?
I was thinking of just using os.path.normpath and asserting it's a
relative path.
> Sub-sources can be placed over an empty directory, when
there's no
> existing entry at that path, or when there's a git submodule there AND
> the commit of the submodule matches the `submodule-commit` field.
Since git cannot have empty directories present in the tree, I suspect only the
no path or the submodule option are the relevant ones here.
Excellent point.
> The `submodule-commit` check exists as a safeguard against the
parent
> repository being updated and requiring a new version, which your specified
> commit isn't appropriate for.
So the idea of submodule-commit is that it has to match the commit in the git
tree you're cloning on top of, but the commit is what you actually check out?
Yes, so you're able to select a delta instead if you need it.
[snip]
In addition, you're going to need to define the topological sort mechanism
needed to ensure that you check out the multiple sources in the right order.
Either that, or re-work the definition structure such that the nested
structure is expressed in the data.
> Cache key changes
> =================
>
> This shouldn't require any cache key changes to existing definitions,
> but builds that make use of multiple source repositories will also hash
> the commit tree and path.
I'd be tempted to introduce a cache key change which causes the current
morphology layout to produce a cache key along the lines of a single source
entry whose path is '.' sooner rather than later. However if you're not
happy
with that, then I'm okay for changes to only occur when we have multi-source
chunks in.
I guess the useful metric about whether it's worth keeping the cache
keys the same is how much extra code it is to keep the existing stuff
producing the same keys.
> Alternative solutions for submodules
> ====================================
>
> We could continue to use the current model, and deal with the pain of
> having to make multiple branches in multiple repositories to satisfy
> the change to the repository paths.
>
> We could have a magic look-up table to replace repository urls when we
> parse the submodules.
>
> We could use git-replace to magically make git do the url correction, but
> then we'd have to handle replacements properly, rather than dropping them.
None of these are delightful :-( I vote +1 to your concept for multi-source
chunks.
There's a risk that I've produced a false dichotomy, in saying that
these are all our options, so the alternatives I've listed being terrible
aren't sufficient proof that we must do my proposed implementation to
allow multi-source chunks.
I'd be delighted if I've overlooked a better solution.
12:56 a.m.
On Fri, Nov 21, 2014 at 15:12:49 +0000, Richard Maw wrote:
> Given it's at source creation time, I assume the
normalisation algorithm will
> involve collapsing sequential /s and then collapsing ..s in some manner?
I was thinking of just using os.path.normpath and asserting it's a
relative path.
I'll reserve judgement on that until I've seen how normpath behaves, I'll
look
it up when I have a moment.
> In addition, you're going to need to define the topological
sort mechanism
> needed to ensure that you check out the multiple sources in the right order.
Either that, or re-work the definition structure such that the nested
structure is expressed in the data.
Hmm, can you produce a side-by-side of the two options so we can decide if the
config format is worth the code. (either way)
I guess the useful metric about whether it's worth keeping the
cache
keys the same is how much extra code it is to keep the existing stuff
producing the same keys.
And the new stuff producing equivalent keys, surely? Since you don't want the
conversion of the definitions to the new format to result in new keys if you're
wanting the old format to continue to produce the old keys.
> > Alternative solutions for submodules
[snip]
> None of these are delightful :-( I vote +1 to your concept for
multi-source
> chunks.
There's a risk that I've produced a false dichotomy, in saying that
these are all our options, so the alternatives I've listed being terrible
aren't sufficient proof that we must do my proposed implementation to
allow multi-source chunks.
I'd be delighted if I've overlooked a better solution.
If I'd have been able to think of any better options, I'd have suggested them
instead of agreeing to your proposal :-)
D.
--
Daniel Silverstone http://www.codethink.co.uk/
Software Engineer GPG 4096/R Key Id: 3CCE BABE 206C 3B69
1:20 a.m.
On Fri, Nov 21, 2014 at 15:56:15 +0000, Daniel Silverstone wrote:
I'll reserve judgement on that until I've seen how normpath
behaves, I'll look
it up when I have a moment.
I've now managed to review normpath -- I think the approach is acceptable.
D.
--
Daniel Silverstone http://www.codethink.co.uk/
Software Engineer GPG 4096/R Key Id: 3CCE BABE 206C 3B69
2:53 a.m.
Hi,
On 12/11/14 16:05, Richard Maw wrote:
We currently handle definitions that need to pull multiple source
repositories together in an ad-hoc way.
For gcc we import the gmp, mpc and mpfr source trees in by checking them
into our delta branch directly.
Some upstreams handle importing the sources by adding submodules.
However, we have to make them fetch sources from our Trove for
reproducibilty and latency concerns, so we end up having to add a delta
on top to change where the submodules fetch their sources from.
This works against one of our goals for minimising the delta, so we
need a way to encode in our builds how to deal with components that need
sources from multiple repositories to function.
Our current approach from submodules introduces a _lot_ of extra work
when we need to update multiple submodules recursively, so we need a
better solution.
Proposal
========
To solve this, I propose we extend the source repository information from
just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
So rather than:
name: base
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
unpetrify-ref: baserock/morph
⋮
build-depends: []
We extend it to be able to take a "sources" field.
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
sources:
- repo: upstream:foo
ref: baserock/morph # used to validate commit is anchored
# properly and as a hint for where
# you ought to merge changes to
commit: cafef00d…
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The `path` field is used to specify that this tree should be placed in
the parent tree at this position.
The `path` field is optional, defaulting to `.`
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Sub-sources can be placed over an empty directory, when there's no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
If there's a file, symlink, non-empty directory or submodule that doesn't
match `submodule-commit` there, then it results in a build-failure
at build-time.
The `submodule-commit` check exists as a safeguard against the parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
If you get a build fail because the submodule isn't appropriate then
you have to check whether the version you specify works, then update
the commit.
I like the approach, but as only a few modules have submodules, I'd
prefer the following layout:
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
submodules:
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
Or
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
submodules:
gnulib
repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The names of the fields can be tweaked. This layout would also make
it easier to convert the existing definitions.
<snip>
Tiago
11:54 p.m.
On 06/11/15 17:53, Tiago Gomes wrote:
Hi,
On 12/11/14 16:05, Richard Maw wrote:
> We currently handle definitions that need to pull multiple source
> repositories together in an ad-hoc way.
>
> For gcc we import the gmp, mpc and mpfr source trees in by checking them
> into our delta branch directly.
>
> Some upstreams handle importing the sources by adding submodules.
>
> However, we have to make them fetch sources from our Trove for
> reproducibilty and latency concerns, so we end up having to add a delta
> on top to change where the submodules fetch their sources from.
>
> This works against one of our goals for minimising the delta, so we
> need a way to encode in our builds how to deal with components that need
> sources from multiple repositories to function.
>
> Our current approach from submodules introduces a _lot_ of extra work
> when we need to update multiple submodules recursively, so we need a
> better solution.
>
>
> Proposal
> ========
>
> To solve this, I propose we extend the source repository information from
> just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
>
> So rather than:
>
> name: base
> ⋮
> chunks:
> - name: foo
> morph: strata/base/foo.morph
> repo: upstream:foo
> ref: cafef00d…
> unpetrify-ref: baserock/morph
> ⋮
> build-depends: []
>
> We extend it to be able to take a "sources" field.
>
> ⋮
> chunks:
> - name: foo
> morph: strata/base/foo.morph
> sources:
> - repo: upstream:foo
> ref: baserock/morph # used to validate commit is
> anchored
> # properly and as a hint for where
> # you ought to merge changes to
> commit: cafef00d…
> - repo: delta:gnulib
> commit: deadbeef…
> ref: baserock/GNULIB_0_1_2
> path: .gnulib # where to put the cloned source
> submodule-commit: feedbeef… # check that this matches, so you
> # can be told if you need to update
> # your delta
> ⋮
>
>
> The `path` field is used to specify that this tree should be placed in
> the parent tree at this position.
>
> The `path` field is optional, defaulting to `.`
>
> If multiple paths, after normalising, clash, then it results in a
> build-failure at source creation time.
>
> Sub-sources can be placed over an empty directory, when there's no
> existing entry at that path, or when there's a git submodule there AND
> the commit of the submodule matches the `submodule-commit` field.
>
> If there's a file, symlink, non-empty directory or submodule that doesn't
> match `submodule-commit` there, then it results in a build-failure
> at build-time.
>
> The `submodule-commit` check exists as a safeguard against the parent
> repository being updated and requiring a new version, which your
> specified
> commit isn't appropriate for.
>
> If you get a build fail because the submodule isn't appropriate then
> you have to check whether the version you specify works, then update
> the commit.
I like the approach, but as only a few modules have submodules, I'd
prefer the following layout:
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
submodules:
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
Or
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
submodules:
gnulib
repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The names of the fields can be tweaked. This layout would also make
it easier to convert the existing definitions.
Hi,
After pondering about this a bit more, I suggest a new layout:
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
extra-sources:
- repo: delta:repo1
path: repo1_sources
- repo: delta:repo2
ref: da2es32…
extra_sources:
- repo: delta:repo3
ref: qw24s6r…
path: repo3_extra_sources
The field 'ref' will be now optional. If not given, the build tool
will do a `git ls-tree` on 'path' to find the submodule commit sha1
anchored by the parent repo.
This avoids the hassle of requiring the user to do a `git ls-tree` by
themselves when writing the definitions, and makes it possible to
override the submodule commit used without needing the former
'submodule-commit' field described by Richard.
'ref' will be a mandatory field if the extra source is not a submodule
of the parent repo.
Recursive submodules will handled by specifying recursive extra_sources.
Thoughts?
Tiago
12:26 a.m.
New subject: [RFC] Multiple source repositories per-chunk and git submodule handling
On 2015-11-17 14:54, Tiago Gomes wrote:
After pondering about this a bit more, I suggest a new layout:
Why not...
⋮
chunks:
contents:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
extra-sources:
contents:
- repo: delta:repo1
path: repo1_sources
- repo: delta:repo2
ref: da2es32…
extra_sources:
contents:
- repo: delta:repo3
ref: qw24s6r…
path: repo3_extra_sources
The field 'ref' will be now optional. If not given, the build tool
will do a `git ls-tree` on 'path' to find the submodule commit sha1
anchored by the parent repo.
This avoids the hassle of requiring the user to do a `git ls-tree`
by themselves when writing the definitions, and makes it possible to
override the submodule commit used without needing the former
'submodule-commit' field described by Richard.
'ref' will be a mandatory field if the extra source is not a
submodule of the parent repo.
Recursive submodules will handled by specifying recursive
extra_sources.
Thoughts?
Tiago
_______________________________________________
baserock-dev mailing list
baserock-dev(a)baserock.org
http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/baserock-dev-ba...
12:40 a.m.
On 17/11/15 15:26, Paul Sherwood wrote:
On 2015-11-17 14:54, Tiago Gomes wrote:
> After pondering about this a bit more, I suggest a new layout:
Why not...
>
> ⋮
> chunks:
contents:
> - name: foo
> morph: strata/base/foo.morph
> repo: upstream:foo
> ref: cafef00d…
> extra-sources:
contents:
> - repo: delta:repo1
> path: repo1_sources
> - repo: delta:repo2
> ref: da2es32…
> extra_sources:
contents:
> - repo: delta:repo3
> ref: qw24s6r…
> path: repo3_extra_sources
<snip>
I don't disagree :) But I think that is part of the "rethink the
chunks/strata/systems terminology" plan and orthogonal with what I am
trying to fix.
Tiago
1:32 a.m.
On Tue, Nov 17, 2015 at 02:54:24PM +0000, Tiago Gomes wrote:
Hi,
After pondering about this a bit more, I suggest a new layout:
Sorry it took so long for me to reply.
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
extra-sources:
I pondered this approach when I put together the proposal.
My justification for not doing it this way is that I wanted to be able to have
no top-level source, or multiple top-level sources.
Allowing no top-level sources would be handy for builds which are for combining
the results of installed components, which would be a way to handle
gdk-pixbuf-loaders or ld.so.cache file generation.
Allowing multiple top-level sources would be handy for building incestuously
linked projects, or projects that are difficult to bootstrap independently,
such as gcc and its support libraries, or clang and llvm.
- repo: delta:repo1
path: repo1_sources
- repo: delta:repo2
ref: da2es32…
Is the path intentionally missing here?
extra_sources:
- repo: delta:repo3
ref: qw24s6r…
path: repo3_extra_sources
The field 'ref' will be now optional. If not given, the build tool
will do a `git ls-tree` on 'path' to find the submodule commit sha1
anchored by the parent repo.
The justification for it being mandatory was to reduce the number of operations
that you need to have the chunks' source trees available for, so that we can
rid ourselves of the code required to fetch the sources in.
This avoids the hassle of requiring the user to do a `git ls-tree`
by themselves when writing the definitions, and makes it possible to
override the submodule commit used without needing the former
'submodule-commit' field described by Richard.
The purpose of having both the ref and the submodule-commit to specify both the
check and the override, was so that when you update the ref of the parent
repository and that happens to update the submodule, the tool forces you to
verify whether you still want to override the submodule.
'ref' will be a mandatory field if the extra source is not a
submodule of the parent repo.
Recursive submodules will handled by specifying recursive extra_sources.
Isn't this redundant when you are already providing the path?
You can determine the order in which sources need to be extracted by which
paths contain each other.
I considered a nested sources structure when I wrote my proposal,
but decided since I needed to put some form of path in anyway,
and the dependency could be derived from that,
that I'd prefer to keep the structure flat.
Of course, since you're encoding the sources in a list,
you could require that sources are made available in that order.
Thoughts?
Please don't take my feedback as criticism of the form
"I already thought of that and here's why you're wrong".
This is not my intention, I really did consider those options,
and those are the reasons why I proposed my design as it was.
It looks like you are the one who intends to implement it,
and I'm glad of that.
If you think your way is better that's fine,
I just wanted to make sure that you were aware of why my proposal differed.
2:05 a.m.
On Thu, 2015-11-19 at 16:32 +0000, Richard Maw wrote:
On Tue, Nov 17, 2015 at 02:54:24PM +0000, Tiago Gomes wrote:
> Hi,
>
> After pondering about this a bit more, I suggest a new layout:
Sorry it took so long for me to reply.
> ⋮
> chunks:
> - name: foo
> morph: strata/base/foo.morph
> repo: upstream:foo
> ref: cafef00d…
> extra-sources:
I pondered this approach when I put together the proposal.
My justification for not doing it this way is that I wanted to be able to have
no top-level source, or multiple top-level sources.
Allowing no top-level sources would be handy for builds which are for combining
the results of installed components, which would be a way to handle
gdk-pixbuf-loaders or ld.so.cache file generation.
Allowing multiple top-level sources would be handy for building incestuously
linked projects, or projects that are difficult to bootstrap independently,
such as gcc and its support libraries, or clang and llvm.
+1 to something handy for building incestuously linked projects. I've
had difficulty working out how to make navit-based projects build due to
navit's recommendation of building navit modules inside navit's source
tree.
1:15 a.m.
On 19/11/15 16:32, Richard Maw wrote:
On Tue, Nov 17, 2015 at 02:54:24PM +0000, Tiago Gomes wrote:
> Hi,
>
> After pondering about this a bit more, I suggest a new layout:
Sorry it took so long for me to reply.
No worries. Thanks for providing feedback.
> ⋮
> chunks:
> - name: foo
> morph: strata/base/foo.morph
> repo: upstream:foo
> ref: cafef00d…
> extra-sources:
I pondered this approach when I put together the proposal.
My justification for not doing it this way is that I wanted to be able to have
no top-level source, or multiple top-level sources.
Allowing no top-level sources would be handy for builds which are for combining
the results of installed components, which would be a way to handle
gdk-pixbuf-loaders or ld.so.cache file generation.
Allowing multiple top-level sources would be handy for building incestuously
linked projects, or projects that are difficult to bootstrap independently,
such as gcc and its support libraries, or clang and llvm.
I am not a fan of having multiple top-levels. That would involve
putting all the chunk repo and ref fields in a list, despite most of the
chunks only having a single source.
Also, if there were multiple top-level sources, from which directory
would the build commands run if there isn't a parent source directory?
I don't know about gdk-pixbuf-loaders, but gcc supports specifying the
location of the support libraries at configure time, so you could clone
them to the gcc dir and use them from there.
Clang build instructions also have it building inside a parent source
directory (llvm) [1].
That said, if there are really projects which require a set of sources
cloned side-by-side, a way of dealing with this would be supporting a
`path` field for the parent source as well. So that
chunks:
- name: foo
repo: delta:foo
path: foo
extra-sources:
- repo: delta:bar
path: bar
- repo: delta:qux
path: qux
would produce the following directory structure in staging area:
foo.build/
├── bar/
├── foo/
└── qux/
> - repo: delta:repo1
> path: repo1_sources
> - repo: delta:repo2
> ref: da2es32…
Is the path intentionally missing here?
No, sorry, I forgot to put it there.
> extra_sources:
> - repo: delta:repo3
> ref: qw24s6r…
> path: repo3_extra_sources
>
>
> The field 'ref' will be now optional. If not given, the build tool
> will do a `git ls-tree` on 'path' to find the submodule commit sha1
> anchored by the parent repo.
The justification for it being mandatory was to reduce the number of operations
that you need to have the chunks' source trees available for, so that we can
rid ourselves of the code required to fetch the sources in.
I am not sure which code you are talking about. AFAIK, if there is a
configured morph-cache-server, you only need the sources if you need
build them.
If morph-cache-server API grows to support getting the submodule commit
given a certain path, there would be no need to fetch the sources.
> This avoids the hassle of requiring the user to do a `git ls-tree`
> by themselves when writing the definitions, and makes it possible to
> override the submodule commit used without needing the former
> 'submodule-commit' field described by Richard.
The purpose of having both the ref and the submodule-commit to specify both the
check and the override, was so that when you update the ref of the parent
repository and that happens to update the submodule, the tool forces you to
verify whether you still want to override the submodule.
I understood that. With my suggestion there is no need for that extra
field, as if the user specified a ref for the submodule, that will be
the ref used independent of the parent ref given.
> 'ref' will be a mandatory field if the extra source is not a
> submodule of the parent repo.
>
> Recursive submodules will handled by specifying recursive extra_sources.
Isn't this redundant when you are already providing the path?
The idea of using recursive sources is to make it explicit which one is
the parent repo for getting the submodule commit id. The alternative
would be some kind of path detection.
chunks:
- name: foo:
- repo: delta:foo
extra-sources:
- repo: bar
path: bar
- repo: qux
path: bar/apath/qux
For example, let's suppose that qux is a submodule of bar. I could
perhaps find the parent repo for qux by checking which extra-source has
a path that shares the most initial path components.
You can determine the order in which sources need to be extracted by which
paths contain each other.
I considered a nested sources structure when I wrote my proposal,
but decided since I needed to put some form of path in anyway,
and the dependency could be derived from that,
that I'd prefer to keep the structure flat.
Of course, since you're encoding the sources in a list,
you could require that sources are made available in that order.
That's how I thought it would work and why is a list. The user would be
responsible for providing a correct order.
> Thoughts?
Please don't take my feedback as criticism of the form
"I already thought of that and here's why you're wrong".
This is not my intention, I really did consider those options,
and those are the reasons why I proposed my design as it was.
I don't. Every design has its strengths and weaknesses. But it would
be nice to find a compromise.
It looks like you are the one who intends to implement it,
and I'm glad of that.
If you think your way is better that's fine,
I just wanted to make sure that you were aware of why my proposal differed.
There is a WIP implementation at [2]. I am happy to abandon the branch
if my changes are considered problematic. But I would like to know what
other people think.
T.
[1]: http://clang.llvm.org/get_started.html
[2]:
http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/log/...
2:48 a.m.
On Fri, Nov 20, 2015 at 04:15:17PM +0000, Tiago Gomes wrote:
On 19/11/15 16:32, Richard Maw wrote:
>On Tue, Nov 17, 2015 at 02:54:24PM +0000, Tiago Gomes wrote:
>>Hi,
>>
>>After pondering about this a bit more, I suggest a new layout:
>
>Sorry it took so long for me to reply.
No worries. Thanks for providing feedback.
>
>>⋮
>>chunks:
>>- name: foo
>> morph: strata/base/foo.morph
>> repo: upstream:foo
>> ref: cafef00d…
>> extra-sources:
>
>I pondered this approach when I put together the proposal.
>
>My justification for not doing it this way is that I wanted to be able to have
>no top-level source, or multiple top-level sources.
>
>Allowing no top-level sources would be handy for builds which are for combining
>the results of installed components, which would be a way to handle
>gdk-pixbuf-loaders or ld.so.cache file generation.
>
>Allowing multiple top-level sources would be handy for building incestuously
>linked projects, or projects that are difficult to bootstrap independently,
>such as gcc and its support libraries, or clang and llvm.
I am not a fan of having multiple top-levels. That would involve
putting all the chunk repo and ref fields in a list, despite most of
the chunks only having a single source.
It's one extra line and a small amount of indentation,
but you could keep the old syntax when you have just 1 source,
and decide how to parse it based on the presence of a top-level sources key.
I'd just rather avoid stuffing more stuff into the top-level format,
when it might make more sense to take a step back.
Also, if there were multiple top-level sources, from which directory
would the build commands run if there isn't a parent source
directory?
You'd still run from /foo.build, but for multiple top-level sources you'd
specify a path for all of them, and have a tree looking like you do below.
I don't know about gdk-pixbuf-loaders, but gcc supports
specifying
the location of the support libraries at configure time, so you
could clone them to the gcc dir and use them from there.
Clang build instructions also have it building inside a parent
source directory (llvm) [1].
That assumes that the build system deals with extra content being present
nicely, you could have one which decides that it's not a "clean" build so
tags
it differently, and then people get concerned that there may be local patches
which aren't tracked.
I have independently seen projects put an unclean marker in their version
string if there is additional content in the repository,
and I have seen people get concerned about the marker,
so it's not a theoretical concern.
That said, if there are really projects which require a set of
sources cloned side-by-side, a way of dealing with this would be
supporting a `path` field for the parent source as well. So that
chunks:
- name: foo
repo: delta:foo
path: foo
extra-sources:
- repo: delta:bar
path: bar
- repo: delta:qux
path: qux
I'd assert that this looks more consistent and is the same length:
chunks:
- name: foo
sources:
- repo: delta:foo
path: foo
- repo: delta:bar
path: bar
- repo: delta:qux
path: qux
would produce the following directory structure in staging area:
foo.build/
├── bar/
├── foo/
└── qux/
This looks good to me.
<snip>
>The justification for it being mandatory was to reduce the number
of operations
>that you need to have the chunks' source trees available for, so that we can
>rid ourselves of the code required to fetch the sources in.
I am not sure which code you are talking about. AFAIK, if there is
a configured morph-cache-server, you only need the sources if you
need build them.
Or if the morph-cache-server is unreachable.
If morph-cache-server API grows to support getting the submodule
commit given a certain path, there would be no need to fetch the
sources.
This still requires a round-trip to the morph-cache-server and back,
latency can be more important than the amount of data transferred,
and if you can't know in advance whether you need to check with the remote server,
then you have to check every time which makes things slow
and unable to make it work offline.
>>This avoids the hassle of requiring the user to do a `git
ls-tree`
>>by themselves when writing the definitions, and makes it possible to
>>override the submodule commit used without needing the former
>>'submodule-commit' field described by Richard.
>
>The purpose of having both the ref and the submodule-commit to specify both the
>check and the override, was so that when you update the ref of the parent
>repository and that happens to update the submodule, the tool forces you to
>verify whether you still want to override the submodule.
I understood that. With my suggestion there is no need for that
extra field, as if the user specified a ref for the submodule, that
will be the ref used independent of the parent ref given.
I can see that,
but I still think the consistency check has value,
and an extra field is required if you have the consistency check.
>>'ref' will be a mandatory field if the extra source
is not a
>>submodule of the parent repo.
>>
>>Recursive submodules will handled by specifying recursive extra_sources.
>
>Isn't this redundant when you are already providing the path?
The idea of using recursive sources is to make it explicit which one
is the parent repo for getting the submodule commit id. The
alternative would be some kind of path detection.
chunks:
- name: foo:
- repo: delta:foo
extra-sources:
- repo: bar
path: bar
- repo: qux
path: bar/apath/qux
For example, let's suppose that qux is a submodule of bar. I could
perhaps find the parent repo for qux by checking which extra-source
has a path that shares the most initial path components.
For what it's worth,
I'd prefer it to use the path rather than nesting the data structures.
>You can determine the order in which sources need to be extracted
by which
>paths contain each other.
>
>I considered a nested sources structure when I wrote my proposal,
>but decided since I needed to put some form of path in anyway,
>and the dependency could be derived from that,
>that I'd prefer to keep the structure flat.
>
>Of course, since you're encoding the sources in a list,
>you could require that sources are made available in that order.
That's how I thought it would work and why is a list. The user
would be responsible for providing a correct order.
I'm struggling to think of a case where topologically sorting based on
the path would be wrong and a manual ordering right, but that doesn't
mean there isn't one.
Potentially you could make source preparation faster if you could unpack
in topology order rather than in listed order, as there is some level
of parallelism possible.
I think it unlikely that we'd need that level of optimisation,
but I'm wary of selecting an implementation that would prevent it.
>>Thoughts?
<snip>
>It looks like you are the one who intends to implement it,
>and I'm glad of that.
>
>If you think your way is better that's fine,
>I just wanted to make sure that you were aware of why my proposal differed.
There is a WIP implementation at [2]. I am happy to abandon the
branch if my changes are considered problematic. But I would like
to know what other people think.
T.
[1]: http://clang.llvm.org/get_started.html
[2]:
http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/log/...
I'll make a note to look at it when I have time.
2:29 a.m.
On Fri, Nov 20, 2015 at 05:48:02PM +0000, Richard Maw wrote:
On Fri, Nov 20, 2015 at 04:15:17PM +0000, Tiago Gomes wrote:
>
>
> On 19/11/15 16:32, Richard Maw wrote:
> >On Tue, Nov 17, 2015 at 02:54:24PM +0000, Tiago Gomes wrote:
<snip>
<unsnip the following quote from Tiago for context>
Tiago: "The field 'ref' will be now optional. If not given, the build tool
will do a `git ls-tree` on 'path' to find the submodule commit sha1
anchored by the parent repo."
> >The justification for it being mandatory was to reduce the
number of operations
> >that you need to have the chunks' source trees available for, so that we
can
> >rid ourselves of the code required to fetch the sources in.
>
> I am not sure which code you are talking about. AFAIK, if there is
> a configured morph-cache-server, you only need the sources if you
> need build them.
Or if the morph-cache-server is unreachable.
> If morph-cache-server API grows to support getting the submodule
> commit given a certain path, there would be no need to fetch the
> sources.
This still requires a round-trip to the morph-cache-server and back,
latency can be more important than the amount of data transferred,
and if you can't know in advance whether you need to check with the remote server,
then you have to check every time which makes things slow
and unable to make it work offline.
This sort of optional parameter carried performance implications in the past.
You'll remember we switched from optional declaration of the build-system
to mandatory declaration of the build-system to avoid needing to fetch
sources in order to figure out the build-system.
The simplest way to avoid these sorts of performances hits is to ensure
that we keep as much information as possible in the definitions repo.
hth
Richard Ipsum
9:38 p.m.
On Fri, Nov 20, 2015 at 04:15:17PM +0000, Tiago Gomes wrote:
There is a WIP implementation at [2]. I am happy to abandon the
branch if my changes are considered problematic. But I would like
to know what other people think.
[2]:
http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/log/...
"morphloader: remove duplicated API"
The purpose of having both was to have a variant that didn't do the validation,
which was required in some circumstances.
I don't know whether we still have those circumstances,
if you know we don't then I'm happy for that patch to be applied.
"morphloader: invert repo/name logic"
Yes, that particular logic is legacy from a very early version of morph that
looked up repositories in a search path.
A repository URL should be a valid name for a chunk, so if people named their
chunks after the repository url that should also work.
I agree that this way around makes a bit more sense,
so I'd be happy with this change.
I don't see anything in your "WIP Add support for multiple sources per
chunk"
branch that would concern me at its current stage of development.
I'm still unsure about the extra-sources approach, but your implementation seems ok so
far.
2861
days inactive
3237
days old
14 comments
7 participants
participants (7)
-
Daniel Silverstone -
Jonathan Maw -
Paul Sherwood -
Richard Ipsum -
Richard Maw -
Richard Maw -
Tiago Gomes