4:36 a.m.
Hello
I'd like to propose a change to the Baserock definitions format. Which
would bring us to version 6. The change is: require specifying build
instructions for every chunk in definitions.git.
Wait! Put the fiery pitchfork down for a moment. This just means adding
an extra field to stratum .morph files, so you have to specify
'build-system' alongside 'repo' and 'ref' for any chunks that
don't have
their own .morph file. It doesn't mean we now need to write .morph files
for all 763 chunks in definitions.git.
The reason for this change is: I'd like to make it easy to go from a
definitions.git repo full of .yaml files, to a full set of build
instructions. Currently you can't do that -- for every chunk that
doesn't specify build instructions, you need to know the list of files
in the given commit being built. Which, in practice, means you need an
internet connection, or a local cache of all the chunk repos which is
up to date with your definitions.
I think that's too much to ask of people who just want to find out how
to build something. I realise that if you want to actually run the
build, you need the repos available at some point anyway. But you may
just want to analyse definitions or convert them to a different format,
and it's pretty lame to require an internet connection or a huge local
Git cache just for that.
I propose that we remove the ability to store chunk .morph files in
chunk repos, for the same reason. I don't think anybody has been using
this feature for years.
- Migration script for definitions v6:
https://gerrit.baserock.org/1002/
- Example of migrated definitions:
https://gerrit.baserock.org/1003/
- Morph patch to add support for definitions v6:
https://gerrit.baserock.org/1004/
- YBD patch to build definitions v6:
https://github.com/ssssam/ybd/tree/sam/definitions-version-6
- Example Morph patch to remove support for definitions v3-v5:
https://gerrit.baserock.org/1001
Please let me know your thoughts! In the form of a +1 or -1 perhaps.
I think it's best if we keep discussion of the overall approach on the
mailing list, rather than in Gerrit. I've posted patches to
illustrate the proposal, but I don't want them to be merged unless the
readers of this list are happy with the change to the definitions
format.
If this change is approved, I'll look at moving the predefined split
rules and build commands into definitions.git, at which point it'll
be possible to actually get a full set of build instructions just by
analysing definitions.git.
Thanks!
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
4:59 a.m.
On 2015-07-24 09:36, Sam Thursfield wrote:
Hello
I'd like to propose a change to the Baserock definitions format.
Which
would bring us to version 6. The change is: require specifying build
instructions for every chunk in definitions.git.
Wait! Put the fiery pitchfork down for a moment. This just means
adding
an extra field to stratum .morph files, so you have to specify
'build-system' alongside 'repo' and 'ref' for any chunks that
don't
have
their own .morph file. It doesn't mean we now need to write .morph
files
for all 763 chunks in definitions.git.
The reason for this change is: I'd like to make it easy to go from a
definitions.git repo full of .yaml files, to a full set of build
instructions. Currently you can't do that -- for every chunk that
doesn't specify build instructions, you need to know the list of
files
in the given commit being built. Which, in practice, means you need
an
internet connection, or a local cache of all the chunk repos which is
up to date with your definitions.
I think that's too much to ask of people who just want to find out
how
to build something. I realise that if you want to actually run the
build, you need the repos available at some point anyway. But you may
just want to analyse definitions or convert them to a different
format,
and it's pretty lame to require an internet connection or a huge
local
Git cache just for that.
I propose that we remove the ability to store chunk .morph files in
chunk repos, for the same reason. I don't think anybody has been
using
this feature for years.
- Migration script for definitions v6:
https://gerrit.baserock.org/1002/
- Example of migrated definitions:
https://gerrit.baserock.org/1003/
- Morph patch to add support for definitions v6:
https://gerrit.baserock.org/1004/
- YBD patch to build definitions v6:
https://github.com/ssssam/ybd/tree/sam/definitions-version-6
- Example Morph patch to remove support for definitions v3-v5:
https://gerrit.baserock.org/1001
Please let me know your thoughts! In the form of a +1 or -1 perhaps.
I think it's best if we keep discussion of the overall approach on
the
mailing list, rather than in Gerrit. I've posted patches to
illustrate the proposal, but I don't want them to be merged unless
the
readers of this list are happy with the change to the definitions
format.
If this change is approved, I'll look at moving the predefined split
rules and build commands into definitions.git, at which point it'll
be possible to actually get a full set of build instructions just by
analysing definitions.git.
I really like the approach, am happy with the format change, and would
very much like to see the migration added to and applied to definitions
master. (and a similar approach taken with split rules if possible)
You know how I'm drawn to patches that remove 50 lines of code, but
iiuc applying your patch would make ybd fail on any previous version of
definitions, which would cause surprise/friction for users who have not
migrated so far. Alternative approaches I can think of include
1) tidy up so that current buildsystem.py stays as a vestigial
fallback, only called if bs not found, and then drop the possibility of
fallback at some time in the future.
2) implement your patch but provide a clearer instruction to the user
on hitting the error.
From my pov option 2) seems best? but i don't think the instruction
could be 'run the $migrate script' because maybe that would fail too (eg
VERSION != 5)
5:16 a.m.
On 24/07/15 09:59, Paul Sherwood wrote:
On 2015-07-24 09:36, Sam Thursfield wrote:
I really like the approach, am happy with the format change, and would
very much like to see the migration added to and applied to definitions
master. (and a similar approach taken with split rules if possible)
Great!
You know how I'm drawn to patches that remove 50 lines of code,
but iiuc
applying your patch would make ybd fail on any previous version of
definitions, which would cause surprise/friction for users who have not
migrated so far. Alternative approaches I can think of include
1) tidy up so that current buildsystem.py stays as a vestigial fallback,
only called if bs not found, and then drop the possibility of fallback
at some time in the future.
2) implement your patch but provide a clearer instruction to the user on
hitting the error.
From my pov option 2) seems best? but i don't think the instruction
could be 'run the $migrate script' because maybe that would fail too (eg
VERSION != 5)
Right now YBD ignores the VERSION file completely. Is that deliberate,
or just 'not implemented yet' ?
In Morph, the behaviour is conditional based on the contents of the
VERSION file. This makes the code a bit ugly while we have to support
multiple versions, but it does mean that when I want to remove support
for a version, I can run 'git grep definitions_version' to find and
remove old code paths. It was quite fun doing that yesterday!
If YBD reads VERSION then option (2) is super easy to do. Option (1) is
a bit more work either way, but I still think making it conditional
based on VERSION is better than making it conditional based on whether
the 'build-system' field is missing or not, because the latter approach
could fail in complex ways if someone was using definitions format 6 but
forgot to put in the 'build-system' field for one chunk, or something
like that.
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
5:45 a.m.
On 2015-07-24 10:16, Sam Thursfield wrote:
On 24/07/15 09:59, Paul Sherwood wrote:
> On 2015-07-24 09:36, Sam Thursfield wrote:
> I really like the approach, am happy with the format change, and
> would
> very much like to see the migration added to and applied to
> definitions
> master. (and a similar approach taken with split rules if possible)
Great!
> You know how I'm drawn to patches that remove 50 lines of code, but
> iiuc
> applying your patch would make ybd fail on any previous version of
> definitions, which would cause surprise/friction for users who have
> not
> migrated so far. Alternative approaches I can think of include
>
> 1) tidy up so that current buildsystem.py stays as a vestigial
> fallback,
> only called if bs not found, and then drop the possibility of
> fallback
> at some time in the future.
>
> 2) implement your patch but provide a clearer instruction to the
> user on
> hitting the error.
>
> From my pov option 2) seems best? but i don't think the instruction
> could be 'run the $migrate script' because maybe that would fail too
> (eg
> VERSION != 5)
Right now YBD ignores the VERSION file completely. Is that
deliberate, or just 'not implemented yet' ?
Deliberate. By default I prefer ybd to work if it can, rather than
failing. While I appreciate the thinking and work that has gone into the
VERSION implementations, I'm still hoping we can find a solution that is
less friction for users.
In Morph, the behaviour is conditional based on the contents of the
VERSION file. This makes the code a bit ugly while we have to support
multiple versions, but it does mean that when I want to remove
support
for a version, I can run 'git grep definitions_version' to find and
remove old code paths. It was quite fun doing that yesterday!
:-)
But still, this hasn't been a problem for ybd so far in part because
there's less code and in part because as a matter of policy i'm trying
to make it continue to work, rather than deprecating anything. If/when
ybd's code gets too tortuous for upstream to understand I'll think
again. And I'm really hoping that the RDF/OWL approach you've been
looking at will alleviate this pain anyway.
If YBD reads VERSION then option (2) is super easy to do. Option (1)
is a bit more work either way, but I still think making it
conditional
based on VERSION is better than making it conditional based on
whether
the 'build-system' field is missing or not, because the latter
approach could fail in complex ways if someone was using definitions
format 6 but forgot to put in the 'build-system' field for one chunk,
or something like that.
I may be missing something, but I think that until such time as we
actually actually *modify* the default build instructions themselves,
ybd would work unchanged (i.e. without caring about VERSION
0,1,2,3,4,5,6).
In the use-case you described (VERSION 6 but missing build-system for
one chunk) morph would fail, because there's a problem in the
definition, but ybd would build it. You could say a successful build in
that context is technically a fail - the correct result should be that
the build breaks. But this presumes that the build tooling is taken as
the arbiter of whether a given set of definitions is
syntactically/semantically correct. I think it might be better for
validation of the definitions themselves to be done by 'something else'
(via schema etc) so that the input to the build tooling can be presumed
correct.
But maybe that's off-topic. On reflection if the migration is approved
I'll be happy to accept this ybd change and clarify the error message
instructions
7:02 a.m.
On 24/07/15 10:45, Paul Sherwood wrote:
On 2015-07-24 10:16, Sam Thursfield wrote:
>
> Right now YBD ignores the VERSION file completely. Is that
> deliberate, or just 'not implemented yet' ?
Deliberate. By default I prefer ybd to work if it can, rather than
failing. While I appreciate the thinking and work that has gone into the
VERSION implementations, I'm still hoping we can find a solution that is
less friction for users.
OK. There's a tradeoff here between predictability and convenience.
I think it is better to have a single file controlling how the
definitions are interpreted. This means less convience for users
up-front, but it also means there's less chance of a confusing failure case.
One of the great things about having two build tools is we can try both
approaches, if you're up for it.
But still, this hasn't been a problem for ybd so far in part
because
there's less code and in part because as a matter of policy i'm trying
to make it continue to work, rather than deprecating anything. If/when
ybd's code gets too tortuous for upstream to understand I'll think
again. And I'm really hoping that the RDF/OWL approach you've been
looking at will alleviate this pain anyway.
Do you know of anyone regularly testing YBD against old versions of
definitions?
I had a go at building the baserock-15.02 tag of definitions.git inside
a Baserock 15.25 'build' system, and stage1-gcc fails to build, with
both Morph and YBD. Seems to be because it's trying to build
documentation which it didn't used to, maybe because a new tool was
added to the 'build' system. Building 15.10 (after Tiago did the upgrade
to GCC 4.9.2) works fine still.
I don't really have a point here, except that the universe is always
going to find a way to outsmart the build tool at some point, and that
how a tool works when given bad/inconsistent input is just as important
as how it works when given good input.
> If YBD reads VERSION then option (2) is super easy to do. Option
(1)
> is a bit more work either way, but I still think making it conditional
> based on VERSION is better than making it conditional based on whether
> the 'build-system' field is missing or not, because the latter
> approach could fail in complex ways if someone was using definitions
> format 6 but forgot to put in the 'build-system' field for one chunk,
> or something like that.
I may be missing something, but I think that until such time as we
actually actually *modify* the default build instructions themselves,
ybd would work unchanged (i.e. without caring about VERSION 0,1,2,3,4,5,6).
Yes, that's true.
In the use-case you described (VERSION 6 but missing build-system
for
one chunk) morph would fail, because there's a problem in the
definition, but ybd would build it. You could say a successful build in
that context is technically a fail - the correct result should be that
the build breaks. But this presumes that the build tooling is taken as
the arbiter of whether a given set of definitions is
syntactically/semantically correct.
If my proposed patch to Morph was merged, then Morph would fail in this
case (before building anything) with an error message saying "Chunk xx
in stratum yy has no build-system defined, ...". I think that's quite
convenient, you just need to set build-system and rebuild.
YBD would build it, which is also convenient, but if we introduce a tool
in future which, for example, converts definitions to Bitbake recipes,
that tool would fail on the invalid stratum. And then the user has to
add the missing field anyway.
I think it might be better for
validation of the definitions themselves to be done by 'something else'
(via schema etc) so that the input to the build tooling can be presumed
correct.
If you expect validation will be done, then *where* it is done isn't
really important. If you allow the user to run the build tool on
invalidated definitions then you risk the build tool crashing instead of
giving useful errors.
I think doing validation outside the build tool is a good idea, if we
can do it in a way that means the build tool can't ever see invalid
data. But that validation would still raise an error if 'build-system'
wasn't defined for a chunk that needed it.
But maybe that's off-topic. On reflection if the migration is
approved
I'll be happy to accept this ybd change and clarify the error message
instructions
OK :)
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
2:49 p.m.
On 24 July 2015 at 09:36, Sam Thursfield <sam.thursfield(a)codethink.co.uk> wrote:
Hello
I'd like to propose a change to the Baserock definitions format. Which
would bring us to version 6. The change is: require specifying build
instructions for every chunk in definitions.git.
Wait! Put the fiery pitchfork down for a moment. This just means adding
an extra field to stratum .morph files, so you have to specify
'build-system' alongside 'repo' and 'ref' for any chunks that
don't have
their own .morph file. It doesn't mean we now need to write .morph files
for all 763 chunks in definitions.git.
The reason for this change is: I'd like to make it easy to go from a
definitions.git repo full of .yaml files, to a full set of build
instructions. Currently you can't do that -- for every chunk that
doesn't specify build instructions, you need to know the list of files
in the given commit being built. Which, in practice, means you need an
internet connection, or a local cache of all the chunk repos which is
up to date with your definitions.
Agree completely
I think that's too much to ask of people who just want to find
out how
to build something. I realise that if you want to actually run the
build, you need the repos available at some point anyway. But you may
just want to analyse definitions or convert them to a different format,
and it's pretty lame to require an internet connection or a huge local
Git cache just for that.
I propose that we remove the ability to store chunk .morph files in
chunk repos, for the same reason. I don't think anybody has been using
this feature for years.
+1
- Migration script for definitions v6:
https://gerrit.baserock.org/1002/
- Example of migrated definitions:
https://gerrit.baserock.org/1003/
- Morph patch to add support for definitions v6:
https://gerrit.baserock.org/1004/
- YBD patch to build definitions v6:
https://github.com/ssssam/ybd/tree/sam/definitions-version-6
- Example Morph patch to remove support for definitions v3-v5:
https://gerrit.baserock.org/1001
Please let me know your thoughts! In the form of a +1 or -1 perhaps.
I think it's best if we keep discussion of the overall approach on the
mailing list, rather than in Gerrit. I've posted patches to
illustrate the proposal, but I don't want them to be merged unless the
readers of this list are happy with the change to the definitions
format.
+1, nice work
If this change is approved, I'll look at moving the predefined
split
rules and build commands into definitions.git, at which point it'll
be possible to actually get a full set of build instructions just by
analysing definitions.git.
Yes please
--
Javier Jardón Cabezas
7:15 a.m.
On 24/07/15 09:36, Sam Thursfield wrote:
Hello
I'd like to propose a change to the Baserock definitions format. Which
would bring us to version 6. The change is: require specifying build
instructions for every chunk in definitions.git.
Thanks for the comments so far, I'm going to go ahead with this change
based on two +1s.
I don't know exactly when I'll have time to do a release of the
reference systems, so that we can start to use version 6 in our
definitions.git. I'm going to look at moving default rules into
definitions.git this week, and am on holiday for 2 weeks after that. If
anyone is reading who wants to do a release before that, Pedro or
someone else will be able to help you!
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
2536
days inactive
2539
days old
6 comments
3 participants
participants (3)
-
Javier Jardón -
Paul Sherwood -
Sam Thursfield