On Sun, Apr 10, 2016 at 4:03 PM, Paul Sherwood
> On Sat, Apr 9, 2016 at 11:14 PM, Paul Sherwood
> <paul.sherwood(a)codethink.co.uk> wrote:
>>> While I agree that spec and its use should be close, I don't think we
>>> should be scared of having multiple changes proposed at once
>> This is not practical with the current mechanism - for example, any
>> explicitly needs to include documentation and a migration from Vx to
>> for a known x.
> Only if they affect the same aspect of the definitions format.
No, that's not correct. Every migration is hard-coded to a given TO_VERSION
number, and won't apply unless the current VERSION is TO_VERSION - 1.
I think that's a minor thing that could be fixed by whoever does the
final merge. When developing the migration, use whatever FROM_VERSION
is convenient, on merge, change it to whatever is correct. I'd be
happy to do that when merging patches for spec.git. I don't see how it
prevents developing two migrations in parallel.
That said, maybe you're thinking of an approach that avoids this
problem? If new features were given labels, instead of numbers, for
example ... but I think that'd lead to the VERSION file being a
massive list of features, and it'd be less obvious to work out if your
build tool didn't support feature X because feature X is obsolete and
your definitions are old, or because feature X is new and your build
tool is too old.
Plus there are various references to current version in the docs,
people are bound to forget to fix, or which will conflict on merge.
Right.. we do need to stop putting the version in so many places. It's
good to have the version number listed in each .json-schema file in
case they get copied out of the repo. But we could drop it from there,
it's no big deal.
> example, if you propose a change that renames build-depends to
> ''depends', and someone else proposes changing the
> field to being a dict instead of a list, then those two changes will
> conflict (although in a pretty minor way). If you propose renaming the
> build-depends field, and someone else proposes changing the format of
> cluster .morph files completely, and someone else proposes allowing
> systems to contain other systems, I don't see how those 3 changes
> would conflict. For each of these changes, you could write a migration
> from version X which would also work against X+1 or X+2.
Yes, I could. But none of the previous migrations has been done this way
(and I think it's harder to do) and all references to current version in the
text would need to be aliased, because... I'm proposing a change, I don't
know which version it will be applied to, but afterwards everything in the
repo needs to be automatically at version+1.
I see. I think if the version is only in 1 place this is a minor
problem. But right now because lots of places list the version number,
it's a pain.
> For example, we didn't think to keep testing the migrations
examples of definitions....
with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory:
Command '../spec/migrations/007-defaults-in-definitions.py' returned
non-zero exit status 1
Oops, this is my fault for not testing the migrations after moving
them to spec.git. I think that one hardcodes an assumption that it's
being run from a certain place, which was just bad design. I'll send a
So if a user with an old version of definitions actually followed our
here (how would they, btw? there's nothing about spec.git on the wiki)
they'd find that the migrations don't work.
The steps are documented in the definitions README:
It would be good to document this on the wiki too. I don't feel like
doing this myself right now, i'm not quite sure where the info would
go and would probably end up getting sucked into trying to rewrite all
the documentation without really knowing it's who it's for or how to
>> But in any case, changes to the schema or spec are quite
invasive - once
>> definitions are migrated to the new schema/spec, in most cases any tools
>> operating on the definitions (eg build tools) will fail.
> Up til now you've claimed that YBD can build all historical versions
> of definitions. You seem to now be saying the opposite. I must be
YBD *parses* definitions all the way back to before VERSION was introduced.
It doesn't *build* them all, for reasons summarised in 
We can't predict future versions, so once a future change is made and
definitions are migrated, the build tools can be assumed/expected to fail
until/unless we fix them at the same time.
> OK. I can only see one way to solve your objections, and it comes
> the cost of turning the definitions format back into an implementation
> detail of a specific build tool. Feel free.
I don't think that would be better.
As I've said before, if the definitions were code, everyone would be ok
having tests with the code. Verifications that the definitions meet the
schema and the spec can and should be considered as tests for definitions,
I suggest we consider some or all of the following:
- return the contents of spec and migrations to definitions.git
Long term I still think this is a bad design. As Tristan said it's
like keeping the ISO C standard in the same repo as the only C program
that's ever expected to exist.
Short term, I can see that it reduces the number of repos people need
to consider, which is good. Do you see any other benefits beside that
I think it's OK to do this for the time being if it really helps. Or
maybe we could add spec.git as a submodule of the definitions ?
- scrap changelog.mdwn, and insist that the commit message describes
OK. That's quite an easy patch -- just delete the file and specify the
policy somewhere :-)
That said, if we merge spec.git into definitions.git those commit
messages will become a bit harder to find, creating friction for
people trying to understand previous versions of the spec.
- or have validation on the text files, to check for version <n
What does the "or" relate to? Yes, a validation script for the spec
repo is a great idea, as long as it doesn't grow into a huge Bash
- remove as many of the hard-coded values of current-version in the
- implement a generic migrator so that specific migrations don't
know which version they apply to.
I'm not quite sure how this would work, but would be great if it's possible
- implement pre-merge CI to ensure that
- the new migration works, and the migrated definitions satisfy the new
- all migrations (including the new one) work on previous versions of
definitions (and fix the previous migrations if they don't)
- fully migrated old versions of definitions also satisfy the schema
- at least one tool can build ci.morph at the new version
Completely agree, that would be very good to have.