Hello As you could tell by the continuous patches and emails, over the past few weeks I have been looking into schemas and migrations for the Baserock definitions format. The actual goal was to make it easy to understand and to change the Baserock definitions YAML syntax and the Baserock data model! It took me a good week to do definitions version 6 (and we still can't use it in the reference systems til a release is done) so I can hardly claim that making changes is now easy. I think this is mostly because we still have a dependency tangle between build tools, definitions.git, the definitions format, and the released reference systems. I'm not going to be looking at this further for now, so this mail is a summary on where we are and an open question on where we might go from here. What we have ------------ - A pattern for doing automated forward migrations of existing Baserock definitions (see `migrations/*` in definitions.git[1]). - (Proposed) JSON-Schema descriptions of the Baserock definitions YAML representation format: <https://gerrit.baserock.org/1022/>;. - A 'homepage' for the Baserock definitions format: <http://wiki.baserock.org/definitions/>;. Actually, the main thing I've wanted to do is separate the concept of the 'data model' from the .morph files that we actually deal with. It's important to represent data in a way that people can read and edit, and it's also important to represent data in a way that computer programs can operate on. These two things are usually at odds, though, and I feel that in the past we've been derailed by trying to optimise for both use cases at once. [1]: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.gi... Where we can go from here...? ----------------------------- We need to do a release of the Baserock reference systems so that we can use definitions format version 6 (and hopefully version 7, if accepted) in our definitions.git repo, and drop support for older versions from Morph and YBD. The description of the definitions format at <http://wiki.baserock.org/definitions/current/> needs bringing up to date. Hopefully a lot of it can be removed and replaced with a link to JSON-Schema and OWL schemas. Beyond that, I have no concrete plans. But here are some ideas... Morph's validation code could be replaced with the Python 'jsonschema' module, although so far I've not been overly happy with the error messages that module gives, so far. I would like to see how simple a Baserock build tool can be. YBD is interesting, but now that there is no need to do build-system autodetection (as of definitions format version 6) I think you could write a 'definitions repo -> massive shell script full of build instructions' tool in a couple of hundred lines. It should also now be simple(ish) to convert Baserock definitions to other languages such as Bitbake recipes, Makefiles, the Nix package language, or anything else interesting. We must fix the dependency tangle between Morph/YBD/distbuild, the Baserock reference systems, the definitions format, and the Baserock reference system definitions. This is key to making it actually easy to change the Baserock definitions format. The best way to do it is... left as an exercise. Define a better data model. I came up with an initial one in `baserock.owl` as part of <https://gerrit.baserock.org/1022/>;, but I don't think the terms 'Chunk', 'Stratum', 'System' or 'Cluster' are really much use there. Our tools deal with 'something that has build instructions and/or contents', largely. We need to find a name for that concept (or extend an existing concept such as 'Package' from SPDX), and make a more simple and flexible data model. YBD's internal data model has gone some way in this direction already. Once we have a better data model we can update the YAML serialisation format to match. We could try to change both things in parallel, but I think that would be much more difficult. Conditionals! I have some thoughts on how these could be done, which won't fit in this email. The key idea is that conditionals should exist at the level of the YAML representation format. In the abstract, all the possible variants of a system do actually exist, and we need to keep track of which ones are built and which ones are tested. Write a Git merge tool that understands the Baserock definitions YAML representation format. That would make merge conflicts in definitions repos much nicer to deal with, I think. One thing we are still missing is a good name for the "Baserock data model" and especially the "Baserock definitions YAML representation format" :) Sam -- Sam Thursfield, Codethink Ltd. Office telephone: +44 161 236 5575