[RFC] Declarative Definitions

[Paul Sherwood]

Hi
at yesterday's meetup there was a BoF on improving Definitions (which 
we often have referred to as morph files, or morphologies)

My rough notes of the discussion [1] lead me to think the actions are

- document migration philosophy and tooling (expect frequent small 
changes)
- get 'version: 0' in place, tooling to detect it
- define 'version: 1', small delta, .def, 
s/system|stratum|chunk/component/

I've taken a shot at the first of these steps at [2], and would 
appreciate feedback. For those who would prefer to comment inline on 
this list, the text of [2] follows at the end of this mail.

Subject to feedback, I'll be aiming to work up drafts of 'version: 0' 
and 'version: 1' at [3] and [4] - others are welcome to contribute 
things there before me or with me :-)

br
Paul

[1] http://wiki.baserock.org/meetup/declarative/
[2] http://wiki.baserock.org/meetup/declarative-definitions/
[3] http://wiki.baserock.org/meetup/declarative-definitions/V0/
[4] http://wiki.baserock.org/meetup/declarative-definitions/V1/
# Declarative Definitions

We want to establish a standard syntax for describing software 
components and systems. Learning from the experiences we have had with 
morph files, we have established some general principles:

- We create descriptions of software components, systems, and sets of 
systems. We call the descriptions Definitions.
- Software tools can use Definitions to create and operate on these 
components and systems. By adopting the Definitions approach we are 
aiming for traceability, reproducibility and ultimately deterministic 
behaviour of the Defined systems and components.
- As far as possible Definitions contain all the information necessary 
for build, deployment, instantiation etc.
- Definitions are a a set of text files in a directory tree.Typically 
the files are stored in a single Git repo
- Definitions describe Components. Each Definition file may describe 
one or more Components.
- A Component may consist of
   - a single software project e.g. systemd, or
   - a set of software projects, e.g. GNOME, or
   - an integrated software system, e.g. a web server appliance, or
   - a set or cluster of software systems, e.g. a ci controller with 
several worker nodes
- Components may be nested and contain other components. The 
information which specifies a Component may be split across multiple 
Definition files.
-  We have currently chosen YAML as the format for representing data 
relationships in Definitions.
- Most Definitions contain YAML, but some definitions contain shell 
scripts or other instructions to be executed as part of a configuration, 
build, installation, deployment, instantiation, upgrade or other process 
which tools can apply to Components or systems.

## VERSION

We have evolved morph files through many states over several years, 
without providing a mechanism for versioning as the syntax has evolved. 
As of the [[meetup]] we have decided to adopt a versioning scheme, 
broadly as follows:

- The Definitions directory tree contains a file called VERSION. If 
this file is not present, tooling may assume [[definitions-V0]] or fail 
gracefully
- The first line of the VERSION file must contain `version: ` followed 
by an integer. The integer uniquely defines the version of the 
Definitions syntax
- Later lines of the VERSION file are expected to contain YAML 
describing default information which tooling such as Morph may require. 
This information could include, for example
   - default PATH
   - locations for git server and artifact server
   - default ARCH target etc.
- By specifying defaults here we aim to reduce complexity, hidden logic 
and implicit assumptions in tools such as morph. The overall objectives 
for this are
   - running baserock tools on a set of Definitions should give 
predictable and reproducible results - if it works for Aananth on his 
machine it should work for Brenda on hers too
   - it should be possible for new tooling to instrument, analyse and 
make operations on the information contained in Definitions, with 
confidence that there are no hidden tricks performed by existing tools 
(e.g. anything that has to be done differently for ARMv7 should be 
explicitly declared in Definitions, not as conditional logic in the 
build tool itself)
- We expect that the version number will rise relatively quickly, so
   - tooling should expect and handle changes gracefully
   - if something is expected to break between VERSION N and VERSION 
N+1, a migration script must be provided. We see this as similar to ruby 
on rails migrations, and hope to standardize this process over the first 
few VERSION iterations