On Tue, 2017-05-16 at 13:09 +0100, Paul Sherwood wrote: Hi ! So I want to jump in at the top here and say that this is an ideal time for us to take a step back and consider/define "What is Baserock" more clearly. Sorry if this is thread-hijacking but I think it's basically inline with this thread, so I'm branching it off with a different subject heading. Since I got involved over a year ago, this has never really been very clear, after spending some time in the project I can tell that Baserock was an idea which grew and snowballed into various things, the main components of which I would think are:  o The format and and tooling  o Trove (a continuous aggregator of various revision control systems    and tarballs into git)  o The reference builds (definitions) With the tight coupling we've had, it's been quite difficult to gain any traction or widespread adoption of the concepts innovated in this project, but they are all quite nice things when you take them in more bite sized pieces. So I'd like to just break the above down and highlight some things:   Format and tooling   ~~~~~~~~~~~~~~~~~~   With BuildStream we've been trying to take the concepts innovated   in the Baserock project and make them applicable to a much wider   variety of use cases, and I think so far we've been successful with   this.   In addition to being able to build full stack operating systems, one   will be able to build and deploy a variety of things such as Docker   images, Bundled applications such as Flatpaks, Snaps or AppImages,   SDKs and standalone runtime environments, Distro specific packages,   etc.   Also we are not tightly coupled to any VCS, making migrations to   BuildStream more user friendly for various projects which have always   been using a variety of upstream tarballs and other repositories,   without stating that one absolutely must use Trove (However I do   think trove is a good idea, and would recommend that Baserock   reference systems continue to use it).   Trove, software aggregation into git   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   This is a relatively small component of Baserock but I think it shows   great potential on it's own.   When building large complex stacks, whether they are base operating   systems or otherwise, all from source; a system like this helps a   great deal with being able to track stack-wide branches and various   downstream patches. In fact it would be interesting if Trove could   itself have a concept of stack-wide branches and perhaps a frontend   allowing developers to more easily track the work they have applied   across multiple git repositories.   This has always been unidirectional, though; software aggregated into   Trove stays in trove and downstream patches live in git branches.   I think it would be a novel idea if this could be made bidirectional,   or at least assist organizations and developer teams in the   upstreaming of downstream managed work branches - we already have   the lorry plugins/modules for aggregating incoming commits; why   not extend these to allow a developer to:     a.) Work on a git managed repository with an upstream SVN/CVS     b.) Allow the same developer to do either of the following:       b1.) Push downstream git-managed work branches onto upstream            SVN/CVS/HG/BZR branches       b2.) Produce foreign VCS friendly patches that are convenient            for upstream maintainers to merge into their own            repositories that may not be managed with git   Baserock reference systems   ~~~~~~~~~~~~~~~~~~~~~~~~~~   I think this is the only thing we should end up keeping under the   "Baserock" banner. Let Trove/Lorry be a separate project.   What I think we need to concentrate on here is becoming a welcoming   and "Safe" venue for people to contribute to the building of some   linux reference builds from scratch.   As Paul mentions below, I have been towing the line about stable   release branches for as long as I have known about Baserock, without   this discipline, Baserock will never be a safe venue for contributing   real integration fixes.   Beyond this, Baserock can also be a venue for demonstrating good   CI practices.   It would also be amazing I think if we could generate documentation   from BuildStream YAML and annotations to rival those of the LFS,   project, maybe even one day we could get the LFS folks on board and   merge the two projects, since they are doing something very similar,   and it's sad to see resources split across various projects doing   almost the same things. Personally I am most interested in the last point here, I want to know what are we calling "Baserock" and what are our priorities with the Baserock project and brand moving forward ? Best Regards,     -Tristan

On 18/05/17 10:48, Paul Sherwood wrote: I think this is a good move. Firstly, if Baserock just provides software build and integration instructions it makes it much easier to explain what it is. It's "A set of BuildStream definitions for producing operating systems". It as we've proved over the last few years.can usefully be compared with Yocto or Debian or anything else that provides OS build instructions. Trove is an interesting thing in my opinion and part of me likes the idea of "a server appliance that helps you follow best practices" but that description has the same issue as many of our old ideas of what Baserock is: it doesn't actually tell you what it *does*. So I would be in favour of splitting Trove up into "a Git importing and mirroring service", and whatever other parts are needed. We can write a "best practices for developing OSes with BuildStream" document that recommends mirroring all your code, and recommends setting up whatever we think is the best Git server, and whatever we think is the best Git import and mirror service, and whatever else. When writing that document, if it turns out that there's no good existing solution we can make one from the ashes of Trove. However I think the current concept of Trove should die along with the current concept of Baserock as they are both too nebulous. This depends on resources. I would be happy if Baserock gains enough resources to provide stable branches and security updates. Without an ongoing commitment of engineering time to make that possible, I would prefer for us to describe it as an unstable semi-maintained playground because in practice that's what it'll be. Sam -- Sam Thursfield, Codethink Ltd. Office telephone: +44 161 236 5575

On Thu, 2017-05-18 at 12:38 +0100, Sam Thursfield wrote: Right,      honestly I like the idea of a GitLab which does trove-like things (which Paul mentioned), I dont think gitlab does any kind of continuous aggregation from foreign VCS repositories, I dont think this kind of feature fits into a kind of extension/plugin of GitLab either, however perhaps using git + GitLab web APIs something interesting could be achieved. Lacking the aggregation-to-git features that Trove offers, we would need some other (simpler ?) mirroring service that ensures we have a permanent copy of anything we want to use in Baserock; I suppose in the short term Baserock will anyway continue to use Lorry/Trove, but changing this is something to think about. I disagree with this, and I have difficulty believing that we can achieve anything of value if we're not thinking about this. I.e. it does not matter so much how *good* the quality of the systems we produce are (that can improve with time), but it's IMO of paramount importance that we nail a decent branching policy which allows us at *least* to produce systems that do not regress because of constantly moving targets all around, the current approach makes not sense to me at all. Implementing a release schedule and stable/unstable branching policy is basically an activity that we can do once. After this is done, maintaining this is a matter of following policy. Also, I dont think we can expect to grow this community to extend to serious contributors which actually collaborate in order to release production systems, if we do not *first* create a safe venue for that. In other words, it's OK if our first "stable branch" never receives any security patch or integration fix at all, at least that branch is a safe location to contribute a patch to, where we know that nobody is going to just go ahead and bump the major version of anything. Right now there is no such safe location, so theres no incentive to collaborate. Best,     -Tristan

On Thu, 2017-05-18 at 13:46 +0100, Sam Thursfield wrote: [...] I've been pretty tied up with other things and forgot to get back to this email... I'll answer the second part first, what you are describing is really only a matter of perception and naming of such, if I understand your concern correctly. It's rather petty IMO, also, i.e. the fact that obviously the quality of one stable branch will be low until some critical mass is reached, is not an argument to not bother with proper release schedules at all. That said, I find that naming of "stable" releases with code names, as Yocto (e.g. "jethro") or Debian (e.g. "jessie") does, instead of outright calling it only "stable", is a decent solution to this concern. How would we get this started ? First we would agree on a release cadence, I personally feel that a longer release cadence would be less costly than a short one (I think I would prefer 1 year over something very short like 6 months), because once a final release is reached, module major point versions should be set in stone and only minor point version bumps can be acceptable (i.e. the API never breaks, system wide in a final release version). With a short release cadence we would either:   A.) End up releasing randomly selected bleeding edge software,       without really knowing the impact of having "foo 1.2"       interfacing with "bar 3.4"       There will be a certain amount of this anyway, but if we       are low on resources, we wont have a good enough turn around       with a short cadence. Or:   B.) End up having some consecutive final release branches with       virtually no changes, because nobody had time to integrate       newest kernel + gcc + glibc and so everything is largely       the same as the last. After deciding on a cadence, we need to implement a schedule, and the schedule must be the law. This means we have 2 or 3 levels:  * master branch is always bleeding edge    This is where the version junkies have their fun, upgrading all    kinds of modules and having the bleeding edge of everything, without    a care in the world of whether the resulting systems actually work.    It builds: Ship it !  * testing branch is where we are pulling in changes from master,    and trying to ensure that the package version vector makes sense    (eventually this will have some real world testing going on too,    I would hope)    This is where I think the real integration engineering happens,    maintainers of the "testing" branch, start their release cycle    from the mess that is today's master (after a "final" release)    At this point, we start seeing if systems actually boot, and find    that "foo 1.2" actually does _not_ work properly with "bar 3.4",    in upgrade to "foo 2.0" or back down to "bar 2.8" so that the    software which was intended to work together actually does, at    least mostly.    Here in "testing" we ensure we have a sane PAM configuration    strategy and that keyrings work, that the display manager boots    into a real session and most things are generally working.  * final branches are release branches and they are like a clock,    they take whatever is "testing" and call it "final" (or different    code named branch for every cycle).    If "testing" is not ready, I think just too bad. We release anyway,    the clock is god and if we fudged this release really badly, we'll    get the next one better, but no delays.    After final branches are released, major/minor point software    versions must stay the same, only accept micro point version bumps    and security patches etc.    Also the history of these "final" branches should serve as a good    basis for the next "testing" cycle; i.e. if we had to patch a bunch    of things in this "final" release, we should take a look at that    history and see if these things still apply to "master" when    constructing the next "testing". The important thing to note with testing and final branches, is that we have to be aware that they are the clock, and the clock is the law. Unlike corporate driven software development, a stable release is never postponed because management wants to have some fancy new feature in "stable", that should really be in "master/unstable" at this phase. As mentioned above, if we fudged this release; we'll get it right in the next. This attitude towards releases lets us define it as policy, without necessitating that resources be allocated and saying that time must absolutely be spent; the release cycle is only a clock with some rules and branches, the quality of the releases will vary; but at the very least we will _have_ a policy defined to start with. In the above I've thrown out a lot of my own opinions of how it can be managed, but we should first draft a release schedule and argue about those rules and define them a bit better, it's mostly an example of how we can get started. Cheers,     -Tristan