BuildStream conversions of Baserock definitions
by Sam Thursfield
Hi all,
I'm involved in a new project to produce some OS images for unusual
targets, and since the Baserock reference systems are great for this
I'd like to reuse them. There's a requirement to use BuildStream on our
side, so following on from Tristan's proposal of adopting BuildStream
in Baserock[1] I started looking at converting the whole definitions
repo.
A while back Tristan wrote the conversion tool defs2bst[2] which seems
to work well. And Javier already set up a GitLab CI pipeline[3] to
convert the gnome-system to BuildStream then try to build it. So far
this has turned up some bugs but is a bit out of date.
What I have done is a manual conversion in a branch of
definitions.git, which is here:
<https://gitlab.com/baserock/definitions/tree/sam/buildstream>
The defs2bst script works well for nearly everything but you can
see in the commit log that a few manual fixes were needed.
The old Baserock definitions are moved to `old/`, and converted
BuildStream defintions are in `elements/`. The converted systems
are in the elements/systems dir.
Not all systems are converted and probably never will be. If I've missed
one that you care about please reply here.
Only the x86_64 conversions are done, and I think all the
platform-specific stuff needs to be manually converted in order to
reduce duplication and take advantage of BuildStream's 'variants'
feature. I've not looked into how best to do this yet.
The devel-system is the only one that has a deployment target,
`systems/devel-system-image.bst`. This is based on what Tristan did
for the gnome-system conversion. The BuildStream project is currently
working on integrating support for image deployments into the tool
itself, get involved there if you're interested in deploying images.
The `convert` script in the top directory can be used to redo the
conversion, it will need some tweaking but this makes it possible to
keep up with changes on the 'master' branch of definitions.git. Keep an
eye on `git diff` before committing updates to avoid un-doing the
various manual fixes.
I don't know if I'll be driving this migration further or not at this
point. I do hope that we can keep moving this forward though. I'm
interested in hearing views from people who are actively using the
Baserock definitions.git repo about this migration and what blockers you
see to merging this branch to 'master' and switching to BuildStream as
a build tool.
Thanks
Sam
[1]
https://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/201...
[2] https://gitlab.com/BuildStream/defs2bst/
[3] https://gitlab.com/baserock/definitions/blob/jjardon/bst2/.gitlab-ci.yml
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
5 years, 11 months
- 3
- 3
Next steps for Baserock: scope/tooling/infrastructure
by Paul Sherwood
Hi all,
as many readers will know, Codethink has been gradually shifting
emphasis in the Baserock project over the last couple of years. We've
come a long way since the original work on morph and trebuchet from
2011, where we started working on bootstrapping whole-stack builds and
atomic updates for custom embedded Linux systems. Now many other FOSS
projects are implementing solutions that overlap or supersede our work,
in various different ways.
Codethink (and indirectly our customers) continue to be the main
sponsors of Baserock, the project has been a key testing ground for core
ideas, techniques and tools we have exploited on advanced commercial
projects and in support of our wider contributions to various FOSS
initiatives.
We're now considering what we should keep, what would be better
deprecated, and what gaps still need to be filled. The remainder of this
email will aim to summarise our current perspective:
Build tooling
=============
We've effectively deprecated morph already in favour of ybd, but there
are some morph functions that ybd does not (and probably won't) fulfil.
Similarly work is well underway in BuildStream, and we believe that this
represents a better long-term solution than either of its predecessors.
So broadly our hope will be to officially retire both morph and ybd
around the end of this year. Note, however - both tools are in use on
customer projects, and they are both FOSS projects - so the source will
remain available and and we'll support any community users wanting to
continue using these tools for specific situations.
From a commercial point of view, we'll be encouraging customers to
migrate their projects forward to BuildStream - this should be
relatively painless, since one of the key successes in Baserock has
been the evolution of whole stack build instructions as declarative YAML
definitions, and we already have tooling to automate the conversion of
definitions YAML into BuildStream YAML. We'll be happy to assist with
migration of even ancient morph files, either in public or private as
required.
Definitions
===========
We experimented with versioning of the definitions format, but in the
end with YBD I tried as far as possible to make the code support all
versions without impacting users. It remains to be seen how BuildStream
will handle this over time, but we've already learned that
developer-users won't upgrade their tooling unless the upgrade approach
is trivial, painless and bulletproof.
So our aim will be to expressly migrate our example/reference
definitions into BuildStream format once BuildStream becomes our
de-facto build tool.
Reference systems
=================
Over the years we have created many examples with Baserock - minimal
embedded systems for various architectures (x86, ARM, MIPS, POWER),
Big-Endian systems, cross-bootstrap systems, GENIVI demonstrators,
OpenStack systems and Trove. Some of these are clearly ready for
retirement now, so we'll clean them out of mainline when we transition
to BuildStream.
Tristan and others have flagged that even in scenarios where frequent
updates are expected, downstream users normally need a 'stable'
branch/releasing approach, since no-one should be forced to deal with
large-scale upstream changes before they are ready. For the most part
downstream users have expressly elected to fork definitions rather than
consume our updates, which may or may not be as a result of our release
model and/or friction in consuming our updates. Maybe it's actually the
best approach for this kind of metadata, but I believe more thinking is
required.
Web and Wiki
============
While there is lots of useful historical information at
wiki.baserock.org (which is the default landing site for Baserock), it's
mostly out-of-date and so the plan is to archive the current content and
get to a simpler site with a smaller set of information and links to
other places (eg BuildStream and other upstream pages)
Infrastructure and Services
===========================
We maintain several services including git.baserock.org, wikis,
pastebin, IRC logging, CI systems, build machines, and credentials/keys
for various sites. We also have multiple issue trackers.
Our aim is to rationalise these services, broadly migrating anything we
can onto GitLab. However we do expect that we'll have unusual
requirements that will continue to require custom services (eg
Big-Endian systems). And in any case we'll continue to host
git.baserock.org, as an independent reference/fallback against the
possibility of compromise/outages at GitLaba and other upstreams.
I hope this makes sense. Please let me know on-list or off, if you have
any questions or concerns.
br
Paul
5 years, 11 months
- 3
- 7
- ← Newer
- 1
- Older →