[RFC] Multiple source repositories per-chunk and git submodule handling
by Richard Maw
We currently handle definitions that need to pull multiple source
repositories together in an ad-hoc way.
For gcc we import the gmp, mpc and mpfr source trees in by checking them
into our delta branch directly.
Some upstreams handle importing the sources by adding submodules.
However, we have to make them fetch sources from our Trove for
reproducibilty and latency concerns, so we end up having to add a delta
on top to change where the submodules fetch their sources from.
This works against one of our goals for minimising the delta, so we
need a way to encode in our builds how to deal with components that need
sources from multiple repositories to function.
Our current approach from submodules introduces a _lot_ of extra work
when we need to update multiple submodules recursively, so we need a
better solution.
Proposal
========
To solve this, I propose we extend the source repository information from
just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
So rather than:
name: base
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
unpetrify-ref: baserock/morph
⋮
build-depends: []
We extend it to be able to take a "sources" field.
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
sources:
- repo: upstream:foo
ref: baserock/morph # used to validate commit is anchored
# properly and as a hint for where
# you ought to merge changes to
commit: cafef00d…
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The `path` field is used to specify that this tree should be placed in
the parent tree at this position.
The `path` field is optional, defaulting to `.`
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Sub-sources can be placed over an empty directory, when there's no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
If there's a file, symlink, non-empty directory or submodule that doesn't
match `submodule-commit` there, then it results in a build-failure
at build-time.
The `submodule-commit` check exists as a safeguard against the parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
If you get a build fail because the submodule isn't appropriate then
you have to check whether the version you specify works, then update
the commit.
Cache key changes
=================
This shouldn't require any cache key changes to existing definitions,
but builds that make use of multiple source repositories will also hash
the commit tree and path.
Alternative solutions for submodules
====================================
We could continue to use the current model, and deal with the pain of
having to make multiple branches in multiple repositories to satisfy
the change to the repository paths.
We could have a magic look-up table to replace repository urls when we
parse the submodules.
We could use git-replace to magically make git do the url correction, but
then we'd have to handle replacements properly, rather than dropping them.
7 years, 3 months
[PATCH 0/6] Firehose Implementation
by Lauren Perry
Hi all,
Back in November I sent to the list an RFC for Firehose. This preliminary patch series demonstrates the current work I have done, based on the pre-existing prototype Firehose, and the steps taken towards automating this process.
Currently, Firehose is not fully automated; there are still a number of steps that need to be performed manually, such as setting up a Firehose user, enabling the firehose.service and firehose.timer files and installing the firehose shell for general use. These steps are demonstrated in firehose.configure (prelim) and I will post a follow up patch with a README demonstrating steps that need to be taken currently.
Firehose will also need a Gerrit user configured with an appropriate set of ssh keys to access Gerrit.
A Firehose machine has been demonstrated to work properly on OpenStack, though it requires the firehose command to be done manually.
Regards,
Lauren
Lauren Perry (6):
Add tools for installing Firehose and its dependencies
Add systemd styled timers to automate Firehose checking process
Removed morph after placing its functionality in firehose.sh
Modify firehose email configuration and push location to ensure
commits are pushed to the testgerrit server with an appropriate
commit message and branch name
Install githook to add Change-Id value to commit message
Add firehose.configure, containing the necessary instructions to run
firehose
firehose.configure | 39 ++++++++++++++++++++++++++
firehose.service | 12 ++++++++
firehose.sh | 21 ++++++++++++++
firehose.timer | 11 ++++++++
{plugin => firehose/plugin}/firehose_plugin.py | 20 +++++++++----
morph | 17 -----------
setup.py | 16 +++++++++++
7 files changed, 113 insertions(+), 23 deletions(-)
create mode 100644 firehose.configure
create mode 100644 firehose.service
create mode 100755 firehose.sh
create mode 100644 firehose.timer
rename {plugin => firehose/plugin}/firehose_plugin.py (92%)
delete mode 100755 morph
create mode 100644 setup.py
--
1.8.4
--
1.7.10.4
7 years, 9 months
[RFC] First pass at schema for definitions
by Paul Sherwood
Background
----------
there have been various discussions on irc and elsewhere about
improving how we specify definitions, and manage changes to the format.
Sam and others have done a lot to document the current situation [1] and
establish a process for versioning [2]. I also tried to highlight that
we have lots of moving parts [3], which makes forward-and-backward
compatibility hard. Richard also commented about the possibility of
making definitions self-describing [4]
Daniel suggested that we need to properly specify what we have, and
then maintain the specification as we evolve definitions further. I
pushed back on that, because I thought he was referring to a
documentation activity, and over the years we've been relatively
unsuccessful at those. However I now understand that Daniel was
suggesting formalizing the schema for definitions, so that actual
instances of definitions could be validated against the schema, and this
seems to me to be clearly a good idea, hence this RFC.
Implementation
--------------
Some googling led me to the json-schema.org [4], and Julian Berman's
python implementation of validation [5]. As it turns out this is already
present in Baserock build-systems :-) I struggled to understand the
json-schema spec until I came across Michael Droettboom's work at [6].
From there I was able to create a relatively short schema which
describes the current definitions format. I have pushed it as a branch
for definitions at [7], and incorporated the call to jsonschema.validate
into ybd [8]. This schema implementation is not tight - it doesn't check
lots of things yet. I noticed that even making 'name' or 'kind'
mandatory, or forcing unpetrify-ref to be a string identifies
counter-examples in master. So it seems to me that going down this road
we'd be best to start gently and tighten the constraints iteratively.
Incidentally I also noticed that 'definitions' is already a keyword for
the json-schema project.
What do folks think about
- the overall approach
- inclusion of the schema in mainline definitions/schema/
br
Paul
[1] http://wiki.baserock.org/definitions/current/
[2]
http://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2015...
[3] http://wiki.baserock.org/back-and-forth/
[4]
http://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2015...
[5] http://json-schema.org
[6] https://github.com/Julian/jsonschema
[7] http://spacetelescope.github.io/understanding-json-schema/
[8]
http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.gi...
[9]
https://github.com/devcurmudgeon/ybd/commit/2526c79cdb51ba5ec3f3382719c03...
7 years, 9 months
[RFC] Minimisable debuggable systems
by Richard Maw
Hi all.
There's been some interest in building minimal Baserock systems recently.
We currently do this by having split rules in the build-essential to only
include some of the libc and busybox. We can still trim this further though, as
we don't make any effort to strip out debug symbols anywhere.
We still want to be able to debug systems though, so we need to build with the
symbols then strip them out into separate files.
Javier had some success with a chunk that ran the strip commands and placed the
stripped binaries in DESTDIR, which is a neat hack, but it would be more
appropriate to do the stripping effort as early as possible, so the result can
be cached and we can reduce the amount of duplicated effort.
I've written up my thoughts on how to achieve this sensibly in the storyboard as
https://storyboard.baserock.org/#!/story/47
I'd appreciate comments from anyone who is interested.
7 years, 9 months
The minimum core chunks needed to build OpenWRT
by Richard Dale
I'm building OpenWRT which tries to be as small as possible. OpenWRT
needs to be built with relatively obscure architectures such a MIPS.
Optionally I am building OpenWRT with the musl libc library instead of
glibc.
I need to build the core stratum before I can build the various OpenWRT
chunks on top of core. Because I am possibly building with odd
architectures, and odd libc libraries, I need to keep the core stratum
as small as possible and only port core chunks that I absolutely have to.
I created a stratum called 'core-wrt.morph' and only added chunks from
'core.morph' when I found I needed them. These are the chunks I didn't need:
ca-certificates
curl
cython
git
glib
gnome-common
gobject-introspection
gperf
gtk-doc-stub
intltool
libexpat
libffi
libxml2
libxslt
patch
XML-Parser
And these are the chunks that I did need:
acl
attr
autoconf-tarball
automake
bash
bc
bison
bzip2
cmake
cpython
flex
gdbm
gettext-tarball
help2man
libcap2
libtool-tarball
linux-pam
mini-utils
ncurses
openssl-new
perl
pkg-config
readline
shadow
sqlite3
texinfo-tarball
util-linux
xz
The description in the core stratum says the following:
"Core components of a Baserock base system that are build tools required
for the BSP and Foundation strata, but a step above build-essential"
I am not saying that the chunks I didn't need should be removed from the
core stratum, but this mail is intended to be a data point for what
tools are needed for building small systems such as OpenWRT.
I think it should be possible to build languages, such as Erlang on top
of core.morph. At the moment most languages seem to depend on
foundation.morph which is quite large and heavyweight.
There are four different XML libraries in core.morph, none of which I
needed - libxml2, libxslt, XML-Parser and libexpat. Do we really need
four XML libs - what chunks depend on these libraries for building?
I did need pkg-config even though OpenWRT doesn't use any other Gnome
libraries as far as I know. But I would much rather have a stand alone
pkg-config. Otherwise, we've recently added glib, gobject-introspection
and libffi, and those are large libraries that could be problematic to
port to unusual architectures and libc libraries.
I've posted a patch on gerrit to removed git and curl dependencies from
the bison build, and bison is the only chunk that uses git:
https://gerrit.baserock.org/#/c/679/3
The git chunk depends on curl, and so that could be removed from
core.morph too.
I don't know what chunks depend on 'ca-certificates' or 'cython'. But
'patch' and 'intltool' sound like they belong in a core stratum:
ca-certificates
cython
intltool
patch
These could go in a stratum called 'gnome-core.morph'?:
glib
gnome-common
gobject-introspection
gperf
gtk-doc-stub
libexpat
libffi
libxml2
libxslt
XML-Parser
7 years, 10 months
Baserock Contributors Coordination Meeting (BCCM)
by Pedro Alvarez
Hi Baserock community!
Some days ago Emmet suggested in #baserock [1] the idea of "having
monthly development meetings, where people talk about their priorities
for the next month, so that we can have a better idea when various bits
will land."
Some people in #baserock seemed to like the idea, although others seemed
to dislike it a bit. I'm sending this email to create some discussion to
clarify the purposes of these meetings, and see how many supporters
there are.
I've tried to write a draft explaining the purpose, format, and process
of these meetings, and here it is. I hope it's clear, and that the
poeple know that I'm not imposing anything. It might be a bit
incomplete, but I think it's impossible for one person to cover all the
things we need to cover. Please, comment my draft emailing to the list.
Thanks!
Baserock Contributors Coordination Meeting
==========================================
Purpose:
--------
The mains goals for having these meetings are:
* Discuss the priorities of the Baserock project (from the point of view
of the people attending to the Meeting) to identify tasks that need to
be done, and identify the priority of those.
* Coordinate contributors so that they:
- Can work together in big tasks.
- Have sponsors (or people with experience, but no time) assigned to
help them with some tasks.
- Know how can they help in Baserock.
* Getting things done, and pointing in the same direction.
Format:
-------
How these meeting should be? I don't think I have a clear idea yet, but
we should cover the following points on this discussion:
1 - When is the meeting going to be?
2 - Where should we meet?
3 - How long should it be?
Here my suggestions:
1 - First Tuesday of the month, because Mondays are normally busy days,
and weekends are there for other things. At 15:00 UK time (It would
normally be around 16:00 CET and 10:00 EST).
2 - IRC, meetings are fast, there will be less communication problems,
and we don't have to take notes during the meeting. #baserock could be
the right place, although we can do it in a different channel without
any problem.
3 - Nobody likes long meetings I believe, and in my opinion the meeting
is to coordinate us, not to get things done. I think that 30 min could
be a reasonable length.
Process:
--------
I really don't have any idea about how this should be, and here is where
I'll need some help. My idea for this process is to have 2 different
lists of actions, one of things to do during the month, and the other of
things to do during the meeting.
### Actions during the month
* Maintain the doing-stuff-with baserock page [2] (NOTE: for this we can
use a different page, but I thought this page is almost what we need)
- Update the status of the ongoing tasks.
- Add more things to do, if we think they are appropriate (a previous
discussion with the community would be appreciated).
- The tasks should have links to storyboard when appropriate.
### Actions during the meeting
* Someone should run the meeting, but this can be a rotating role.
* List all the ongoing tasks, followed by quick status by task.
- If the owner of the task is absent, we will skip this point.
- The owner of the task will write a brief description of the current
status of the tasks that he is assigned to.
- Brief discussion regarding the task if needed, to determine further
actions like: finding a sponsor, discussions after the meeting, etc.
* Prioritization of the tasks.
- Discussion regarding the priority of some tasks
* Volunteering to do more tasks
- Try to find contributors willing to take some priority tasks
- Try to find tasks for contributors wanting to do something useful,
but not ready for these priority tasks.
[1]
https://irclogs.baserock.org/%23baserock.2015-05-07.log.html#t2015-05-07T...
[2]: http://wiki.baserock.org/doing-stuff-with-baserock/
--
Pedro
7 years, 10 months
Some thoughts on the current 'Contributing' page
by Pete Fotheringham
Hi
Paul Waters asked me to have a look at the current contributing page on
w.b.o [1]. I sent him the following, and he asked me to send to the list
for comments. It would be great if you have time to provide comments by
early next week.
Thanks
Pete
[1]: http://wiki.baserock.org/contributing/
As promised, some thoughts on the current page
I've not emailed to the baserock-dev list, but I have no objection to
doing that, and will stand behind what I've written. Let me know if you
want me to do anything further.
I'm going to start with a metaphor and then extend it way beyond
breaking point :)
If the Baserock project were an international journey from A to B, the
the current 'Contributing' page is a bit like the Owner's Manual for a
car: it can tell you how to start the car, and operate all the lights
and stuff, but it's not not a lot of help if you
- don't know where A and B are and / or
- don't know how to drive and / or
- don't have a passport and / or
- don't have the key to the car and / or
- the best route from A to B involves travel by sea and by air as well
as by road.
It's no help to know how to turn on the lights (or to get an OpenID
account for working with gerrit) if you don't know how to start the
engine (or how to find out what features are missing from current Baserock).
The information that is there is probably both necessary and useful to a
potential new contributor, but nowhere near sufficient, unless perhaps
the new contributor happens to work in the same office - or at least the
same company - as existing contributors, and can obtain information from
them easily and quickly.
Here is a list of what I believe a potential contributor would need to know:
- what tasks need doing? The task might include
- triage reported bugs
- fix bugs
- close fixed bugs
- close implemented feature stories
- specify new features
- design new features
- implement new features
- test new feature
- reviews of
- bug fixes
- new features
- run tests, automated or otherwise
- work with documentation
- wiki.baserock.org
- in the code
- create tutorials
- define more systems
- contribute to discussion in irc and the mailing list
- do they have the skills needed to complete, or to help with, those tasks?
- for coding: python. And ...??
- for other tasks: ??
- what tools and processes does the project use to manage those tasks?
- how to get more information?
Before we could tell the new contributor about those things we would
need to plugs some gaps in our current knowledge of the project. Here is
a list of some things that we don't seem to know about the project (or
maybe we do know but we forgot to write it down):
- what is the project roadmap?
- where is the design of existing software documented?
- what bugs do we have
- that need to be fixed?
- that have already been fixed?
- that won't be fixed?
- what features do we have
- that work, and are tested to work?
- that were implemented once, are no longer tested, and may or may
not currently work?
- that are being worked on?
- that are wanted but not being worked on, and what is the current
state of the work?
- that are not wanted?
- what automated testing do we perform, and how do we do it?
- why would any new contributor want to contribute?
Also, before we can write a better 'Contributing' page, we would need to
know a bit more about potential (target?) new contributors. For example
- what is their interest in Baserock?
- why do they wish to contribute?
- probably they are a user, or their employer is paying them. Are
there any other reasons?
- will they be contributing their own time or their employers' time?
- possibly some more stuff that I haven't thought of.
Anything we do to the current 'Contributing' page (or other pages for
that matter), is just tinkering until we get sort out some of the
questions above. More importantly, I don't think we have a chance of
attracting any new (non-Codethink) contributors until we work out our
story on these matters.
--
Pete Fotheringham
+44 7740 351755
https://appear.in/petefoth
Codethink Ltd
http://codethink.co.uk
7 years, 10 months
[PATCH] Update submodules of 'parted' repo
by Pedro Alvarez
Repo: upstream:parted
Branch: baserock/pedroalvarez/v3.2-gitmodules
Sha1: 9163bb452918067db84b00887e391842f53b932a
Landing: baserock/v3.2
Pedro Alvarez (1):
Update .gitmodules to point at upstream:gnulib
.gitmodules | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
2.1.4
7 years, 10 months
Baserock 15.19.2 release tag created
by Sam Thursfield
Hello
I've created a baserock-15.19.2 tag of definitions.
This is, again, just to update the version of Morph, after we found a
regression in 15.19.1 that would cause various Morph subcommands to
break. It also includes a fix for a potential crash in the distbuild
controller.
These are the only changes from the 15.19.1 tag, so I recommend that
nobody uses 15.19.1. Just go straight to 15.19.2.
As always, please ask on baserock-dev(a)baserock.org if you have any
questions about this release. Or you can visit the #baserock channel of
the Freenode IRC network.
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
7 years, 10 months
[PATCH 0/2]
by Graham Finney
sha1: 48bacf85c3a79a4787d00c164a7679f278dced66
repo: git://git.baserock.org/baserock/baserock/definitions
branch: master
description
AudioManager needed updating and the build command needed changing as a
result. This involved a change to the genivi.morph file to update the
ref, and the addition of a new morph file, audiomanager.morph to provide
the required build parameter.
strata/genivi.morph | 5 +++--
strata/genivi/audiomanager.morph | 10 ++++++++++
7 years, 10 months
