[RFC] Multiple source repositories per-chunk and git submodule handling
by Richard Maw
We currently handle definitions that need to pull multiple source
repositories together in an ad-hoc way.
For gcc we import the gmp, mpc and mpfr source trees in by checking them
into our delta branch directly.
Some upstreams handle importing the sources by adding submodules.
However, we have to make them fetch sources from our Trove for
reproducibilty and latency concerns, so we end up having to add a delta
on top to change where the submodules fetch their sources from.
This works against one of our goals for minimising the delta, so we
need a way to encode in our builds how to deal with components that need
sources from multiple repositories to function.
Our current approach from submodules introduces a _lot_ of extra work
when we need to update multiple submodules recursively, so we need a
better solution.
Proposal
========
To solve this, I propose we extend the source repository information from
just (repo, ref) to be a list [(repo, ref, path?, ?submodule-commit)].
So rather than:
name: base
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
repo: upstream:foo
ref: cafef00d…
unpetrify-ref: baserock/morph
⋮
build-depends: []
We extend it to be able to take a "sources" field.
⋮
chunks:
- name: foo
morph: strata/base/foo.morph
sources:
- repo: upstream:foo
ref: baserock/morph # used to validate commit is anchored
# properly and as a hint for where
# you ought to merge changes to
commit: cafef00d…
- repo: delta:gnulib
commit: deadbeef…
ref: baserock/GNULIB_0_1_2
path: .gnulib # where to put the cloned source
submodule-commit: feedbeef… # check that this matches, so you
# can be told if you need to update
# your delta
⋮
The `path` field is used to specify that this tree should be placed in
the parent tree at this position.
The `path` field is optional, defaulting to `.`
If multiple paths, after normalising, clash, then it results in a
build-failure at source creation time.
Sub-sources can be placed over an empty directory, when there's no
existing entry at that path, or when there's a git submodule there AND
the commit of the submodule matches the `submodule-commit` field.
If there's a file, symlink, non-empty directory or submodule that doesn't
match `submodule-commit` there, then it results in a build-failure
at build-time.
The `submodule-commit` check exists as a safeguard against the parent
repository being updated and requiring a new version, which your specified
commit isn't appropriate for.
If you get a build fail because the submodule isn't appropriate then
you have to check whether the version you specify works, then update
the commit.
Cache key changes
=================
This shouldn't require any cache key changes to existing definitions,
but builds that make use of multiple source repositories will also hash
the commit tree and path.
Alternative solutions for submodules
====================================
We could continue to use the current model, and deal with the pain of
having to make multiple branches in multiple repositories to satisfy
the change to the repository paths.
We could have a magic look-up table to replace repository urls when we
parse the submodules.
We could use git-replace to magically make git do the url correction, but
then we'd have to handle replacements properly, rather than dropping them.
7 years, 6 months
[RFC] Some thoughts about rawdisk.write, partitioning and upgrades
by Pedro Alvarez
Context
-------
Now that the Partitioning support work is merged (Thanks Ed!)
we are able to generate disk images with different partitions
containing some parts of the systems, auto-generating the fstab
file to mount these partitions, etc.
With this work, I'm able to generate a disk image for the
Raspberry Pi, creating one partition for `/boot` and the rest
in the second partition. On a fast attempt of getting it working,
I created the `config.txt` file needed for u-boot to know what to
do, at build-time of one of the chunks of the BSP (this is obviously
wrong), under the `/boot` folder, so that it ends up in the `boot`
partition with everything else (u-boot, kernel, etc.).
Today I've been researching about using a `extlinux.conf` file to
configure the bootloader in the Raspberry Pi, on the same way we do
with the Jetson-tk1, and it looks like it should be possible. The
reasons to do this is to support Baserock Upgrades on the Raspberry
Pi, rollback, etc.
After investigating about all of this I found that our current
situation is far from perfect, and I'd like to point out here
what I believe needs more thinking and some rework.
Current status and problems
---------------------------
There is no way of specifying where to put the `extlinux.conf` file,
it will always go to the main partition. Jetson-tk1 deployment fixes
this problem on its flashing script. This script creates the partitions,
and copies files to the right place.
There is a variable `BOOT_DEVICE` that is only used for the Upgrades
scripts (system-version-manager), but you need to specify it in the
cluster morphology of the system that you want to make upgradeable.
This variable doesn't do anything else, and of course it might have
wrong values if used in combination with the partitioning support, or
if you use it when deploying a system that won't have any partition,
given that this is currently possible and doesn't trigger errors or
warnings. This variable will only be used for the system that is going
to be upgraded, when receiving an upgrade. If this variable is not
properly set, your system will fail to upgrade.
There are probably more problem, but I didn't have time to investigate
them all.
Possible solutions
------------------
Use `BOOT_DEVICE` in combination with partitioning to install
extlinux.conf and kernel in other partitions?
Install extlinux.conf in the partition `/boot` if exists and
if not default to `/`?
Create a variable in the partitioning files to specify where
to create the bootloader configuration files and kernels?
Put extlinux.conf in `/boot` by default and create a symlink
from `/` to it to make it work when no partitions? This idea is
probably the worst one. The file should be created **before**
moving things to partitions, otherwise it won't end up in the
boot partition if created.
There are probably more possible solutions for this, although
I think that `extlinux.conf` is not the only file, and we also have
to take care about the kernels.
Ideal solution
--------------
Developer wants to put `/boot` in another partition. `extlinux.conf`
and kernel are created there.
Developer want's to upgrade the system. system-version-manager knows
what partition to mount when writing new `extconfig.conf` file and
kernels.
BOOT_DEVICE and ROOT_DEVICE are not required for this, and
partitioning should create enough metadata to get this right.
If the system is deployed without partitions, and then flashed to a
board, is up to the flashing script generating the metadata needed
to make system-version-manager work properly (the replacement of
BOOT_DEVICE and ROOT_DEVICE)
--
Pedro
7 years, 7 months
GNOME on Baserock - Milestone 2
by Tristan Van Berkom
Hi all,
This is another status update on GNOME on Baserock
(gnome-system-x86_64). It took a little longer than expected, as gdm has
been up and running for some time, however enabling graphical boot by
default would have left you in a situation where you are locked into the
desktop with no terminal, or worse, you would just not have any user to
get through gdm's login screen.
So, instead I postponed this update until there was a pretty functional
user experience.
At this point, assuming that the fix-samba-build[0] and
gnome-control-center[1] branches have landed in definitions by the time
you are reading this, you should get a pretty user friendly experience
when booting the image.
Initial Setup
~~~~~~~~~~~~~
When booting the image for the first time, you will automatically boot
into the GNOME Initial Setup user experience[2], where you will choose a
system language, keyboard layout, privacy settings, setup some online
accounts and finally create a username and password.
Without any effort on my part, the final splash screen when completing
the initial setup will say "Welcome to Baserock" with a nice fat button
which says "Start using Baserock", or, it will say something else if you
have not selected an English locale, or course.
Currently the only noticeable problems with the initial setup are with
the online accounts; the icons (for google, facebook, owncloud, etc)
seem to have gone missing in the last build; and when trying to create a
google or facebook online account, you will get a message about an
unacceptable TLS certificate, some more integration-foo is needed in
that area.
After Logging In
~~~~~~~~~~~~~~~~
Once you log in, you will have a "quite" usable GNOME Shell UX.
The shell seems to respond well and nothing seems to be missing after a
few passes. When browsing your installed applications, you will have
access to "Terminal" (gnome-terminal), and from there you can simply
type "su -".
The user created by the initial setup tool is not a sudoer, but for now
we still have a passwordless root account, you can just give it a
password and ssh into your box in the normal way afterwards, or do
anything you want to your system from there.
You have your basic graphical file browsing experience provided by
nautilus which works nicely and GNOME Control Center is also installed,
so you don't have any useless buttons provided by the GNOME Shell:
everything exposed by the shell does something.
Asides from some missing icons in the control center, it seems to behave
quite well also, you can go ahead and set your wallpaper and lock screen
backgrounds without issue, also it withstands a language change quite
well (telling you that you need to restart the session for the change to
take effect - gdm's login screen wont update it's language until the
next boot but I suspect that is an upstream issue).
Next Steps
~~~~~~~~~~
What we see with the GNOME system so far seems to work well but requires
more testing - there are issues with online accounts and the journal
shows that geoclue lacks permission to access some services. I was
unable to test with internet access from the VM so far but I suspect
that the Initial Setup would be able to guess my location automatically.
I would like to iron out the kinks with the currently installed basics
before continuing to installing applications. As more GNOME components
get integrated properly, we gain momentum and integrating the higher
level bits gets easier as they receive a more GNOME friendly environment
to run in.
We should soon also be ready to pursue CI and automatic production of a
"Live CD", but I would really appreciate some feedback on this front, as
nobody has replied yet to my last email on that subject[3].
Stay tuned for the next episode of GNOME on Baserock, which should have
the remaining integration kinks ironed out and add a new stratum with
GNOME applications.
Best,
-Tristan
[0]:https://gerrit.baserock.org/#/q/topic:fix-samba-build
[1]:https://gerrit.baserock.org/#/q/topic:gnome-control-center
[2]:https://wiki.gnome.org/Design/OS/InitialSetup
[3]:http://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2...
7 years, 7 months
Permission to push images in http://download.baserock.org/
by Javier Jardón
Hi!
As the GNOME system is getting in shape thanks to the amazing work of
Tristan, Id like to generate a GNOME baserock image I can offer to the
community to try the latest releases
This is a problem the release team always have had as the only options at
the moment are:
- Tell the users to build everything themselves
- wait for distros to package all the components. This is not even going to
happen for unstable releases
We can put the image in a personal server somewhere but I think it would be
interesting for baserock to host them; if you agree, what is the process to
push something there?
Cheers,
Javier
7 years, 7 months
[RFC 0/2] Trove URL redirect and Cgit URL addition filter
by Lauren Perry
Hi all, I've created a couple of patches to trove-setup. The effect of these should be to remove cgi-bin/cgit.cgi from all trove URLs and redirect appropriately, and to add a cgit filter for stratum morphologies in definitions.git that will add a hyperlink to the repo, ref and unpetrify-ref values, linking to their appropriate locations in the trove.
I'm currently upgrading my test trove to test these changes, however if anyone has an active trove currently they would like to test with, that would be fine!
Lauren Perry (2):
Redirect cgi-bin/cgit.cgi urls and remove the index refresh
Add hyperlink filter script and reference in cgitrc
etc/hyperlink_filter.py | 53 +++++++++++++++++++++++++++++++++++++++++++++
etc/lighttpd/git-httpd.conf | 2 ++
http-assets/index.html | 1 -
share/etc/cgitrc | 4 ++++
4 files changed, 59 insertions(+), 1 deletion(-)
create mode 100644 etc/hyperlink_filter.py
--
2.1.4
7 years, 7 months
Proposal: you should be able to list chunks in a stratum in any order
by Sam Thursfield
Hello
I raised this on gerrit.baserock.org already, but since it is a change
to the definitions format, I want to mention it here as well so that it
reaches (hopefully) everyone that cares about the definitions format,
not just those reviewing Morph patches.
Quoting from <https://gerrit.baserock.org/1313/>:
"Currently Morph enforces that chunk A must be defined before anything
that build-depends on it. YBD doesn't enforce that.
Definitions format at <http://wiki.baserock.org/definitions/current/>
doesn't mention ordering currently. I propose that we make Morph be
permissive about ordering, like YBD is, and update the spec to mandate
no restrictions on ordering.
Since behaviour was previously undefined, making Morph be more
permissive about this shouldn't require a new version number of the
definitions format.
I still think we need to make sure stratum .morph files are ordered
logically, but that is in the realm of 'code style', it shouldn't be
being enforced by a build tool."
Let me know if you agree or disagree with this! If the patch to Morph is
merged, I will update the definitions format specification to say that
chunks can be any order within a stratum.
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
7 years, 7 months
Baserock VIMEO channel admin
by Agustin Benito Bethencourt
Hi,
does anybody know who is the admin of Baserock VIMEO account. I am
interested in the videos published an hour ago in the channel.
https://vimeo.com/baserock
Best Regards
--
Agustin Benito Bethencourt
agustin.benito(a)codethink.co.uk
7 years, 7 months
[RFC]: Deprecating lorried repositories
by Richard Maw
This document is also recorded in the storyboard as
https://storyboard.baserock.org/#!/story/62,
but this may be a more comfortable place to comment.
# Deprecating lorried repositories
Sometimes we lorry changes, then later no longer need that repository,
because the change gets moved into its primary repository,
or it was part of work in progress that was abandoned before completion.
We'd like to be able to:
1. Stop polling its upstream repository
2. Hide the deprecated repository from listings by default,
so new users don't start using them.
This needs to be hidden by both the cgit UI and the gitano ls,
so downstream troves also don't mirror deprecated repositories.
3. Potentially have downstream troves also deprecate their repositories.
4. Reclaim any used space.
5. All while still allowing old URLs to work.
## Stopping future polling
This is currently already possible by taking the lorry file out of
the active set.
## Hiding deprecated repositories
This already partially possible, since cgit has the ability to hide
repositories from listing, and gitano will hide repositories with
project.archived set.
Propagating this archived flag would require lorry controller to be able to
query this config in a more performant way than running `config $repo show
project.archived` for every repository.
This information is stored in the admin ref, so you could trigger some merge
based on whether that has changed, but you most certainly don't want to fetch
it in directly (and can't with the default configuration).
## Reclaiming space while allowing old URLs to work
Without any magic or knowledge about which repositories share space, the best
that can be done to reclaim used space would be to aggressively repack when a
repository is deprecated.
If you *do* know some repositories that are likely to contain shared objects,
such as projects that split out components, or forks, then there's a few
approaches possible:
### git-relink to de-duplicate objects by hardlinks
`git relink` will de-duplicate loose git objects,
but given objects tend to be packed, this may have limited success.
### Sharing object stores
You can use `.git/objects/info/alternates` to have the fork/child-project
repositories share the common history objects with the parent repository.
Doing this post-hoc is more awkward:
1. Create a repository containing all the objects and refs,
potentially by nominating a master and fetching all the refs in
a namespaced manner.
2. Duplicate this repository for each of the fork repositories.
3. In each repository, remove refs that aren't supposed to be there.
4. In the primary repository, `git gc --prune=all`
5. In all the forks, set .git/objects/info/alternates to point to the
primary repository's object store, so that they can look for
objects in the other repository.
6. Perform a `git gc --prune=all` or a `git repack -l` in forks
to discard any objects that were available locally,
but are also in the alternates object store.
After this, the fork repositories will share all common objects
that came about because the forked branch was merged,
but if the history is removed from the primary repository,
then the forks will stop to function.
### Merging forks into the same repository with namespaced branches
The unified repository that we would create as a step in sharing object stores,
could be used directly with some work,
if the namespaced manner is proper git namespaces.
You would then be able to have logical repositories by performing git
operations on physical repositories with only the branches in a different
namespace.
Either the git-{{upload,receive}-pack,upload-archive} commands need to be made
natively aware of how to locate the physical repository,
when run in a stub repository,
or you would have a proxy service run before the git service process,
which redirects the operation to the physical repository
by changing the git directory
and setting GIT_NAMESPACE to the name of the logical repository.
This approach has the advantage over the alternates approach because it won't
break everything if you delete refs from the physical repository and repack.
#### Handling http[s]:// URLs
git-http-backend's manpage has examples showing how have all
repositories matching a given path be shared, this wouldn't work for us,
as we need it to be more dynamic, but it proves the concept,
so we'd have a proxy cgi that asks gitano how to find the parent repository,
then munges the paths and sets GIT_NAMESPACE before exec'ing.
Naturally we'd need to run this service rather than using cgit's version.
#### Handling ssh://git@ URLs
Similar redirection could be added to gitano's processing of the commands,
though it may require a rethink of which admin-ref is checked.
#### Handling git:// URLs
The standard git-daemon would either need to be wrapped in -i mode,
or re-implemented, to have it proxy to the physical repository.
The protocol is fairly simple though, with a big-endian hex-encoded
header length 4 bytes, followed by an optional NUL terminated
host=$HOST:$PORT of what the client thinks the server's address
is (to allow for virtual hosting), followed by the service name
(e.g. git-upload-archive), a space, then the path part of the
git:// URL.
After this the service subcommand should be run from the git directory
with the stdio proxied through.
So it would be feasible to have a program that parses the header,
determines what namespace and repository to use, sets GIT_NAMESPACE,
forks off the git-daemon in -i mode, writes a new header, then
proxies the stdio of the client into the git-daemon.
#### Handling the CGit web UI
For CGit, the repositories need to either be hidden,
or it needs to be made aware of how to show the logical repositories,
since there is currently no way to hook this in,
and CGit is not ref namespace aware.
7 years, 7 months
[RFC] definitions change: make build-system key mandatory in chunk morphs
by Richard Ipsum
Hello all,
Sam commented on https://gerrit.baserock.org/#/c/1213/ suggesting that
any definitions updates should be suggested here, so here goes...
Change 1213 is an RFC for a suggested tweak to the definitions format,
it seems more natural to me that if the build-system key is required for
chunk definitions that do not hold chunk morphs then the build-system
key should also be required in the chunk morphs themselves.
I've noted in change 1213 that my motivation for this change is that
I am frequently tripped up by the tooling assuming that when I don't
define a build-system key I really mean build-system: manual, when more
often than not I'm trying to override some subset of a given build-system.
I do not assume I know what is best for the format overall but this change
seems small enough that I feel comfortable submitting it for your consideration.
The RFC happens to come with an implementation that would update morph,
I can fashion an equivalent patch for ybd if required.
Thanks,
Richard Ipsum
7 years, 7 months
On CI for GNOME and other Baserock systems
by Tristan Van Berkom
Hi all,
One of my goals with the GNOME baserock system is to achieve some
continuous integration setup for the GNOME system, hopefully generating
some automated deployment of the latest bleeding edge GNOME image and
making these images available for people to easily test the latest
GNOME.
To this end, I'd like to get the opinion of those working on CIAT and
try to assess how far we are from achieving this in some form.
Below, I've listed some of the steps I imagine will be involved in the
CI process, I would be grateful if those working on CIAT could both
clarify the status of how far we are with these and also correct me and
explain how this is intended to work from your own point of view.
Best Regards,
-Tristan
Pulling latest git revisions from upstreams
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>From what I understand, there is already a system in place to check the
upstreams for a given "chunk" and modify a local definitions checkout so
as to point to the latest version.
a.) Is this already solved ?
b.) Does it only work for master, or is there some way - perhaps based
on a chunk's 'unpetrify-ref'; to consume the latest revision of
the active branch desired for the given chunk ?
We want the images to actually boot and work, so I think 'b' is
critical, currently definitions has master branches here and stable
branches there, a sort of melting pot that needs some cleanup but at
least with the integration work I've been doing; we will be starting
with a baseline that is known to successfully boot in to GDM, launch
gnome-shell and have applications and desktop applets that generally
work.
Handling git submodules
~~~~~~~~~~~~~~~~~~~~~~~
Currently we have a hand full of downstream deltas in the trove to
modify the .gitmodules files, obviously this is not practical and might
be a stumbling block for CI.
In any case, downstream deltas need to be considered irregardless of the
submodule situation.
Do we plan on addressing the submodule situation for the purpose of CI,
is there already an approach for this in place ?
I could imagine a couple of approaches... as the lorries repository
already serves as a database to identify and swap a submodule for an
existing clone in the trove this seems quite solvable.
a.) Possibly we patch the submodules after checkout in ybd's
build phase directly
b.) Possibly we invoke git in a special way (not exactly sure
how)... so that we can override the method to clone a submodule
on git's behalf; fetching the matching trove module instead.
Applying our downstream deltas
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Asides from submodules, there will surely always be some cases where a
downstream patch is required in order to achieve a fully functional
build.
One which I encountered recently while integrating libpwquality:
https://git.gnome.org/browse/jhbuild/tree/patches/libpwquality.libintl-li...
Currently we build with --disable-nls and avoid the patch, but this
disables the translations; resulting in untranslated strings such as
"This password is too weak", which can be embarrassing when booting up a
GNOME image for the first time, after choosing your preferred language
and proceeding to set a password for the new user in gnome-initial-setup
(https://wiki.gnome.org/Design/OS/InitialSetup), which I'm trying to
integrate for my next milestone (almost there).
Considering that the CI must operate on a separate repository (or
branch) of the definitions repository, and that it is not an option to
change git.baserock.org/delta in a way that breaks our upstream mainline
definitions builds, I am curious as to how we will handle this.
A.) Will the CI use an entire duplicate trove (expensive) ?
B.) Will the CI be creating some special branches in
git.baserock.org/delta repositories for the purpose of trying and
testing a build ?
In either case, the goal is of course to rebase our downstream patches
against the head of an upstream repository *without* effecting the
results of a regular build (CI builds should remain encapsulated and
running them should not compromise our production builds).
Consuming definitions changes from CIAT and upgrading definitions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Once CIAT has successfully built and tested a given build, this build
should be marked somehow as ready/passed so that baserock maintainers
have some way of integrating the results into our upstream production
definitions repository/branches.
I think the idea thus far is to have CIAT submit patches to the
definitions repository over gerrit.
Anyone care to clarify and elaborate on what exactly the plan is for
this on the mailing list ?
Configuration(s) of CIAT's CI instances
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
One thing I think we're going to need from day one in order to produce
booting and well running GNOME images, is a way to configure our GNOME
CI instance so that we can select which "stratum" are eligible for CI.
In other words, it would also be fun to try to build every single thing
from master every time - but I would prefer to start with a more
conservative approach and only pull new chunk revisions for specific
stratum (i.e. lets build master of all gnome.morph modules against a
stable system and have more useful images as a result).
NOTE: I did try for example, to upgrade the X server and all X libraries
in one shot when upgrading libXi a couple weeks ago, however I was
unable to get the new X server to start, I'm not sure why; but I would
rather start with automatically generating something that mostly always
works for the purpose of GNOME image testing.
In the current setup thus far, is there any mechanism in place to
configure a CI instance to pull new revisions of modules selectively,
such as "only update gnome.morph chunks" ?
If not, can we please have this ?
Publishing
~~~~~~~~~~
Once this is all up and running, we will want to provide an easy way for
people in general to download a "GNOME nightly" image to try it out on
any given day.
Do we have a plan on how publishing these will work ?
Just have the CI instance make the successfully built images available
in a shared directory somewhere the public can download from ?
7 years, 7 months
