Hi Sam ! On Thu, 2017-04-06 at 13:23 +0100, Sam Thursfield wrote: Good to hear. We've had this conversation before :) It is just as necessary, although I dont believe stage 3 has really ever been necessary except as a proof that what you've built in stage 2 actually works, this needs looking into. It's necessary for two reason:  * First off, just because we build from a deterministic precompiled    toolchain, does not mean that we do not need to completely seal the    build off from that original toolchain, this is still a bootstrap.    I'm not sure how else to say this to get the point across, this    is quite exactly the same as it was with using host tools, we    drop the original host tools after bootstrapping the compiler so    nothing from the original import is present to link to anyway    beyond that bootstrapping process.    The only difference here is that the host you are bootstrapping    from, are not the ones installed on your host.    NOTE: This is _not_ the same as compiling an alternative gcc in          one shot on your host, that you plan to use on your same          host, the tools we import we are going to throw away directly          after building the toolchain. This is in fact what makes it          a bootstrap.  * Second, is that we should be able to refine this bootstrap so that    we can use it to cross-bootstrap for any target arch we support.    For this reason I hope to nuke stage three.    All we really want in the bootstrapping of the toolchain is:      1) Build a cross compiler for ${target}, even where target is         the same arch as ${host}, we create this cross for the sake         of isolation and bootstrapping (same as Baserock currently         does)      2) Use that cross compiler to build a native compiler for         ${target} (this is essentially a "canadian cross")    If we remodel build-essential (now 'gnu-toolchain') to work this    way, and never actually *use* the tooling built in stage2 during    that build, we can then take this binary output and use it to stage    in a slightly different build sandbox. A build sandbox which not    only uses bubblewrap for sandboxing, but additionally uses qemu    user mode to continue from that point on and cross compile using    virtualization. Hosting an OSTree along with the trove I think is the most optimal solution for this yes. OSTree is interesting because it saves space with the way it revisions filesystem trees (so when things change in this base, you save space over other methods of storage). However, there is nothing preventing you from using tarballs or even git for this purpose, we've tried similar before using git and it works, it's just not an optimal solution. Yes, the base is unnecessarily enourmous, in fact the org.freedesktop.BaseSdk from that same repository should be much, much smaller and also sufficient for bootstrapping. With that said, for the Baserock project I'm much more interested in hosting the gnu-toolchain binaries we build ourselves, and using those (or alternatively a binary of the converted 'foundation.morph'). We've again had this conversation before :) I think it will be interesting to make this circular and build only from what we've built before, for two reasons:   1) It proves that what we build, can be built with what we've built.   2) It makes Baserock entirely independent of any other system out      in the wild. For (2), it is particularly important to have this freedom moving forward, otherwise we are simply trusting that either:   a) We will never require a newer compiler than say, gcc5, in order      to compile whatever version of gcc will exist in 10 years from      now.      But we know this is sort of untrue, parts of gcc are now written      in C++, maybe gcc version 12 will be written in rust, or whatever,      nobody knows.   b) That somebody else will always be around to bootstrap a base       system for *us* to bootstrap from, for any host arch we care      to build on. I dont like either of the above (a) or (b), and there is no reason why Baserock's build instructions themselves cannot be entirely responsible for creating everything it needs. The trick is only that we need some x86 binary sysroot the first time around to start bootstrapping and cross bootstrapping from an x86 host, once we have built it once, there's no reason to depend on anything else at all. This is a little tricky to respond to but I'll try. Bare in mind that the truth is we dont need to revision anything until the first time we break compatibility. So, the tricky part with versioning in buildstream is that buildstream is not one monolithic tool but rather a system for integration of various elements. So instead of thinking "BuildStream format 1.0", think "autotools element format 1.0". This is true for the most part, except for some fundamental things in the BuildStream loader (dependency expression, variants, variables etc). For first class citizen elements of BuildStream (i.e. element plugins which we maintain inside the actual BuildStream repository), the stance is that these elements should always support every format which has ever existed, and that any project needs to declare the format version they intend to use, both for the overall format and for any element specific versions. With that said, breaking format compatibility is something we should avoid, I would aim for a cut off date for an ultimately "stable" release of buildstream this summer around GUADEC and from that point attempt to never break it; If, and only if we need to intruduce a feature that requires a format change, BuildStream must absolutely continue to support previous versions, and projects which want to use the newer format need to declare it in their metadata. This should be mostly outlined in the roadmap here:   https://wiki.gnome.org/Projects/BuildStream/Roadmap/CrossBuilding Regarding time and commitment, I think we're going to have a clearer answer to that in around 2 or 3 weeks (some meetings are scheduled with an interest towards solving the cross building problem, especially with an eye towards never depending on there ever having been a build for that target arch ever before, i.e. highly custom arches for which you may need to build support for yourself into the kernel). On my part, I have made a huge commitment to make this fly and solve problems in GNOME, so I dont see myself having time to work on cross building until some time after GUADEC. That said, there is a huge chance that resources will be allocated to that much much sooner, and I very much hope it will be YOU who will work on it :D Cheers,     -Tristan

On Thu, 2017-04-06 at 15:44 +0100, Sam Thursfield wrote: Because the base is going away as soon as the toolchain is bootstrapped. Performing the bootstrap has much less to do with how much we know and trust the tools we build with than it has to do with, well, bootstrapping. If you havent built gcc and glibc in such a way that they can run reliably without the base that you've built them with, how can they run once you take that base away ? Sure, however it should be ridiculously simple to wrap up the output of a foundation build and start using that as the basis for bootstrapping instead of a foreign built toolchain ultimately from yocto, and that small effort automatically proves self reliance. The effort here should be quite low, but granted there is no fire in the building right now. This is a good question, it seems to me that it would be possible that some misguided souls try to use BuildStream to perform full stack cross compilation, and it might be their misfortune that BuildStream has not put any checks in place to prevent that. That's entirely my opinion, however. It seems to me the current architecture can indeed be used to stage things in such a way to make full stack cross compilation possible. I see what you mean regarding leaving the choice open for users, but I do strongly disagree that some projects "cross compile well", especially when we are talking about projects that build a full stack operating system. Just this week I had the misfortune of trying to pry a patch through the yocto bureaucracy because, nasm is required on the host in order to build jpeg-turbo, but only when targeting intel architectures with jpeg-turbo. Simply maintaining these conditionals across two sysroots is time consuming, and for what benefit ? This has happened before, and if we dont solve it, it will happen again, the buildroot project actively sucked the life out of the uClibc _and_ busybox projects when that was a thing, and today I'm seeing so, so many developer hours getting poured into this "busy work" that is the maintenance of yocto. I'm not saying that BuildStream _today_ solves all these problems, what I am saying though; is if you were to take all of those man hours focused on maintaining dual sysroot builds and patching modules to "behave in cross compilation", and if you were to divert a fraction of those hours into just making virtualized cross builds work cleanly, then you will have saved the world countless developer hours which could be spent actually doing useful things, instead of being bogged down in the friction of full stack cross compilation. Cheers,     -Tristan

On Thu, 2017-04-06 at 17:26 +0100, Sam Thursfield wrote: [...] Not sure we understood eachother on this point but; we only really need to commit/store binaries for host architectures we intend to support cross bootstrapping *from*, which can easily just be x86_64, once we've solved the cross bootstrap story. Once we have a working cross bootstrap, the beginning of all build stories becomes "cross bootstrap on x86_64 using this binary sysroot as a base", and then the artifact caching takes care of the rest; once you have cross bootstrapped something for arch "foo", then that runtime is already in the artifact cache. Does this make sense ? Admittedly, even if we want a fully natural native build on non x86, we need some base to start with on that host as well, but this should hopefully be easy to solve if that base is available in a shared artifact cache too. Cheers,     -Tristan

On Fri, Apr 7, 2017 at 4:53 AM, Sam Thursfield < sam.thursfield(a)codethink.co.uk&gt; wrote: I think that's fair. Users can start off with x86_64, cross-compile the bootstrap to the target, and then finish up on the native architecture to get a full system. For most cases that I can conceive, we can assume that there is a bootable kernel available - but, it won't have the userspace set up to permit a Baserock distro build. Hence, we need to produce either the userspace/chroot so that we can then go native. Once you have the cross, then you can be native from that point forward. Cheers. -- justin