Hi all, I'm trying to cut some fat and reduce complexity here, and making (yet another ?) case for throwing away git tree shas from the cache key algorithm in favor of the git commit sha instead. I have looked through the mail archives and found not much conversation on the matter, I did find this: Richard Maw writes:   "Frankly, to get all the way out of needing to talk to git during the    build graphing operation, we would need some lookup between commit    sha1 and tree sha1 anyway, since we use the tree in the cache key,    not the commit." https://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/201... In any case, I am mostly curious if people on this list have first hand knowledge of just how much disk space on the artifact servers we save by trading away the simplicity of just using commit shas. The decision should be simple, if we can maintain a simpler process and consequently a simpler and more straight forward code base by removing this, we should. Unless the cost is truly demonstrably high: We should not be bending over backwards to use these git tree shas. Below is some more text on the subject, clearly I want to kill the git tree shas. Cheers,     -Tristan The benefits of using the tree sha are: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   o Two commit shas may possibly point to the same tree with identical     sources.     So in the case that definitions are modified to point to a new     commit that is the same tree, and in that definitions set; the said     modified chunk has all the same dependencies which they themselves     have not changed, this would potentially result in the caching of     2 identical artifacts under differing cache keys.     In other words: It can happen approximately never. Now I can in fact think of one situation where it could happen a bit more often than never, but it comes with a counterargument too. Imagine the following steps: 1.) There is some CI which triggers a build on every push to every     branch in a given definitions git repository 2.) The users are working on a wip/<username>/<project> branch in     an upstream gitlab or trove. 3.) A commit is pushed to a wip branch in definitions, pointing     to a wip ref in an upstream gitlab or trove. 4.) After CI passes, the upstream gitlab gets its wip branch     now merged 5.) A new merge request is created to definitions to point to the new     merged upstream commit. At this point the second time we push a definitions commit which points to the newly merged wip branch in the upstream, it will point to the same tree. Using the git tree sha, if definitions have not changed between (3) and (5), we successfully avoid a duplicate artifact. That is, unless the upstream also has merge handling like gitlab or gerrit which adds an additional informational commit telling a story about why the upstream wip branch was merged, in which case the trees would differ anyway. So, this case exists, and it happens a bit more than approximately never, do we know how much ? Is the optimization worthwhile ? The benefits of using the commit sha are mostly obvious: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   o Ability to make cache key calculations with only the set of     target definitions, without having access to gits or "tree     servers"   o Less moving parts; if for example we would build GNOME modules     with ybd/definitions, why would we bother with a trove for the     GNOME modules when they are all hosted at git.gnome.org ? Would     we have to implement a tree server to optimize the builds ?     Same goes for some infrastructure and CI setup with gitlab,     does gitlab provide it's own tree server ?   o Reduced complexity in surrounding tooling, code is more presentable     and readable and maintainable without this, the operation looks     questionable and convoluted.     Not only does it make the surrounding code complex (cache key     calculation code depends on code which manages git mirrors),     but the operation itself is complex. Mitigation of damage ~~~~~~~~~~~~~~~~~~~~ Now to go a bit deeper, should an artifact server run for years on end I can expect it to consume loads of disk space. Especially if we're caching artifacts for every wip branch ever made, instead of only long term caching the artifacts which are relevant to definitions master and a set of registered release branches (of definitions). This disk space consumption might be slightly increased were we to ditch the cumbersome git tree lookups. On the other hand, if we did not have to use the tree sha and could use the commit sha itself to calculate the cache keys, it would be relatively straight forward to write a pruning or culling algorithm, with the inputs:   o definitions git repo   o list of branches of importance to track   o artifact cache directory A script could be made to iterate over the commits in the tracked branches of definitions and generating the keys for each artifact that an "important" branch could possibly generate; while removing artifacts from the server which are of no relevance (accumulated cruft from various merge requests throughout 10 years of history). The above could be made more fancy as well, i.e.: for each release branch, keep artifacts from the latest release tag -> HEAD, and keep artifacts for every other release tag since the creation of that branch. Yes, we *could* implement the above with access to all of the gits under the sun present at the same time, but refreshing all the git mirrors in order to calculate trees is in itself time consuming, and we would be racing against the next definitions commit to do so.