Introducing a per-commit key/value store for Git

Jannis Pohlmann jannis.pohlmann at codethink.co.uk
Wed Jan 2 01:37:01 GMT 2013


Hi all,

I've just pushed the first code for a small utility called "gitpercs"
online. It is a versioned per-commit key/value store for Git written in
Python. What this means is that it essentially allows to associate
arbitrary key/value data with any commit in a Git repository.

The key/value pairs are stored in the same repository in a special ref
(refs/heads/percs at the moment but this will most likely change to
something like refs/percs/default), allowing them to be cloned along
with the real repository content.

What is this useful for? In Baserock, we're planning to do things like
automatic version and license detection for the source code commits that
we build into Baserock systems. Those operations are considerably
time-consuming. Since the content of source code commits never changes,
detected versions, licenses etc. can easily be cached - this is what
gitpercs is for, mainly. It can also be applied to existing features of
morph, e.g. the build system auto-detection.

It can also be used as a cache for rendering websites from content
stored in Git. Websites often re-render templates quite frequently but
when the underlying content is stored in Git, re-rendering is only
necessary when (a) the content changes (new commits) or (b) the
templates change. Storing the rendered HTML using gitpercs (with some
added information to support on-demand re-caching) can improve web
application performance and server load quite a bit.

gitpercs is written in Python for easy integration into morph, other
Baserock tools and Python-based websites. It depends on pygit2 for
convenience and performance reasons though, but I think we should port
morph to pygit2 anyway.

The source code is available at

  https://github.com/Jannis/gitpercs

Please have a skim through the code and comments (esp. the main doc
string for the Store class in gitpercs/store.py). I'd appreciate
feedback to the current design. I reckon especially Daniel might
come up with remarks wrt the usage of Git internals here. ;)

  - Jannis

-- 
Jannis Pohlmann
Senior Software Developer
Codethink Limited
http://codethink.co.uk




More information about the baserock-dev mailing list