This report covers weekly developments in the moby/datakit, mirage/irmin, mirage/ocaml-git and mirage/ocaml-9p repositories. This week also saw @dinosaure and @eyyub also release two new libraries that replace functionality formerly written in C with pure OCaml implementations:
- Digestif is a standalone cryptographic hashing library.
- Decompress is a zlib implementation in pure OCaml.
Both are now in the Mirage GitHub organisation and will be integrated into Irmin and DataKit over the next few months. The first step is to replace the use of camlzip fully, and then switch ocaml-git from using Nocrypto and GMP to the pure OCaml digestif instead. Irmin and DataKit support for both will follow after that.
There is also an experimental new Git packfile encoder and decoder called Sirodepac to let ocaml-git perform compression of repositories more easily, also by @dinosaure.
There has also been significant progress this week on having a filesystem that Irmin and DataKit can use when compiled as unikernels (and hence only have access to a raw block device):
- Tom Ridge announced the initial release of a formally verified btree filesystem. It still needs to be integrated with the MirageOS filesystem interfaces, so volunteers who want to glue things together are welcome to participate.
- Gabriel de Perthuis also responded to note that he is continuing to work on a flash-optimised pure OCaml filesystem (using hitchhiker trees). It is not quite ready to open-source yet, but he anticipates doing so in the next month or so.
Between both of these efforts, it looks like a fully unikernel-aware, type-safe DataKit that persists onto a block storage device is not too far away.
Finally, we welcome David Udelson, a junior from Cornell who has been selected as a Google Summer of Code student to work on a REST API for Irmin!
This week saw a minor moby/datakit:0.10.1 release of DataKit to support the latest versions of the 9P and Lwt libraries. Both of these have some backwards-imcompatible changes, so the 0.10.1 release of DataKit lets us use the latest features.
- DataKit support for the latest 9P versions was added, including upstreaming the local changes to the 9P library that were only present in DataKit before. (datakit#549 datakit#550 datakit#552 @talex5 @djs55 @samoht @avsm).
- Irmin no longer logs errors on invalid branch names. (irmin#441 irmin#440 @samoht).
- Irmin support for Cstruct contents was fixed to pretty print the raw contents, not the hexdump. (irmin#442 @samoht).
Irmin.Hash.of_stringnow does not raise an exception on invalid hash (irmin#443 @samoht).- OCaml Git got a
commit-add-parentcommand (git#208 @samoht). - Git also switched to using the new Decompress library (git#211 git#145 git#145 git#210 @dinosaure @samoht).
The Jbuilder build porting journey continues to be successful:
- Irmin was ported to Jbuilder, speeding it up a lot. (irmin#444 @samoht).
- OCaml-Git was also ported to Jbuilder (git#209 @samoht).
- The build now also supports Mtime 1.0, which had some API incompatiblities from previous versions (git#212 datakit#554 @dinosaure @samoht).
- The
ogitbinary was temporarily missing, now fixed (git#213 @samoht).
- Work continues on switching to Digestif instead of Nocrypto (which has C stubs) (git#214 @dinosaure @samoht).
- The 9P interfaces are being made safer against leaking exceptions on read/write (9p#126 9p#125).
- The tests, client and server code has been refactored to make transport layer abstract. The default is still to use 9p but this is the first step to replace it by gRPC: datakit#551 @samoht.
Other reports in this series can be browsed directly in the repository at moby/datakit:/reports.