Image Registry - How to make our archives addressable and fetchable (Docker Registry vs Nixpkgs vs IPFS vs Dat)

[mokuki082]

Usability, portability.
Running registries vs flat files: command linenix-env -f URL.

[CMCDragonkai]

The new docker registry code (delivered by Docker the company) is here: https://github.com/docker/distribution

[CMCDragonkai]

The consideration of the Matrix AI CIIDD system. Where it's tasked to "build" Automaton artifacts. We need to consider whether we write our own "builder" or we wrap existing builders. A builder is something that takes in imperative instructions and executes those instructions and outputs a "serialised" archive format. It is serialised, because it needs to be portable. (These build artifacts must be transported over the network to target hosts). The CIIDD system must also then transparently cache the building process (so that it can be done with minimal work) when you need to rebuild or whatever. Also content addressable. In terms of wrapping existing builders, this depends the archive format we support. Consider if we support Dockerfiles. This means our CIIDD system must be able to read/parse/interpret the Dockerfile and produce a runc/OCI container artifact. Should we do this by building our own "builder", or should we wrap buildkit? (If we wrap buildkit, this means we bind into it, call it, shell out to it... and expect a runc container archive (that is portable)).

[CMCDragonkai]

Investigations into integration of IPFS and Image/Archive Registries

• docker registries ipfs-shipyard/container-demos#25
• https://github.com/jvassev/image2ipfs/

[CMCDragonkai]

For integration into the Nix system:

• https://hackage.haskell.org/package/libnix-0.2.0.1
• https://hackage.haskell.org/package/language-nix
• https://hackage.haskell.org/package/hnix

This will likely be useful for making the Artifact specification capable of taking Nix derivations/expressions.

With regards to Dockerfile:

From Nix's perspective, Dockerfiles are just another Makefile. And currently Nix deals with this by using make inside the nix builder. You're aware of how Nix builds things right? The nix-build contacts the nix-daemon which initiates a build running inside a "containerised" process using one of the nixbld* users. So if the Artifact specification can both point to finished artifacts and build expressions, one can just use Nix to support both. The former is just a fixed output derivation, while the latter would be a Nix derivation.

[CMCDragonkai]

I have found some other people who is currently working with Docker and Nix integration. In fact their blog post sounds really interesting and I think their library may be useful for us to understand how to integrate container image layers into a Nix store.

• http://hackage.haskell.org/package/hocker
• https://github.com/awakesecurity/hocker
• http://ixmatus.net/articles/hocker-nixos-docker.html
• http://ixmatus.net/articles/docker-without-docker.html

we wanted to maintain the registry’s decomposition of docker images by layer, translating that decomposition (the registry’s image manifest) into something Nix understands

I don't think they are dealing with OCI image yet. But docker v2 should be OCI compatible. Don't be afraid to investigate the source! @mokuki082 @n-zhang-hp

I'm thinking what we'd do is integrate that to help pull in Docker/OCI artifacts from other Docker/OCI registries, then when we push into a nix-store. At this point either our nix-store implementation is already distributed/decentralised/based on IPFS, or that we maintain a distributed binary cache for all the Nodes instead.