RFC: New local fs index.tar-less repository style to simplify "offline" source-distribution

[hvr]

Problem statement

In various situations, there's the desire to provide a self-contained offline buildable source-distribution that (assuming you have a working toolchain installed, i.e. ghc + cabal) doesn't require any network access to build and install a cabal project. This basically requires to include all the dependencies of a project as part of the source-distribution.

In fact, I've hacked up some tooling in the past to accomplish this for my own needs at https://github.com/haskell-hvr/cabal-offline but never got around to polish it and release it properly, as I'm not satisfied with the approach yet.

The way cabal-offline's approach is supposed to work is basically to start from the plan.json file which describes what ingredients the cabal solver deemed necessary for the most recently computed build-plan, and collect the respective .tar.gz files from the local package repo cache (+ any revised .cabal metadata files from the respective 01-index.tar).

Once we have the dependencies' .tar.gz and .cabal files, we need to provide those to cabal's solver. There's basically 3 options here:

1. Point to the tarballs via packages: deps/*.tar.gz
2. Unpack and update the .cabal files, and point cabal to them via packages: deps/*/*.cabal
3. Construct a file-local package repository (with e.g. hackage-repo-tool)

Neither of these options is ideal, and I'll quickly go through these:

Approach 1: Point to the tarballs via packages: deps/*.tar.gz

This only works conveniently for the case where

• (A) There's no revised .cabal metadata required (this can be workarounded by with a allow-newer: *:*+allow-older: *:* combined with a cabal.plan.freeze file which needs to pin down all cabal flag choices as allow-newed/older might break those otherwise), and
• (B) the build-plan doesn't require multiple versions of the same package (this can occur via qualified goals, i.e. for build-tools and custom-setups)

The former (i.e. (A)) can be workarounded (while harming the property (C) mentioned in the next paragraph by repackaging .tar.gz with their revised .cabal file), but the latter (i.e. (B)) is often a real blocker.

(C) On the bright side, dependencies will be properly shared in the Nix-style store, as their sha256sums will be exactly the same you'd have if you had build your project with the real Hackage remote repo. I.e. no recompilation of dependencies needed when you switch between your offline source-distro build and your normal "development" build which might have access to a larger package repo and the build-plan doesn't change.

Approach 2: Unpack and update the .cabal files, and point cabal to them via packages: deps/*/*.cabal

This addresses (A) but not (B), but completely sacrifices (C), as the dependencies' artefacts are no longer cacheable in the Nix-style store, but instead are treated as local packages and lack proper sha256 sums.

Approach 3: Construct a file-local package repository (with e.g. hackage-repo-tool)

This approach nicely addressed (A), (B) while retaining (C). But it's got a few annoyances (which I'll denote as (D)); I'll explain:

So we currently support a number styles of package repositories via the repository directive in ~/.cabal/config and cabal.project files (c.f. https://cabal.readthedocs.io/en/latest/installing-packages.html#repository-specification)

1. Remote non-TUF repositories (this is also what the legacy syntax remote-repo denotes)
2. Remote TUF-secured repositories
3. Local TUF-secured repositories
4. Local non-TUF repositories (c.f. legacy syntax local-repo)

As a side-remark: for the Nix-style build paradigm, the non-TUF repositories are very inconvenient, as they don't provide the tarball sha256sums in their metadata, which the solver phase needs to compute nix-style hashes for the build-plan. This would require the solver to unnecessarily fetch .tar.gzs for the purpose of computing their sha256sums even if they were already cached in the nix-style store and thus we wouldn't have needed to download their source to begin with...

(D) So, one annoyance is that you have to run hackage-repo-tool and construct a redundant 01-index.tar.gz from the set of .tar.gz and revised .cabal files. In order to do that, you also need to create some dummy TUF secret keys to sign the dummy TUF metadata with, and construct a repository-directive with the appropriate root-keys and key-threshold settings in your distribtued cabal.project file. Moreover, another annoyance is that the TUF signatures would expire at some point, and then cabal would refuse to build the source-dist...

So ultimately I don't think that local TUF-secured repos make much sense for this use-case as they come with accidental complexity, and why would you even want to TUF-signatures in the first place for a source-distribution whose local sources you'd typically sign externally anyway by signing the tar'ed up aggregate deliverable source-distro wholesale?

Proposal

Consequently, I propose to combine the benefits the 3 approaches (i.e. addressing (A)+(B), while achieving (C) and avoiding (D)) by providing a new style of index/TUF-less local package repositories, by introducing a new uri schema file+noindex: like

repository my-local-repo
  url: file+noindex:/path/to/local/repo

file+noindex: must point to a folder in the filesystem. The folder /path/to/local/repo/ in the example above (which could also be a relative path such as file+noindex:./deps/) would contain files of the form

• ${pkgid}.tar.gz,
• ${pkgid}.cabal (optional; a corresponding ${pkgid}.tar.gz must exist!), or
• preferred-versions (optional)

Then cabal would scan that folder and compute sha256sums on the fly (which is quite cheap) and construct its package index data-structure (which it could also cache easily).

So the minimal form would be to have the folder contain merely ${pkgid}.tar.gz files, which would already be able to address (B) in the absence of revised .cabal files; i.e. allow for multiple versions of the same package.

If there's also a ${pkgid}.cabal for a given ${pkgid}.tar.gz, then this .cabal file is taken as the package description for ${pkgid}. If it's missing, cabal will extract the .cabal file from the respective ${pkgid}.tar.gz (and ideally cache it).

If there's a preferred-versions file, then do cabal will treat it as if it was a combined preferred-versions file from the index-01.tar.gz (or how it was represented in the legacy 00-index.tar.gz file).

This would considerably simplify the implementation of tooling such as cabal-offline, maybe to the point where we could even simply integrate it directly into cabal proper.

Possible future work

(NB: this is not part of the proposal proper; this merely loud thinking to express possible directions on how this feature could be leveraged)

...imagine a cabal sdist --offline operation; but having this in an companion tool is fine too as I can think of various ways to tweak the offline bundle generation (e.g. take a set of plan.json files for various computed build-plans for different configurations; e.g. for different GHC versions or platforms; and construct an offline source-distro that is able to work on all of these; or have a planjson2bash conversion which is able to emit a bash script for a given plan.json buildplan, and include those in the source-dist -- think cabal's bootstrap.sh; there's lots of ideas to explore here, and these might be better to explore in a decoupled tool rather than bloat cabal and add to its maintenance overhead...)