Support overlay/union-mounted stores

[catern]

There are three motivating ideas:

• Private stores in a home directory would be much more efficient if they could share artifacts by being union-mounted on top of the system-wide store. If private stores became much more efficient, they could be truly viable as a way to solve the secrets problem: with a union mount, a private store created for secrets would need only hold the actual secrets and top-level composition, making them vastly smaller than the current situation.

• Likewise, more efficient private stores would allow individual users more flexibility when prototyping. Many of my users want to build things impurely while they are prototyping. This is impractical at the moment since they would need to copy all their dependencies into a private store, but if they could efficiently create private stores, that would open up new kinds of experimentation. Similarly, it would allow users to more flexibly trust new binary caches that aren't trusted system wide, copy unsigned artifacts around without being trusted users, etc.

• Sharing a single store between hosts by mounting it with NFS and managing it with the daemon could improve performance in some scenarios, but has unclear correctness and could worsen performance in other scenarios. If a host-local daemon-managed store was union-mounted on top of a shared NFS store, that would resolve many of the issues, and return the desirable ability to install packages on only one host instead of everywhere.

Concretely, a "union store" would be a new LocalFSStore class, which maintains the store invariants only if the following three conditions hold:

1. It is in contact with a "backend store", which is another LocalFSStore.
2. The union store's store directory is union-mounted on top of the backend store's store directory.
3. The backend store maintains the store invariants.

The implementation would likely share much with that of LocalStore, and it would always be able to fall back on doing exactly what LocalStore does.

But a union store would have an additional capability over LocalStore: Whenever it needs some store path, it could call out to the backend store to see if the store path is already present. It could even forward builds to the backend store based on some criteria, though we'd need to take care that secrets aren't uploaded. It might be simple enough to say that a union store will never call addToStore or similar calls on the backend store, though that might be undesirable for some use cases.

In any case, if a needed store path is present in the backend store, the union store would:

1. Create a new GC root in the backend store for that path, so that the backend store does not delete store paths relied upon by the union store.
2. Record the path as valid and provided by the backend store.

Store paths in the union store would then have dependencies on store paths provided by the backend store.

If the union store later, during garbage collection, wants to delete a path provided by the backend store, then it simply removes the GC root it created in the backend store for the path.

This is a very skeletal sketch, although I've been thinking about this for a while. The key realization that I had recently is that the same new concept would allow all three of the motivating ideas I mentioned above, which makes me more convinced that this would be a powerful and useful feature. Thoughts?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/multi-user-secure-nix-store-user-quotas/7803/2

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-build-a-nixos-vm-with-nix-in-which-nixos-rebuild-is-a-no-op/7937/3

[anka-213]

Would this cause problems with the store not residing in /nix anymore, so much of binary caching would be broken? Or is there some good workaround for that?

[lheckemann]

Mounting an overlay filesystem to /nix/store should do the trick.

[stale]

I marked this as stale due to inactivity. → More info

[queezle42]

Not stale, right now it's not possible to share builders securely, this would really help.

[stale]

I marked this as stale due to inactivity. → More info

[queezle42]

Not stale (for the same reason)

[bew]

I love this and was wondering about something similar recently as well! 🙃
When you say "talk to the backend store", are you implying "the the system nix-daemon that manages the system store" ?
Or something else?

[stale]

I marked this as stale due to inactivity. → More info

[rhendric]

Not stale. I'm very interested in something along these lines.

[pogobanane]

Mounting an overlay filesystem to /nix/store should do the trick.

This is only partially true.

For me the following workaround approximates this proposal: Mount host1 store readonly on host2 and set up an overlayFS to make the host1 store writeable on host2. This makes files available, but the database of host2 still needs updating to work well with flakes/nix-command:

# --dump-db or --export
@host1> nix-store --dump-db > host-db
# make host-db available on host2: --load-db (faster) or --import (slower). This command can take several minutes :(
@host2> nix-store --load-db < host-db

The NixOS options virtualisation.writableStore and virtualisation.pathsInNixDB do basically that. However it is quite slow in my case when sharing the entire (somewhat big) nix store from host1.

[lheckemann]

Right, I got that wrong: I think as long as the host store doesn't change, you can overlay-mount the whole of /nix, including the database. That would avoid a lengthy import step.