[RFC 0049] Flakes

rfcs/0049-flakes.md

@@ -0,0 +1,384 @@
+---
+feature: flakes
+start-date: 2019-07-09
+author: Eelco Dolstra
+co-authors: TBD
+shepherd-team: Domen Kožar, Alyssa Ross, Shea Levy, John Ericson
+shepherd-leader: Domen Kožar
+related-issues: (will contain links to implementation PRs)
+---
+
+# Summary
+[summary]: #summary
+
+This RFC proposes a mechanism to package Nix expressions into
+composable entities called "flakes". Flakes allow hermetic,
+reproducible evaluation of multi-repository Nix projects; impose a
+discoverable, standard structure on Nix projects; and replace previous
+mechanisms such as Nix channels and the Nix search path.
+
+# Motivation
+[motivation]: #motivation
+
+Flakes are motivated by a number of serious shortcomings in Nix:
+
+* While Nix pioneered reproducible builds, sadly, Nix expressions are
+  not nearly as reproducible as Nix builds. Nix expressions can access
+  arbitrary files (such as `~/.config/nixpkgs/config.nix`),
+  environment variables, and Git repositories. This means for instance
+  that it is hard to ensure reproducible evaluation of NixOS or NixOps
+  configurations.
+
+* Nix projects lack discoverability and a standard structure. For
+  example, it's just convention that a repository has a `release.nix`
+  for Hydra jobs and a `default.nix` for packages.
+
+* There is no standard way to compose Nix projects. Typical ways are
+  to rely on the Nix search path (e.g. `import <nixpkgs>`) or to use
+  `fetchGit` or `fetchTarball`. The former has poor reproducibility,
+  while the latter is bad UX because of the need to manually update
+  Git hashes to update dependencies.
+
+* `nix-channel` needs a replacement: channels are hard to create,
+  users cannot easily pin specific versions of channels, channels
+  interact in *ad hoc* ways with the Nix search path, and so on.
+
+The flakes mechanism seeks to address all these problems. This RFC,
+however, only describes the format and semantics of flakes; it doesn't
+describe changes to the `nix` command to support flakes.
+
+# Detailed design
+[design]: #detailed-design
+
+## Flakes
+
+A flake is a Git repository that contains a file named `flake.nix` in
+the root directory. (In the future, there might be other types of
+flakes, such as Mercurial repositories or tarballs.) `flake.nix`
+specifies some metadata about the flake such as dependencies (called
+*inputs*), as well as its *outputs* (the Nix values such as packages
+or NixOS modules provided by the flake).
+
+As an example, below is the `flake.nix` of
+[`dwarffs`](https://github.com/edolstra/dwarffs) (a FUSE filesystem
+for automatically fetching DWARF debug symbols by ELF build ID). It
+depends on the Nixpkgs flake and provides a package (i.e. an
+installable derivation) and a NixOS module.
+
+```
+{
+  edition = 201911;
+
+  description = "A filesystem that fetches DWARF debug info from the Internet on demand";
+
+  outputs = { self, nixpkgs }: rec {
+    packages.dwarffs =
+      with nixpkgs.packages;
+      with nixpkgs.builders;
+      with nixpkgs.lib;
+
+      stdenv.mkDerivation {
+        name = "dwarffs-0.1.${substring 0 8 self.lastModified}";
+
+        buildInputs = [ fuse nix nlohmann_json boost ];
+
+        NIX_CFLAGS_COMPILE = "-I ${nix.dev}/include/nix -include ${nix.dev}/include/nix/config.h -D_FILE_OFFSET_BITS=64";
+
+        src = self;
+
+        installPhase =
+          ''
+            mkdir -p $out/bin $out/lib/systemd/system
+
+            cp dwarffs $out/bin/
+            ln -s dwarffs $out/bin/mount.fuse.dwarffs
+
+            cp ${./run-dwarffs.mount} $out/lib/systemd/system/run-dwarffs.mount
+            cp ${./run-dwarffs.automount} $out/lib/systemd/system/run-dwarffs.automount
+          '';
+      };
+
+    nixosModules.dwarffs = ...;
+
+    defaultPackage = packages.dwarffs;
+
+    checks.build = packages.dwarffs;
+  };
+}
+```
+
+A flake has the following attributes:
+
+* `edition`: A number that specifies the version of the flake
+  syntax/semantics to be used. This allows the interpretation of
+  flakes to change in the future. It also enables some evolution of
+  the Nix language; for example, the Nix files in the flake could be
+  parsed using a syntax determined by the edition. The only currently
+  allowed value is `201911`. Nix rejects flakes with an unsupported
+  edition.
+
+* `description`: A short description of the flake.
+
+* `inputs`: An attrset specifying the dependencies of the flake
+  (described below).
+
+* `outputs`: A function that, given an attribute set containing the
+  outputs of each of the input flakes keyed by their identifier,
+  yields the Nix values provided by this flake. Thus, in the example
+  above, `inputs.nixpkgs` contains the result of the call to the
+  `outputs` function of the `nixpkgs` flake, and
+  `inputs.nixpkgs.packages.fuse` refers to the `packages.fuse` output
+  attribute of `nixpkgs`.
+
+  In addition to the outputs of each input, each input in `inputs`
+  also contains some metadata about the inputs. These are:
+
+  * `outPath`: The path in the Nix store of the flake's source
+    tree. This means that you could import Nixpkgs in a more
+    legacy-ish way by writing
+
+        with import inputs.nixpkgs { system = "x86_64-linux"; };
+
+    since `nixpkgs` still contains a `/default.nix`. In this case we
+    bypass its outputs entirely and only use the flake mechanism to
+    get its source tree.
+
+  * `rev`: The commit hash of the flake's Git repository.
+
+  * `revCount`: The number of ancestors of the revision `rev`. This is
+    not available for `github:...` repositories (see below), since
+    they're fetched as tarballs rather than as Git repositories.
+
+  * `lastModified`: The commit time of the revision `rev`, in the
+    format `%Y%m%d%H%M%S` (e.g. `20181231100934`). Unlike `revCount`,
+    this is available for both Git and GitHub repositories, so it's
+    useful for generating (hopefully) monotonically increasing version
+    strings.
+
+  The value returned by the `outputs` function must be an attribute
+  set. The attributes can have arbitrary values; however, some tools
+  may require specific attributes to have a specific value (e.g. the
+  `nix` command may expect the value of `packages` to be an attribute
+  set of derivations).
+
+## Flake inputs
+
+The attribute `inputs` specifies the dependencies of a flake. These
+specify the location of the dependency, or a symbolic flake identifier
+that is looked up in a registry or in a command-line flag. For
+example, the following specifies a dependency on the Nixpkgs and Hydra
+repositories:
+
+    # A GitHub repository.
+    inputs.import-cargo = {
+      type = "github";
+      owner = "edolstra";
+      repo = "import-cargo";
+    };
+
+    # An indirection through the flake registry.
+    inputs.nixpkgs.id = "nixpkgs";
+
+Each input is fetched, evaluated and passed to the `outputs` function
+as a set of attributes with the same name as the corresponding
+input. The special input named `self` refers to the outputs and source
+tree of *this* flake. Thus, a typical `outputs` function looks like
+this:
+
+    outputs = { self, nixpkgs, import-cargo }: {
+      ... outputs ...
+    };
+
+It is also possible to omit inputs entirely and *only* list them as
+expected function arguments in `outputs`. Thus,
+
+    outputs = { self, nixpkgs }: ...;
+
+without an `inputs.nixpkgs` attribute will simply look up `nixpkgs` in
+the flake registry.
+
+Repositories that don't contain a `flake.nix` can also be used as
+inputs, by setting the input's `flake` attribute to `false`:
+
+    inputs.grcov = {
+      type = "github";
+      owner = "mozilla";
+      repo = "grcov";
+      flake = false;
+    };
+
+    outputs = { self, nixpkgs, grcov }: {
+      packages.grcov = stdenv.mkDerivation {
+        src = grcov;
+        ...
+      };
+    };
+
+The following input types are specified at present:
+
+* `git`: A Git repository or dirty local working tree.
+
+* `github`: A more efficient scheme to fetch repositories from GitHub
+  as tarballs. These have slightly different semantics from `git`
+  (in particular, the `revCount` attribute is not available).
+
+* `tarball`: A `.tar.{gz,xz,bz2}` file.
+
+* `path`: A directory in the file system. This generally should be
+  avoided in favor of `git` inputs, since `path` inputs have no
+  concept of revisions (only a content hash) or tracked files
+  (anything in the source directory is copied).
+
+* `hg`: A Mercurial repository.
+
+## Lock files
+
+Inputs specified in `flake.nix` are typically "unlocked" in that they
+don't specify an exact revision. To ensure reproducibility, Nix will
+automatically generate and use a *lock file* called `flake.lock` in
+the flake's directory. The lock file contains a tree of mappings from
+the inputs specified in `flake.nix` to inputs specifications that
+contain revisions.
+
+For example, if `flake.nix` has the inputs in the example above, then
+the resulting lock file might be:
+```
+{
+    "version": 4,
+    "inputs": {
+        "import-cargo": {
+            "inputs": {},
+            "narHash": "sha256-mxwKMDFOrhjrBQhIWwwm8mmEugyx/oVlvBH1CKxchlw=",
+            "original": {
+              "type": "github",
+              "owner": "edolstra",
+              "repo": import-cargo"
+            },
+            "resolved": {
+              "type": "github",
+              "owner": "edolstra",
+              "repo": "import-cargo",
+              "rev": "c33e13881386931038d46a7aca4c9561144d582e"
+            }
+        },
+        "nixpkgs": {
+            "inputs": {},
+            "narHash": "sha256-p7UqhvhwS5MZfqUbLbFm+nfG/SMJrgpNXxWpRMFif8c=",
+            "original": {
+              "type": "github",
+              "owner": "NixOS",
+              "repo": nixpkgs"
+            },
+            "resolved": {
+              "type": "github",
+              "owner": "NixOS",
+              "repo": "nixpkgs",
+              "rev": "4a7047c6e93e8480eb4ca7fd1fd5a2aa457d9082"
+            }
+        }
+    }
+}
+```
+
+Thus, when we build this flake, the input `nixpkgs` is mapped to
+revision `c33e13881386931038d46a7aca4c9561144d582e` of the
+`edolstra/import-cargo` repository on GitHub. Nix will also check that
+the content hash of the input is equal to the one recorded in the lock
+file. This check is superfluous for Git repositories (since the commit
+hash serves a similar purpose), but for GitHub archives, we cannot
+directly check that the contents match the commit hash.
+
+Note that lock files are only used at top-level: the `flake.lock`
+files in dependencies (if they exist) are ignored. The lock file
+transitively locks direct as well as indirect dependencies.
+
+## Reproducible evaluation
+
+Lock files are not sufficient by themselves to ensure reproducible
+evaluation. We also need to disallow certain impurities that the Nix
+language previously allowed. In particular, the following are
+disallowed in a flake:
+
+* Access to files outside of the top-level flake or its inputs, as
+  well as paths fetched using `fetchTarball`, `fetchGit` and so on
+  without a commit hash or content hash. In particular this means that
+  Nixpkgs will not be able to use `~/.config/nixpkgs` anymore.
+
+* Access to the environment. This means that `builtins.getEnv "<var>"`
+  always returns an empty string.
+
+* Access to the system type (`builtins.currentSystem`).
+
+* Access to the current time (`builtins.currentTime`).
+
+* Use of the Nix search path (`<...>`); composition must be done
+  through flake inputs or `fetchX` builtins.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+Pure evaluation breaks certain workflows. In particular, it breaks the
+use of the Nixpkgs configuration file. Similarly, there are people who
+rely on `$NIX_PATH` to pass configuration data to NixOps
+configurations.
+
+# Alternatives
+[alternatives]: #alternatives
+
+For composition of multi-repository projects, the main alternative is
+to continue on with explicit `fetchGit` / `fetchTarball` calls to pull
+in other repositories. However, since there is no explicit listing of
+dependencies, this does not provide automatic updating (i.e. there is
+no equivalent of `nix flake update`).
+
+Instead of a `flake.nix`, flakes could store their metadata in a
+simpler format such as JSON or TOML. This avoids the Turing tarpit
+where getting flake metadata requires the execution of an arbitrarily
+complex, possibly non-terminating program.
+
+Flakes could be implemented as an external tool on top of Nix. Indeed,
+there is nothing that flakes allow you to do that couldn't previously
+be done using `fetchGit`, the `--pure-eval` flag and some shell
+scripting. However, implementing flake-like functionality in an
+external tool would defeat the goals of this RFC. First, it probably
+wouldn't lead to a standard way to structure and compose Nix projects,
+since we might well end up with numerous competing
+"standards". Second, it would degrade rather than improve the Nix UX,
+since users would now have to deal with Nix *and* the flake-like tool
+on top of it.
+
+# Unresolved questions
+[unresolved]: #unresolved-questions
+
+* Should flakes have arguments (like "system type")? This must be done
+  in a way that maintains hermetic evaluation and evaluation caching.
+
+* Currently, if flake dependencies (repositories or branches) get
+  deleted upstream, rebuilding the flake may fail. (This is similar to
+  `fetchurl` referencing a stale URL.) We need a command to gather all
+  flake dependencies and copy them somewhere else (possibly vendor
+  them into the repository of the calling flake).
+
+* Maybe flake metadata should be stored in a `flake.json` or
+  `flake.toml` file. This would prevent ambiguities when the Nix
+  language changes in a future edition.
+
+# Future work
+[future]: #future-work
+
+* The "edition" feature enables future Nix changes, including language
+  changes. For example, changing the parsing of multiline strings
+  (https://github.com/NixOS/nix/pull/2490) could be conditional on the
+  flake's edition.
+
+* Currently flake outputs are untyped; we only have some conventions
+  about what they should be (e.g. `packages` should be an attribute
+  set of derivations). For discoverability, it would be nice if
+  outputs were typed. Maybe this could be done via the Nix
+  configurations concept
+  (https://gist.github.com/edolstra/29ce9d8ea399b703a7023073b0dbc00d).
+
+# Acknowledgments
+
+Funding for the development of the flakes prototype was provided by
+[Target Corporation](https://www.target.com/).