Expose the flake directory to commands

[zimbatm]

Is your feature request related to a problem? Please describe.

When running nix fmt path:some/flake, the formatter will likely try to format files in the current directory instead of starting from the root of the flake. There is no mechanism to inform the subprocess of the flake's root directory.

Describe the solution you'd like

Introduce a new environment variable FLAKE_ROOT that points to the root of the flake directory (where the flake.nix file lives). For path flakes, the value should point to the original path location, and for non-path flakes, it should point to the copy of the flake located in /nix/store.

Extend the nix CLI to set the FLAKE_ROOT environment variable before invoking programs such as nix fmt, nix run and nix develop.

Describe alternatives you've considered

One alternative is to use chdir before invoking the command, but this requires translating relative path arguments passed to the program.

Another option is to set FLAKE_URL/FLAKE_URI to resolve to path:/path/to/my/repo for local flakes and the original URL otherwise, but this is not directly usable by CLI tools.

Additional context

This proposal is a subset of #6241

Another example use case is for devenv, which depends on --impure to function correctly: https://discourse.nixos.org/t/change-devenv-root-path-for-devenv/26213

Priorities

Add 👍 to issues you find important.

[zimbatm]

This is a small change that could have a good impact on the usability of flakes.
/cc @roberth

[roberth]

FLAKE_ROOT

This is somewhat ambiguous, as in the case of a subflake, or really a flake in a subdirectory, either of two directories can be considered the root.

• the root of the flake source
• the location of the flake's tree

Although the semantics can be debated, I think it'd be helpful to differentiate the terminology regardless.

• FLAKE_DIR: location of the flake.nix file
• FLAKE_SOURCE_ROOT: parent of .git (or similar)

We may choose not to implement the latter until the core flake semantics are stabilized, ie after lazy trees, and after deciding/implementing on the feedback on the lock file and subflakes.

subset of #6241

A nix dev command resolves the inconsistency between nix run . and nix run git:..., where the latter does not have a worktree and therefore no FLAKE_DIR.

The issue will be:

1. Alice writes a flake with a nix run command that works locally.
2. Alice publishes the flake
3. Bob tries nix run git:... but then it doesn't work because the command tries to load interact with FLAKE_DIR. It worked on Alice's machine.

If we implement this in nix run first

• We would have to break nix run to make it internally consistent when we do implement nix dev.
• A distinct nix dev could automatically do the right thing for development-only dependencies: Development only flake inputs or different category of inputs #6124
  • However, I think the right solution is to make the lock file lazier: Reuse input lock files #7730

So while I do think we should avoid this for nix run, we can already implement it for nix fmt and nix develop. These commands should be more or less just special cases of nix dev anyway.

[zimbatm]

Thanks for unrolling all your thoughts, it's interesting.

To me, semantically speaking, the FLAKE_ROOT is where the flake.nix lives, and for something larger maybe I would call it the FLAKE_WORKSPACE. But I agree that what matters is the functionality. For now, I updated the proposal to make that clearer.

It's true that the command might not work if the folder is read-only in the /nix/store, but I don't think it's as big of an issue as you think. Worst case the command doesn't work and developers will learn about this limitation. It will cause a few surprises and be documented.

Is the nix dev proposal something that comes from the nix core team, or something that you are working on getting consensus over? Ideally, I would like to keep things as orthogonal as possible in order to get this simple change in. There is a whole issue around flake inputs categorization that is worth having for sure.

[roberth]

As a general rule, things haven't been blessed by the team unless it has an idea approved label or a comment with meeting notes. (As of early 2023; process may change in the future)

nix dev has not been discussed by the team yet, and I wasn't part of the team when I wrote that proposal. I would like for it to be discussed noting that naming is hard, and we should strive for a sensible naming scheme, which ties to the broader discussion, including

• Rename 'nix shell' #4715

... so for a quick win, we might want to avoid nix run until the command naming scheme is settled. Or just not care about the "hermeticity" of nix run, I guess.

[zimbatm]

I would prefer to have it applied across the board, so developers don't have to remember which command has the env var. Also because devshells can both be used locally and remotely as well (eg: nix develop github:user/devshells#ruby). Developers that want to ensure the command only runs on mutable folders can check if [[ -w $FLAKE_DIR/flake.nix ]] and abort accordingly.

I don't think introducing this would block the introduction of nix dev as it's pretty complimentary.

In order to move forward I propose that the nix team votes:

1. Are you happy to entertain this idea in general?
2. If yes, do you want to make a distinction between the various commands?
3. Do you prefer FLAKE_DIR or FLAKE_ROOT as the name?

That seem to be the main choices in the decision tree.

[edolstra]

Nix should probably just set the current directory to the flake directory before running the formatter. That removes the need for an environment variable and makes execution more predictable.

[zimbatm]

Consider the following scenario:

$ cd /path/to/myrepo/some_subfolder
$ nix fmt ./fileA.txt

Assuming that /path/to/myrepo is where the flake.nix lives. As a user, I expect the formatter to format /path/to/myrepo/some_subfolder/fileA.txt.

If nix calls chdir("/path/to/myrepo") before calling the program, then Nix is now also in the business of translating function arguments IFF they are paths.

[zimbatm]

Another example is https://github.com/numtide/treefmt-nix, where the config requires a projectRootFile to determine the project's root. This could be simplified if nix provided that env var.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-06-02-nix-team-meeting-minutes-59/28666/1

[domenkozar]

I'd really wish for a cwd flake input, which would be considered "an effect" but still keep all design properties of flakes intact.

[roberth]

@domenkozar I would think that knowing the path at evaluation time adds very little over knowing it at runtime. Is that a wrong assumption? What is something you could do with an eval-time path that you wouldn't be able to do at runtime instead?

[domenkozar]

How do you generate a config file using Nix that requires an absolute path to let's say a stateful directory?

[roberth]

You might be able to use a placeholder and then substitute that at the last moment, in a file that's not in the store, or in the store while rewriting to content addressed paths (but not part of the original build).

That's a lot of extra complexity and fragility though, that you wouldn't have if you could take advantage of the Nix language to construct the right thing in the first place.

I think the main objections about impurities relate to

• reproducibility of packages
• ability to cache evaluations

Reproducibility of packages could still be guaranteed by evaluating nix build, nix shell and nix run commands with this new input disabled (ie make the input return abort "..."), and only allow it for nix develop, nix dev and nix print-dev-env.

By virtue of being an input, evaluation caching won't be a problem, because inputs are already part of the cache key.

[domenkozar]

You might be able to use a placeholder and then substitute that at the last moment, in a file that's not in the store, or in the store while rewriting to content addressed paths (but not part of the original build).

It gets very tricky if you depend on that config from another Nix build. We'd almost have to build this in as a placeholder feature into Nix.

I think the main objections about impurities relate to

reproducibility of packages
ability to cache evaluations

That's why it would be an input, as Nix controls what inputs are "safe" and cwd is one of those that will sacrifice a bit of hermeticity (between machines), but won't break otherwise.

[roberth]

I quite like the impure directory location input idea. (not sure if we should make it cryptic like cwd though)

It would make for a good addition that would be able to implement the environment variable, but there's also merit to establishing a standard environment variable for this, so perhaps we should do both.

[roberth]

You might be able to use a placeholder and then substitute that at the last moment, in a file that's not in the store, or in the store while rewriting to content addressed paths (but not part of the original build).

It gets very tricky if you depend on that config from another Nix build. We'd almost have to build this in as a placeholder feature into Nix.

nix store make-content-addressed would handle deep replacements in a whole closure just fine, but it doesn't do arbitrary extra string substitutions yet. That'd be a small addition to support that, but you don't get proper scoping, so it's still not as good as a flake input.

[poelzi]

My use case which requires something like FLAKE_ROOT:

I use logseq with git-annex for note keeping. I would like to have a flake file with my logseq environment and want to run logseq with the argument of the checked out flake. Referring to the read-only store is absolute useless for this task. I want to open the flake source folder with a program defined in the flake.

This is a quite common case for any editorial work. If you want to package the tools with your project, you need to edit the project too.

[iFreilicht]

For those looking for a workaround: If you need this because you want to run some commands inside a shellHook and be sure they run at the flake root, you can use git to find the root, and just use that. Make sure to check that you're not in a different repository, though, as people can just run nix develop ../your-flake.

 
  devShell =
    pkgs.mkShell
      {
        buildInputs = with pkgs; [
          python3
        ];
        shellHook = ''
          FLAKE_ROOT="$(git rev-parse --show-toplevel)"
          (
            cd "$FLAKE_ROOT"
            if [ -d git-hooks ]; then
              rm -f .git/hooks/*
              cp git-hooks/* .git/hooks/
            else
              echo "Warning: Not running in expected repository, but in '$FLAKE_ROOT'!"
            fi
          )
        '';
      };