Improve support for relative path inputs

[edolstra]

Motivation

We want it to be possible to refer to flake inputs that reside in the same repo via relative paths (e.g. inputs.foo.url = "path:./subdir"). Previously this didn't work very well because those inputs would be separately copied to the store, which is inefficient and makes references to parent directories tricky or impossible. Furthermore, they had their own NAR hash in the lock file, which is superfluous since the parent is already locked. In fact, if relative path inputs worked at all, it was by accident.

Now relative path inputs are accessed via the accessor of the calling flake. This avoids the unnecessary copy and makes it possible for relative path inputs to depend on flakes in a parent directory (so long as they're in the same tree).

Lock file nodes for relative path inputs now have a new parent field:

{
  "locked": {
    "path": "./subdir",
    "type": "path"
  },
  "original": {
    "path": "./subdir",
    "type": "path"
  },
  "parent": [
    "foo",
    "bar"
  ]
}

which denotes that ./subdir is to be interpreted relative to the directory of the bar input of the foo input of the root flake.

Extracted from the lazy-trees branch.

• Depends on Use SourcePath for reading flake.{nix,lock} #10088.
• Closes Allow flakes to refer to other flakes by relative path #3978

Context

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[dpulls]

🎉 All dependencies have been resolved !

[thufschmitt]

Discussed during the Nix maintainers meeting on 2024-03-04.

• Idea approved
• Document that directly accessing .. in subflakes is likely to break in the future
• Use path literals as the url for subflakes

• Great overall
• Right now, the PR allows import ../. in a subflake. Do we want that?
  • Directly accessing .. is convenient, but breaks the encapsulation property of flakes
    • Relates to nix flakes: flakes in subdirectories can access files in parent directories (breaking the implied seal of flake.nix) #4414
  • We can however access other parts of the tree with relative inputs (that can go up until the root)
    • inputs.something_else = ../foo
  • From an implementation perspective, hard to forbid without lazy trees
  • Decision:
    • Don't formally forbid direct access to .. since it's hard without lazy trees
    • Once lazy trees lands, make that conditional on a flag
    • Potentially make the default restrictive. Would be a breaking change, but
      • can be mitigated by only making that change effective after a flake update
      • small breakage as getting the old behavior back would only be a one-line change
• Syntax for relative flakes?
  • Currently reuses url = "path:./foo" that already kinda works on master
  • A more natural syntax would be url = ./foo
    • This is actually what most people seem to try first when they want subflakes

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-03-04-nix-team-meeting-minute-130/40830/1

[edolstra]

Added a release note. IMHO this is ready to be merged.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-01-15-nix-team-meeting-minutes-208/58993/1

[alurm]

Maybe I misunderstood something or made a mistake somewhere, but path:.. doesn't work for me on 043df13. I expected it to resolve to the flake in the directory above the current flake's. I get "error: path /nix/store/flake.nix doesn't exist".

I also tried path:., but it issues a warning about circular imports (which makes sense, I think).

[roberth]

path:..

Is your flake in a git repo?

If not, that would explain the symptoms.

IMO a flake should be self-contained unless a flake attribute specifies otherwise. Flakes that aren't in a repo are assumed to be self-contained, or in a more practical sense copied at the dirOf flake.nix and not any higher directories.
By providing a flake.nix attribute that makes this parent relationship explicit, we could infer which directory is the root of the flake source and work from there.

Alternatively, we'll also want attributes that specify how a flake should be fetch, e.g. enable submodules or lfs. Those could also be assigned the responsibility to determine the flake source root, such that path:.. can work without Git to provide that implicitly.

[alurm]

@roberth, the main flake and the "subflake" both were in the same Git repo.

I had the following layout:

/.git
/flake.nix
/raspberry-pi-5/flake.nix
/thinkpad/flake.nix
...
/shared/...

I have specified nix.package to refer to the commit which closes this issue and I have rebuilt.

I wanted to access /shared from the Raspberry Pi's flake. As it stands, I did not understand how to do this, so now I'm rewriting stuff to have one big /flake.nix.

Well, I figured how to achieve this also using git+file, but it required to constantly nix flake update, which I found inconvenient. Also there's a later commit in Nix which says that relative git+file may be removed in the future.

To be clear, maybe I just did something wrong.

[roberth]

We have a few test cases here that suggest it should work, but maybe yours is more complicated than what we have?
From what you describe I would think something representative is in there, except perhaps a dependency chain like the following.
A chain like sub1 -> sub2 is tested, but maybe root -> sub1 -> sub2 is a problem? Maybe root -> sub1 copies sub1, cutting it off.
Did you try cd-ing into rasperry-pi-5 and build it from there, or did you forward some attr to the root flake?

[alurm]

I'm pretty sure I was in the subdirectory, the top level flake didn't have any nixosConfigurations.

I'll try to see what's going on tomorrow, thanks for the link to the tests.

[alurm]

@roberth, alright, I was doing something wrong, it works. Sorry for bothering.

However I think I found out why I was confused, I think it's an issue with nixos-rebuild repl. I made a small example. Please look at it. If it's a bug indeed, I can create an issue in the tracker (though I'm not sure which project nixos-rebuild belongs to).

/test: Test

/subdir/flake.nix:

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

  outputs = inputs: {
    nixosConfigurations.test = inputs.nixpkgs.lib.nixosSystem {
      modules = [
        {
          nixpkgs.system = "aarch64-linux";
          fileSystems."/".device = "/dev/null";
          boot.loader.systemd-boot.enable = true;
          environment.etc.my-test.text = builtins.readFile ../test;
        }
      ];
    };
  };
}

All of the following is executed in subdir.

Test 1.

$ nix repl
nix-repl> :lf .
nix-repl> nixosConfigurations.test.config.environment.etc.my-test.text
"Test\n"

This works as I expect it to work.

Test 2.

$ nixos-rebuild .#test build
$ cat result/etc/my-test
Test

This works as I expect it to work.

Test 3.

$ nixos-rebuild --flake .#test repl
nix-repl> nix-repl> config.environment.etc.my-test.text
error:
[Part of the log omitted...]
       error: opening file '/nix/store/test': No such file or directory

I expected Test and not an error.

Can you confirm if this is a bug, at least conceptually?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/relative-path-support-for-nix-flakes/18795/7

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-2-26-released/59211/1