Packages are resolved w.r.t to project-file path when absolute, but not when relative

[pranaysashank]

Describe the bug
When a project-file is passed as an absolute path that is not in the current directory. cabal seems to be resolving the packages location relative to where the file is located. This doesn't happen if the file path is relative.
Is this a bug or correct behaviour?

To Reproduce
Steps to reproduce the behavior:

[nix-shell:~/src/tmp]$ mkdir foo

[nix-shell:~/src/tmp]$ cd foo/

[nix-shell:~/src/tmp/foo]$ cabal init
Guessing dependencies...

Generating LICENSE...
Warning: unknown license type, you must put a copy in LICENSE yourself.
Generating CHANGELOG.md...
Generating app/Main.hs...
Generating foo.cabal...

Warning: no synopsis given. You should edit the .cabal file and add one.
You may want to edit the .cabal file and add a Description field.

[nix-shell:~/src/tmp/foo]$ cd ..

[nix-shell:~/src/tmp]$ echo "packages: foo" > ../cabal.project

[nix-shell:~/src/tmp]$ cabal v2-clean; cabal v2-build all --project-file ../cabal.project
Resolving dependencies...
Build profile: -w ghc-8.10.4 -O1
In order, the following will be built (use -v for more details):
 - foo-0.1.0.0 (exe:foo) (first run)
Configuring executable 'foo' for foo-0.1.0.0..
Preprocessing executable 'foo' for foo-0.1.0.0..
Building executable 'foo' for foo-0.1.0.0..
[1 of 1] Compiling Main             ( app/Main.hs, /home/pranaysashank/src/tmp/dist-newstyle/build/x86_64-linux/ghc-8.10.4/foo-0.1.0.0/x/foo/build/foo/foo-tmp/Main.o )
Linking /home/pranaysashank/src/tmp/dist-newstyle/build/x86_64-linux/ghc-8.10.4/foo-0.1.0.0/x/foo/build/foo/foo ...

[nix-shell:~/src/tmp]$ cabal v2-clean; cabal v2-build all --project-file /home/pranaysashank/src/cabal.project
When using configuration(s) from /home/pranaysashank/src/cabal.project, the following errors occurred:
The package location 'foo' does not exist.

[nix-shell:~/src/tmp]$

Expected behavior
Expect cabal to resolve the packages field relative to where I run cabal file or have a consistent behaviour with both relative and absolute file paths.

System information

• Operating system: nixos - 21.05
• cabal, ghc versions: 3.4, 8.10.4

[Mikolaj]

I haven't looked in depth, but would this change the situation: #7581 or is it unrelated?

[pranaysashank]

I looked through the pull request discussion and it looks like it's unrelated to this.

[Mikolaj]

OK, I'm puzzled why it doesn't work in the second case. I did a quick code dive, found BadLocNonexistantFile, but then got stuck figuring out how distProjectRootDirectory is determined. Definitely worth answering, and if there is not an obvious answer I'm missing, investigating in codebase.

[fgaz]

Where relative paths are rooted is not clarified in the docs as far as i can see, and that should be fixed.

I always assumed that packages should be relative to the project root directory, not to cwd. If it wasn't, the project would change depending on cwd, which doesn't sound like a good feature. If my assumption is correct:

• The correct behaviour would be to fail in both cases (./tmp/foo would be the correct path)
• The success in the first case is a bug in how relative paths are handled
  • This wouldn't be the first bug of this kind, and I agree with @phadej here about using better-typed paths

[fgaz]

Okay, I have an idea why this is happening, and my experiments confirm this. From https://cabal.readthedocs.io/en/3.6/cabal-project.html#cmdoption-0 i see that --project-file=PATH roots PATH to the first valid place while walking up from CWD. It may be that that first valid place is treated as project root, so packages is rooted at that place. In the OP example, this means

• when --project-file ../cabal.project is used, the project is rooted in . since ./../cabal.project exists, so packages: foo refers to ./foo
• when --project-file /home/pranaysashank/src/cabal.project is used, the project is rooted in / since //home/pranaysashank/src/cabal.project exists, so packages: foo refers to /foo, which does not exist
• as a further example, if --project-file is not used, it would default to cabal.project, which means the project is rooted in .. since ../cabal.project exists, so packages: foo refers to ../foo and the build would fail

Note that packages is still relative to the project root. The point is that --project-file changes the root in a really weird way.

This does not sound user friendly at all but at least it is consistent and predictable.
We should decide whether this is acceptable and

• if yes, document it better
• if not, document that this is a bug, fix it, and document the new behaviour

edit: here's a place where the documentation conflicts with this behaviour:

https://cabal.readthedocs.io/en/3.6/cabal-project.html#cmdoption-builddir

[...]this directory is resolved relative to the root of the project (i.e., where the cabal.project file lives.)

[pranaysashank]

as a further example, if --project-file is not used, it would default to cabal.project, which means the project is rooted in .. since ../cabal.project exists, so packages: foo refers to ../foo and the build would fail

@fgaz Yes, this is what I get

[pranaysashank@nixos:~/src/tmp]$ ls
foo

$ [pranaysashank@nixos:~/src/tmp]$ cabal build all
When using configuration(s) from /home/pranaysashank/src/cabal.project, the following errors occurred:
The package location 'foo' does not exist.

[pranaysashank@nixos:~/src/tmp]$

[pranaysashank]

I can understand that this behaviour makes sense in case we are in a subdirectory of a project and do a cabal build it resolves all the packages correctly at the top level dir where cabal.project is present.

I am not sure if the same behaviour makes sense when we explicitly pass a project-file though.

[Mikolaj]

@pranaysashank: well put. @fgaz and I think it's worth a simplification. Let me mark it so. [Edit: but it's yet clear how exactly to simplify.]

[pranaysashank]

when --project-file /home/pranaysashank/src/cabal.project is used, the project is rooted in / since //home/pranaysashank/src/cabal.project exists, so packages: foo refers to /foo, which does not exist

@fgaz I am not sure it's so simple since the following works

$ cabal build all --project-file /home/pranaysashank/src/tmp/cabal.project

Based on your comment it should have looked for foo in / and failed

[fgaz]

@Mikolaj

but it's yet clear how exactly to simplify

simple: make the project root coincide to the location of the project file

@pranaysashank

cabal build all --project-file /home/pranaysashank/src/tmp/cabal.project

isn't cabal.project in ~/src, and not in ~/src/tmp? It could also be that absolute path are always resolved absolutely and not rooted anywhere, but that path is still worng

[pranaysashank]

@fgaz No the project is in tmp, and foo is the cabal package inside tmp. So,

$ pwd
/home/pranaysashank/src/tmp
$ ls
foo cabal.project

Note: I moved cabal.project here to tmp to test this. In the original post, cabal.project was in src directory (so outside the project)

[fgaz]

I tried a few more combinations, and I think I got it all now:

• relative --project-file behaves as described above
• absolute --project-file roots the project at the location of the project file <- this is what we want in all cases

the fact that they are different suggests that this (the behaviour on relative paths based on above discussion) is a bug

[pranaysashank]

I think it'd be nice to have the relative behavior in all cases. So if I have some constraints in a global project file, any package can immediately use them (of course dist-newstyle would be in the project file directory then)

[Mikolaj]

@pranaysashank: do you have a convincing speculative use case or a real life use case? We not only need a consistent implementation and documentation of this feature (which I'd love if you could provide; even just to see if the codebase gets more complicated or simpler), but we'd need to maintain it, also after you stop using this particular feature and so lose interest in supporting it.

This consideration is based on our hunch that we can slightly simplify the codebase making the absolute behaviour the only one and that the documentation would be simpler in this case.

[pranaysashank]

I ran into this issue accidentally and decided to report it in case it's a bug. I don't even depend on it working, my above comment was speculative.

[cydparser]

I was looking at implementing #6051 and noticed the difference between absolute and relative --project-file paths while reading the code. For my use case, the relative behavior is ideal, since I would like to store the Nix project files in a nix directory (rather than pollute the root directory). I think there are two separate things here: 1) specifying the project file(s) 2) specifying the project root (currently only inferred). I suspect that the main use case for using an absolute path with --project-file is to specify the project root. Perhaps there should be a separate command for both.

[cydparser]

I have opened a PR (#8454) that addresses this issue by:

1. Deprecating the use of --project-file with an absolute path
2. Adding a --project-dir flag
3. Always resolving project-file relative to project dir

Please feel free to discuss/bikeshed in the PR.

[Mikolaj]

Could I ask for help reviewing the PR fixing this issue (#8454)? Even a cursory look and a comment whether this might fix your use case (bonus points for checking out and confirming) would be very much appreciated.

[Mikolaj]

A new proposal dropped at #8454. Could you kindly voice your opinion?

[pranaysashank]

I'd just like to point out that the bug this issue discovered happens when we pass a relative path which I don't think is fixed by the merged PR

[Mikolaj]

Thank you for the clarification. Let me reopen. If cabal 3.10, when it's released, fixes this, please somebody note it down here.

[gbaz]

The merged pr claims that it has a behavior of Always resolving project-file relative to project dir even though that isn't in the title. So I do think that it should resolve this.

[ulysses4ever]

I'd just like to point out that the bug this issue discovered happens when we pass a relative path which I don't think is fixed by the merged PR (#8454)

@cydparser can you sched light on this?

[cydparser]

The expected behavior in this issue:

Expect cabal to resolve the packages field relative to where I run cabal file or have a consistent behavior with both relative and absolute file paths.

PR #8454 deprecates the use of --project-file with an absolute path (users will see a warning message directing them to use --project-dir). A later release should forbid absolute paths in --project-file without an explicit --project-dir. This would satisfy "or have a consistent behavior."

[pranaysashank]

There was no issue with using absolute paths in project-file. There is a bug in the behaviour of relative paths, here is the current behaviour when you pass a relative file path:

• Start from the current directory
• Check if project file exists at the given relative path
• If not, traverse to the parent directory, set it as the current directory and repeat step 2
• If the file is found, set the current directory (the directory from where the relative path was resolved) as the project root.

From one of the earlier comments in this discussion:

Note that packages is still relative to the project root. The point is that --project-file changes the root in a really weird way.

Please go through the example in the issue as that should clarify the what the actual bug is.