Relative paths in stack.yaml "packages" section seem to not work

[thomasjm]

Apologies if I'm missing something obvious here. I have a monorepo with a couple dependency packages being used by a main package, something like this:

./depA
./depB
./mainApp

Inside mainApp/stack.yaml, I use a relative path to reference the other packages:

# mainApp/stack.yaml
resolver: lts-16.5
packages:
- .
- ../depA
- ../depB

However, when I drop the default.nix from the haskell.nix tutorial into mainApp/default.nix and try to build, it doesn't work. I get errors like

stack-to-nix: /nix/store/XXXXX-mainApp/../depA: getDirectoryContents:openDirStream: does not exist (No such file or directory)

This suggests to me that this use-case is not supported.

But maybe I'm going about it the wrong way anyway? I'd greatly appreciate advice on the right way to use haskell.nix in a monorepo like this--I've been over the manual carefully and I can't figure it out. (For example, I already confirmed that I can create ./depA/default.nix and build depA successfully by itself. Can I then just import it and add it to the packages that mainApp knows about? Sort of confused by the variety of options like hsPkgs, pkg-def-extras, etc. mentioned in the manual.)

[michaelpj]

Typically one puts the "project configuration" (i.e. stack.yaml) in the folder containing all the packages. The project corresponds to building a bunch of packages together, i.e. it's the configuration for your monorepo, so typically lives at the top level. So I would suggest moving your stack.yaml to the top level anyway for a more standard layout.

As for why this isn't working, you've probably put src = ./. or similar in your Nix expression, which will take as the source everything in mainApp, and nothing else. You might be able to solve it by passing src = ../.; stackYaml = "mainApp/stack.yaml" instead.

[hamishmack]

If mainApp, depA and debB are all different git repositories then you might need cleanGits.

[thomasjm]

Thanks for the fast responses!

@hamishmack thanks, I'm trying to keep everything in the same tree (no submodules or anything).

@michaelpj I tried passing the stackYaml argument but I got an error about how cleanGit was called with an unexpected argument. To be clear, I'm using the default.nix from the "getting started" section of the tutorial, so the relevant part looks like the following. Should I pass stackYaml to something other than cleanGit?

  src = pkgs.haskell-nix.haskellLib.cleanGit {
    name = "mainApp";
    src = ./.;
  };

I don't think having a single top-level stack.yaml works for more complicated dependency structures. To elaborate a little bit on mine, the structure also includes another "main" package that also depends on the dep packages, i.e.

./depA
./depB
./mainA # depends on depA and depB
./mainB # depends on depA and depB

Currently mainA and mainB have their own stack.yaml files, because they have very different dependencies aside from their common need for depA/depB.

It seems natural to me for a monorepo to contain all the packages side by side like this, with potentially arbitrary dependencies between them. It works fine with normal Stack (aside from terrible bugs like commercialhaskell/stack#3130...). Is this something reasonable to do with haskell.nix?

[michaelpj]

I would still use a single stack.yaml for everything. Typically people still want to build all their packages in a consistent way. I have never seen someone do what you're doing, but of course it's up to you.

stackYaml is an argument to the project or stackProject function.

[thomasjm]

Okay, I'm trying the single stack.yaml in the root folder. I deleted the stack.yaml files from each of the subfolders. Everything builds okay with Stack. But when I try to run something like nix-build -A mainA.components.exes.mainA, I get the error substitute(): ERROR: file 'stack.yaml' does not exist.

(Am I supposed to put default.nix in the root folder as well, or have one per package/subfolder? I tried both and both gave the same error.)

[thomasjm]

(Could the error be because I have a few other folders in the root folder as well which are not Haskell packages?)

[michaelpj]

It's a bit hard to guess what's going on :) can you post your default.nix and what, specifically, you are doing?

[michaelpj]

(One guess: perhaps you haven't checked in the new stack.yaml? cleanGit will, well, clean out files that aren't in Git! For troubleshooting you may find it easier to just not use cleanGit until you've got it working.)

[thomasjm]

Oh derp, that was it. After git checkin things seem to be building smoothly. Thanks for the help!

Do you want to keep this issue open? I'm happy to close but if haskell.nix is supposed to "just work" against any stack.yaml setup, even with weird relative package paths, the original issue might be worth tracking.

[michaelpj]

Hard to say - it does work with relative paths, it doesn't work with paths outside the stated source! Arguably this is a standard Nix gotcha, but maybe it's worth documenting. I might add an entry to the Troubleshooting section.

[hamishmack]

It is possible to make relative paths in a stack.yaml work with something like:

  src = (pkgs.haskell-nix.haskellLib.cleanGit {
      src = ../.;
      name = "root";
    }) + "/mainApp"; 

but + "/mainApp" completely breaks the cleanSourceWith calls used in haskell.nix to separate dependencies of different packages and components.

Instead use the includeSiblings option of cleanSourceWith that lets you tell it not to hide the siblings of the subDir.

  src = haskell-nix.haskellLib.cleanSourceWith {
    src = pkgs.haskell-nix.haskellLib.cleanGit {
      src = ../.;
      name = "root";
    }
    subDir = "mainApp";     # Where to look for the `cabal.project` or `stack.yaml`
    includeSiblings = true; # Tells it not to exclude other directories
  };

[thomasjm]

This is old and I decided to go with the flow and use single stack.yaml files, so I'm going to close.