Nix flake fix dev shells #2655

guibou · 2022-01-29T15:00:36Z

This is a work in progress.

This clean and fixs the nix develop shells for GHC 9.2 and GHC 9.0.

See individual commits for details, but in summary:

nix develop .#haskell-language-server-ghc921-dev now works.
When entering the shell, it creates an alias for cabal which points to the correct cabal-xxx.project

I'm planning on testing on GHC 9.0 soon too (but right now it is building the world of dependencies).

Future commits to this MR will (hopefully) includes:

Support for GHC 9.0
I'd like to restore the CI steps related to nix/flake, as an optionnal step which can be triggered on demand by developers. It will also populate the cachix cache.
Additional cleanups. I'm trying to remove all the explicit "hacks" if version == ...

michaelpj · 2022-01-29T20:46:49Z

When entering the shell, it creates an alias for cabal which points to the correct cabal-xxx.project

I'm a little wary of this. It's nice if you know it's there, but it's hidden magic and adds a source of confusion ("I can't cabal build with 9.0", "I can".... "oh, one of us was using the magic cabal wrapper").

guibou · 2022-01-29T21:08:11Z

When entering the shell, it creates an alias for cabal which points to the correct cabal-xxx.project

I'm a little wary of this. It's nice if you know it's there, but it's hidden magic and adds a source of confusion ("I can't cabal build with 9.0", "I can".... "oh, one of us was using the magic cabal wrapper").

I agree.

What about if the alias is called cabal_ghc92, so you must explicitly use cabal_ghc92 build, and a message is printed when entering the shell, such as:

Cabal won't be able to build your project without using the project file `cabal-xxx.project`. An alias `cabal_ghcxxx` was automatically created for you and uses this cabal project. Use it like:

cabal_ghcxxx build

michaelpj · 2022-01-29T21:13:54Z

That sounds fine to me, although at that point I might as well type --project-file=... myself :) The warning is nice though!

guibou · 2022-01-31T21:33:47Z

A few changes:

As suggested by @michaelpj, the alias when entering the nix-shell is not cabal, but cabal_project. A message warns the user about the need to use a special cabal.project file, as well about the existence of the alias.
I simplified the shell. Previously, it was trying to build the haskell dependencies using nix. However, this was problematic because:
- Once in the shell, cabal build may decide to use different version, and hence was rebuilding. So we were building dependencies twices.
- Theses dependencies may fail to build, and the nix-shell may not start.
- It clutters the nix setup, because we had to detect the Haskell dependencies used by hls, and manually remove the broken plugins. The new approach is simpler, it just starts a shell with ghc, cabal and a few C dependencies (zlib, ...).

michaelpj

Personally, I'm in favour of what you do here, just having the dev shell contain the system dependencies and letting cabal build the rest. In fact, that's what I do already (I don't use the existing dev shells for that reason!). However, possibly some people (@berberman ?) like the way it's done currently.

flake.nix

jneira · 2022-02-02T10:26:42Z

the nix build in master is failing since https://github.com/haskell/haskell-language-server/actions/runs/1778521865 (the merge of #2668):

error: builder for '/nix/store/5hz103hms77w33chs4170kws3h5mxp63-haskell-language-server-1.6.1.0.drv' failed with exit code 1;
       last 10 log lines:
       > post-installation fixup
       > shrinking RPATHs of ELF executables and libraries in /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0
       > shrinking /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/bin/haskell-language-server-wrapper
       > shrinking /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/bin/haskell-language-server
       > shrinking /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/lib/ghc-8.10.7/x86_64-linux-ghc-8.10.7/libHShaskell-language-server-1.6.1.0-KvnnT3SgOLdLTY2wE4rDC1-ghc8.10.7.so
       > strip is /nix/store/xiq6j4jsyj351p8q3yw9cg1hdqp9m685-gcc-wrapper-10.3.0/bin/strip
       > stripping (with command strip and flags -S) in /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/lib  /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/bin
       > patching script interpreter paths in /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0
       > checking for references to /build/ in /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0...
       > RPATH of binary /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/bin/haskell-language-server contains a forbidden reference to /build/
       For full logs, run 'nix log /nix/store/5hz103hms77w33chs4170kws3h5mxp63-haskell-language-server-1.6.1.0.drv'.
Error: Process completed with exit code 1.

guibou · 2022-02-07T16:34:29Z

I've pushed a change with both type of shell (i.e. simple deps where cabal builds the dependencies, or complete deps).

I've also changed the CI so it builds HLS and shells for all flavors of them, effectively populating cachix for them.

guibou · 2022-02-07T16:35:28Z

the nix build in master is failing since https://github.com/haskell/haskell-language-server/actions/runs/1778521865 (the merge of #2668):

error: builder for '/nix/store/5hz103hms77w33chs4170kws3h5mxp63-haskell-language-server-1.6.1.0.drv' failed with exit code 1;
       last 10 log lines:
       > post-installation fixup
       > shrinking RPATHs of ELF executables and libraries in /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0
       > shrinking /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/bin/haskell-language-server-wrapper
       > shrinking /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/bin/haskell-language-server
       > shrinking /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/lib/ghc-8.10.7/x86_64-linux-ghc-8.10.7/libHShaskell-language-server-1.6.1.0-KvnnT3SgOLdLTY2wE4rDC1-ghc8.10.7.so
       > strip is /nix/store/xiq6j4jsyj351p8q3yw9cg1hdqp9m685-gcc-wrapper-10.3.0/bin/strip
       > stripping (with command strip and flags -S) in /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/lib  /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/bin
       > patching script interpreter paths in /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0
       > checking for references to /build/ in /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0...
       > RPATH of binary /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/bin/haskell-language-server contains a forbidden reference to /build/
       For full logs, run 'nix log /nix/store/5hz103hms77w33chs4170kws3h5mxp63-haskell-language-server-1.6.1.0.drv'.
Error: Process completed with exit code 1.

Yes:

       > RPATH of binary /nix/store/y2m3d2c0a1wa1j0p7skgvfvzm9sqnh5d-haskell-language-server-1.6.1.0/bin/haskell-language-server contains a forbidden reference to /build/

I'm having a look.

guibou · 2022-02-07T17:28:28Z

@jneira the build is now fixed, but now that HLS is not statically linked anymore, it does have the nix ghc package as a dependency, which bumps the uncompressed closure size to 6 GiB.

I initially thought that it should not be an issue, because people are actually using HLS to develop with GHC, so chances are that they also pull ghc in their dev environment. However, due to how nix flake works, they may use a different nixpkgs clone, and hence use a different ghc.

configuration-ghc-921.nix

guibou · 2022-02-08T16:36:02Z

Developement shell for macos for GHC 8.8.4 is failing.

I have no idea how to fix that, I'll disable it.

Theses tools are pulled in the shell, but we don't need them to match the GHC version used for development. Said otherwise, as long as we use a working `cabal-install` to build with GHC 9.2, we don't care if that cabal-install was built with GHC 8.10. This gives more chance for theses tools to work and be in the binary cache of nixpkgs.

Using alias, we get the "correct" behavior when typing `cabal build` in the nix-shell, it points to the current cabal-project file.

This commit restores a working behavior for `nix-shell` or `nix develop`, for all supported GHC versions. When entering a `nix-shell`, or `nix develop .#haskell-language-server-ghcXxX-dev` you will get an environment with all the dependencies needed to run `cabal build`. Haskell dependencies will be built by cabal instead of nix. This may be more robust compared to a shell where dependencies are built by nix: - It won't try to build any dependency with nix. This mean that it won't benefit from nix caching, but on the other hand, it will perfectly match the cabal plan. - The nix shell may fail to start if an (possibly unneeded) dependency was failing to build. Entering nix-shell also generates an alias command `cabal_project` which runs cabal with the correct project file if needed (e.g. for GHC 9.0 and 9.2). Users are notified with a message when entering the shell. The old behavior (i.e. where dependencies are built with nix) is still available with `nix develop .#haskell-language-server-xxx-dev-nix` (i.e. suffix the name of the shell with `-nix`).

All HLS version can be built with `nix build .#all-haskell-language-server`, and all development shells with `nix build .#all-dev-shells`. I had to workaround a limitation in `nix build` which is not able to build multiples targets, instead I used linkFarmFromDrvs. All packages and shells are now named with a unique package name containing the GHC version.

Related to haskell#2668. This fixs the build, however, because now HLS is dynamicly linked, it pulls GHC as a dependency. The uncompressed closure size is now `~6GiB`.

guibou · 2022-02-12T13:59:41Z

I've disable nix dev shell building for all but the current GHC version. (i.e. that's strictly more than what was previously tested). Let's see if it passes test like that, then we'll merge and later I'll try to fix it.

…ells

guibou · 2022-02-16T08:03:49Z

I don't understand the latest failing job. Could we merge with that?

michaelpj · 2022-02-16T12:24:19Z

LGTM!

michaelpj · 2022-02-19T16:48:48Z

Now trying to actually use this:

> nix develop .#haskell-language-server-ghc921-dev 

error: flake 'git+file:///home/michael/projects/forks/haskell-language-server' does not provide attribute 'devShells.x86_64-linux.haskell-language-server-ghc921-dev', 'packages.x86_64-linux.haskell-language-server-ghc921-dev', 'legacyPackages.x86_64-linux.haskell-language-server-ghc921-dev' or 'haskell-language-server-ghc921-dev'

> nix develop
> cabal build ghcide

session-loader/Development/IDE/Session/VersionCheck.hs:17:24: error:
    • Exception when trying to run compile-time code:
        I could not find a GHC installation at /nix/store/2dgrlf17sdjg3vjgs637xbj8kfmd3yxw-ghc-8.10.7/lib/ghc-8.10.7. Please do a clean rebuild and/or reinstall GHC.
CallStack (from HasCallStack):
  error, called at src/GHC/Check.hs:188:7 in ghc-check-0.5.0.6-29a80c8b99e2ee9e01c4d3d0858431da25b86b57be8c1c611f95e6a5d1788945:GHC.Check
      Code: makeGhcVersionChecker
              (fromMaybe GHC.Paths.libdir <$> lookupEnv "NIX_GHC_LIBDIR")
    • In the Template Haskell splice
        $$(makeGhcVersionChecker
             (fromMaybe GHC.Paths.libdir <$> lookupEnv "NIX_GHC_LIBDIR"))
      In the expression:
        $$(makeGhcVersionChecker
             (fromMaybe GHC.Paths.libdir <$> lookupEnv "NIX_GHC_LIBDIR"))
      In an equation for ‘ghcVersionChecker’:
          ghcVersionChecker
            = $$(makeGhcVersionChecker
                   (fromMaybe GHC.Paths.libdir <$> lookupEnv "NIX_GHC_LIBDIR"))

🤷

michaelpj · 2022-02-19T16:53:36Z

Setting NIX_GHC_LIBDIR helped me with the second one.

guibou · 2022-02-19T17:34:44Z

I'm sorry.

I tried your second example and it works fine for me. Have you tried cabal clean first? But first, could you grep -R 2dgrlf17sdjg3vjgs637xbj8kfmd3yxw-ghc-8.10.7 *. I'm suspecting that there is an hidden / leftover state somewhere. For one reason, I do not have /nix/store/2dgrlf17sdjg3vjgs637xbj8kfmd3yxw-ghc-8.10.7 on my machine, so you have a reference to something that my invocation of nix-shell or nix-develop did not pulled.

For the first one

nix develop .#haskell-language-server-ghc921-dev

I can indeed reproduce the problem ;( However it works for me with nix develop .\#haskell-language-server-921-dev (see the missing ghc in my invocation). Is that just an error from your side, or did you read this line from a documentation / example / something and that there is something to fix?

michaelpj · 2022-02-19T20:31:53Z

No need to apologize, just reporting!

I'm suspecting that there is an hidden / leftover state somewhere.

Yes, I suspect you may be right. cabal clean didn't fix it, but I suspect the path may actually have crept into one of the packages in the cabal store...

Is that just an error from your side, or did you read this line from a documentation / example / something and that there is something to fix?

Aha, that does it! I just blindly copied it out of the PR description because I was trying to figure out how to trigger one of the other shells. I should have checked more closely! It does indeed work with the right attribute name 🤦

guibou · 2022-02-19T22:23:49Z

Aha, that does it! I just blindly copied it out of the PR description because I was trying to figure out how to trigger one of the other shells. I should have checked more closely! It does indeed work with the right attribute name facepalm

I use auto completion (with zsh or bash) to get the available entry points. nix flake show is also supposed to do that, but I had never been able to get it to work.

guibou force-pushed the nix_flake_fix_devShells branch from 6a313b4 to 9958922 Compare January 31, 2022 21:26

guibou marked this pull request as ready for review January 31, 2022 21:27

guibou requested review from berberman and michaelpj as code owners January 31, 2022 21:27

guibou changed the title ~~WIP: Nix flake fix dev shells~~ Nix flake fix dev shells Jan 31, 2022

michaelpj approved these changes Jan 31, 2022

View reviewed changes

flake.nix Show resolved Hide resolved

flake.nix Show resolved Hide resolved

berberman reviewed Jan 31, 2022

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

guibou force-pushed the nix_flake_fix_devShells branch from 912f631 to 9048cdc Compare February 7, 2022 16:30

guibou requested review from Ailrun and Anton-Latukha as code owners February 7, 2022 16:30

Ailrun reviewed Feb 8, 2022

View reviewed changes

configuration-ghc-921.nix Show resolved Hide resolved

georgefst referenced this pull request Feb 8, 2022

Attempt to use Fourmolu 0.5 on GHC 8.10.7

cf5cbec

michaelpj approved these changes Feb 9, 2022

View reviewed changes

guibou added 8 commits February 12, 2022 14:56

nix: ghc92: ignore broken plugins

b76ea56

nix: disable shake-bench for ghc921

6cd5891

nix: fix ghc 9.2 shell

7fed1fd

nix: explicit cabal project in devShell

ac96112

Using alias, we get the "correct" behavior when typing `cabal build` in the nix-shell, it points to the current cabal-project file.

nix: remove special case for different ghc versions

55bdbe9

nix: move implicit-hie-cradle as a flake input

7343de6

guibou added 5 commits February 12, 2022 14:56

nix: Build all devShells and all binaries in CI

fe53c28

nix: build HLS with dynamic linking

34c3911

Related to haskell#2668. This fixs the build, however, because now HLS is dynamicly linked, it pulls GHC as a dependency. The uncompressed closure size is now `~6GiB`.

nix; fix CI command

a4164ac

nix: only build dev shell with nix packages for current GHC version

00aa810

guibou force-pushed the nix_flake_fix_devShells branch from b398d1b to 00aa810 Compare February 12, 2022 13:56

guibou force-pushed the nix_flake_fix_devShells branch from 33cbffe to 00aa810 Compare February 12, 2022 19:15

guibou added 5 commits February 15, 2022 23:20

Another tentative to fix the build

021c5c0

Merge remote-tracking branch 'origin/master' into nix_flake_fix_devSh…

01849f2

…ells

fix: Eval and alternate number format works with ghc 9.2

499fd42

fix CI build

2a643d9

ANother tentative fix

0540fcf

michaelpj merged commit 96ea854 into haskell:master Feb 16, 2022

guibou deleted the nix_flake_fix_devShells branch February 16, 2022 17:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nix flake fix dev shells #2655

Nix flake fix dev shells #2655

guibou commented Jan 29, 2022 •

edited by gitpod-io bot

Loading

michaelpj commented Jan 29, 2022

guibou commented Jan 29, 2022

michaelpj commented Jan 29, 2022

guibou commented Jan 31, 2022

michaelpj left a comment

jneira commented Feb 2, 2022

guibou commented Feb 7, 2022 •

edited

Loading

guibou commented Feb 7, 2022

guibou commented Feb 7, 2022

guibou commented Feb 8, 2022

guibou commented Feb 12, 2022

guibou commented Feb 16, 2022

michaelpj commented Feb 16, 2022

michaelpj commented Feb 19, 2022

michaelpj commented Feb 19, 2022

guibou commented Feb 19, 2022

michaelpj commented Feb 19, 2022

guibou commented Feb 19, 2022

Nix flake fix dev shells #2655

Nix flake fix dev shells #2655

Conversation

guibou commented Jan 29, 2022 • edited by gitpod-io bot Loading

michaelpj commented Jan 29, 2022

guibou commented Jan 29, 2022

michaelpj commented Jan 29, 2022

guibou commented Jan 31, 2022

michaelpj left a comment

Choose a reason for hiding this comment

jneira commented Feb 2, 2022

guibou commented Feb 7, 2022 • edited Loading

guibou commented Feb 7, 2022

guibou commented Feb 7, 2022

guibou commented Feb 8, 2022

guibou commented Feb 12, 2022

guibou commented Feb 16, 2022

michaelpj commented Feb 16, 2022

michaelpj commented Feb 19, 2022

michaelpj commented Feb 19, 2022

guibou commented Feb 19, 2022

michaelpj commented Feb 19, 2022

guibou commented Feb 19, 2022

guibou commented Jan 29, 2022 •

edited by gitpod-io bot

Loading

guibou commented Feb 7, 2022 •

edited

Loading