Custom resolvers download external dependencies for each project using them

[dpwiz]

I want to build an custom resolver snapshot with a bunch of packages (all of them, actually 😟) from my local GIT repository. The download is fine (although I'd prerfer a shallow one), but instead of having a STACK_ROOT full of compiled packages I end up with each STACK_ROOT-using project download a compile all the packages for themselves, wasting build time and space.

Steps to reproduce

lts-12.4-extra.yaml

resolver: lts-12.4
name: WithExtras

packages:
- git: [email protected]:haskell-compat/base-compat.git
  commit: feda8f5f39d47c701eb9edc2efb07cc97bcf3542
  subdirs:
  - base-compat

1. Create a new project: stack new exam --resolver=../lts-12.4-extra.yaml
2. cd exam and stack build it.

(Optionally) Repeat 1 & 2 to make a few instances using the resolver.

Expected

Package source gets downloaded into ~/.stack/downloaded/... to be shared across all the packages.

Actual

Package source gets downloaded into .stack-work/downloaded/... of each package using the custom snapshot.

2018-08-01 17:13:24.202849: [info] Cloning feda8f5f39d47c701eb9edc2efb07cc97bcf3542 from [email protected]:haskell-compat/base-compat.git
@(src/Stack/PackageLocation.hs:208:17)
2018-08-01 17:13:24.202972: [debug] Run process within /home/dpwiz/src/os/stack-local-download/exam/.stack-work/downloaded/: /usr/bin/git clone --recursive [email protected]:haskell-compat/base-compat.git /home/dpwiz/src/os/stack-local-download/exam/.stack-work/downloaded/LBKlC1u3csXl
@(src/Stack/PackageLocation.hs:209:17)
Cloning into '/home/dpwiz/src/os/stack-local-download/exam/.stack-work/downloaded/LBKlC1u3csXl'...
remote: Counting objects: 3288, done.
remote: Total 3288 (delta 0), reused 0 (delta 0), pack-reused 3288
Receiving objects: 100% (3288/3288), 395.47 KiB | 436.00 KiB/s, done.
Resolving deltas: 100% (1459/1459), done.
2018-08-01 17:13:26.943025: [debug] Process finished in 2740ms: /usr/bin/git clone --recursive [email protected]:haskell-compat/base-compat.git /home/dpwiz/src/os/stack-local-download/exam/.stack-work/downloaded/LBKlC1u3csXl
@(src/Stack/PackageLocation.hs:209:17)
2018-08-01 17:13:26.943185: [debug] Run process within /home/dpwiz/src/os/stack-local-download/exam/.stack-work/downloaded/LBKlC1u3csXl/: /usr/bin/git --git-dir=.git reset --hard feda8f5f39d47c701eb9edc2efb07cc97bcf3542 --
@(src/Stack/Prelude.hs:141:10)
2018-08-01 17:13:26.954996: [debug] Process finished in 12ms: /usr/bin/git --git-dir=.git reset --hard feda8f5f39d47c701eb9edc2efb07cc97bcf3542 --
@(src/Stack/Prelude.hs:141:10)
2018-08-01 17:13:26.958336: [debug] Start: getPackageFiles /home/dpwiz/src/os/stack-local-download/exam/.stack-work/downloaded/LBKlC1u3csXl/base-compat/base-compat.cabal
@(src/Stack/Package.hs:332:21)
2018-08-01 17:13:27.037128: [debug] Finished in 79ms: getPackageFiles /home/dpwiz/src/os/stack-local-download/exam/.stack-work/downloaded/LBKlC1u3csXl/base-compat/base-compat.cabal
@(src/Stack/Package.hs:332:21)

Stack version

$ stack --version
Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2

Method of installation

• Official binary, downloaded from stackage.org or fpcomplete's package repository

[mihaimaruseac]

That's indended, since external dependencies can change outside of stack's control they are always downloaded to the project's local snapshot.

[dpwiz]

I'm confused. I thought "snapshotting" dependencies to have them unchanged from the moment on is the purpose of the custom snapshot. I have a list of git URLs and revisions in the resolver configuration, so every time I download them I should get same code...

Could you provide more details on what could prevent stack from working properly in a case like this?

[dpwiz]

https://www.fpcomplete.com/blog/2017/07/stacks-new-extensible-snapshots#custom-snapshots

Unlike defining packages in stack.yaml, packages built in a custom snapshot can be shared in the package cache and reused between projects.

But they aren't "shared in the package cache" 😞

You can use all four types of package locations when defining a custom snapshot.

But local paths are prohibited:

Custom snapshots do not support filepaths, as the contents may change over time.

And now I wonder: why can't I take responsibility of maintaining sources unchanged if stack relies on that?

[mihaimaruseac]

Sorry, I missread the issue, my bad.

[dpwiz]

Oh. Let me know if I can be of assistance.

[snoyberg]

I may sound like a broken record soon, but: this is being addressed by the pantry refactoring. It's not quite ready for testing, but the pantry branch should already avoid downloading multiple times.

(By "not quite ready for testing," I mean that I haven't implemented the repo cloning code yet 😄.)

[dbaynard]

This is a duplicate of #3551

[dbaynard]

Yeah, @wiz there's a workaround of sorts in that issue. I'll leave it to @snoyberg to decide which to close, given he's working on this right now.

[snoyberg]

Let's close in favor of #3551, since that one's the original.