"Right Nothing" printed to the console during setup

[silky]

with

>stack --version
Version 1.1.3, Git revision 8f7e62fbe3620a4b0ae5eb833cebfa50820f5050 x86_64 hpack-0.14.0

when i attempt a setup i see:

⚡ 05:37 PM noon ∈ ~>stack --resolver nightly-2016-05-13 setup
Run from outside a project, using implicit global project config
Using resolver: nightly-2016-05-13 specified on command line
Downloaded nightly-2016-05-13 build plan.    
Did not find .cabal file for servant-yaml-0.1.0.0 with Git SHA of d7c8a8eaf9e7c215433f8f39daac7fc6cf116325
Right Nothing
Did not find .cabal file for reducers-3.12.1 with Git SHA of bcebf11be1770ac480d9d0ee6238044f143a57f8
Right Nothing
Did not find .cabal file for math-functions-0.1.7.0 with Git SHA of 5870cda6259e8a838c8cb22043b9d3cbf55d9a8e
Right Nothing
Did not find .cabal file for hw-prim-0.0.0.10 with Git SHA of 3658e799df0b7fceb045899b7a3d686b960b5d01
Right Nothing
Did not find .cabal file for hspec-megaparsec-0.1.1 with Git SHA of 116c887f005a6e599dc9b6ce5627f1b69782f44b
Right Nothing
Did not find .cabal file for dixi-0.6.9.1 with Git SHA of 17cba908e23588d9e5b67541f53afd8590c5c30f
Right Nothing
Did not find .cabal file for cron-0.4.0 with Git SHA of d485877cb0e43ee1b8fcdaa0a7224acbd753186c
Right Nothing
stack will use a locally installed GHC
For more information on paths, see 'stack path' and 'stack exec env'
To use this GHC and packages outside of a project, consider using:
stack ghc, stack ghci, stack runghc, or stack exec

it's mildly unfortunate that Right Nothing is printed here.

introduced a few weeks ago by f483e82#diff-2068556e565acc62de252ac25ed550c4R288

[mgsloan]

Good point, fixed!

[mgsloan]

Then again, perhaps the root issue is not addressed - why are these SHAs missing?

[silky]

i'm not too sure about that one.

let me know if you want more information from me.

[mgsloan]

Oh yeah, to be clear that wasn't a request for more information. More like "this needs further investigation" :)

The failure case here is just a fallback to the old behavior where cabal metadata wasn't fixed to a particular revision. So thankfully this shouldn't be very problematic in practice. Good to fix, though!

[schell]

I seem to be running into a similar issue, though the output is a bit different - notably the "Left ..." result:

> stack --version
Version 1.1.2, Git revision cebe10e845fed4420b6224d97dcabf20477bbd4b (3646 commits) x86_64 hpack-0.14.0

> stack build
Downloading nightly-2016-05-20 build plan ...���������������������������������������������                                             ���������������������������������������������Downloaded nightly-2016-05-20 build plan.
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...����������������������������������������������������������������������������������������������������������������                                                                                                                ����������������������������������������������������������������������������������������������������������������Downloading package index from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...
����������������������������������������������������������������������������������������������������������������                                                                                                                ����������������������������������������������������������������������������������������������������������������Populating index cache ...��������������������������                          ��������������������������Populated index cache.
Did not find .cabal file for zot-0.0.3 with Git SHA of f9092bae03d0044d2b4e3f33609c9544e8c46b19
Left /root/.stack/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for zlib-lens-0.1.2.1 with Git SHA of b5797267b30c16afcccbd42fa0ce4b9fa58a7ca9
Left /root/.stack/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for zlib-enum-0.2.3.1 with Git SHA of 5bb46141c62ea8a473a2cfac9293a018c804189a
...

I'm using stack on bitbucket's new "pipelines" feature which executes stack from the haskell:7.10.3 docker image.

[schell]

The result of using lts-5.2 is similar but different:

stack build
Downloading lts-5.2 build plan ...����������������������������������                                  ����������������������������������Downloaded lts-5.2 build plan.
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...����������������������������������������������������������������������������������������������������������������                                                                                                                ����������������������������������������������������������������������������������������������������������������Downloading package index from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...����������������������������������������������������������������������������������������������������������������                                                                                                                ����������������������������������������������������������������������������������������������������������������Populating index cache ...��������������������������                          ��������������������������Populated index cache.
/opt/atlassian/bitbucketci/agent/build/.stack-work/downloaded/NC9VHrq5ky0p/: getDirectoryContents: does not exist (No such file or directory)

[simonmichael]

Another data point: I saw a bunch of these messages after upgrading stack (from 1.1.0 x86_64 hpack-0.13.0 to 1.1.2 x86_64 hpack-0.14.0) and running stack install. I did stack clean and they went away.

[sjakobi]

I see the Did not find .cabal file messages every time I build a project with a different (dev) version of stack.

Edit: I actually see these messages also for other commands, like stack path.

[DanBurton]

I've seen these messages lately as well, seemingly related to using the latest stack in environments that don't have a ~/.stack directory yet. For example:

(Modified instructions from trying to reproduce this bug)

docker run -it danburton/stackage-build-server:2016-07-11
cd ~
stack unpack gi-gtk-3.0.4
cd gi-gtk-3.0.4
stack init --resolver nightly-2016-06-08

Right Nothing
Did not find .cabal file for text-show-3.2.2 with Git SHA of dd7d87942bbe93e92db8be47a37172ac12be63fa
Right Nothing
Did not find .cabal file for socks-0.5.5 with Git SHA of a4fe3dade53c652f5ef4d04218d17a989e8414e1
Right Nothing
Did not find .cabal file for servant-yaml-0.1.0.0 with Git SHA of 2ef4721a30ef0c982f26f67688cfa09a307a62fc
Right Nothing
Did not find .cabal file for servant-swagger-1.1 with Git SHA of a35531ce5ae19faf483b4a7c5ceb73b67b0612e6
Right Nothing
Did not find .cabal file for servant-lucid-0.7.1 with Git SHA of cf726c02fbee0a185c0b861fcd8cbf1863c7b414

Last month when I ran this with snoyberg/stackage:nightly, these things were not printed to the console.

[yacinehmito]

I still experience the same bug.

[rainbyte]

I'm experiencing the same issue inside a docker container:

When I run the cmd

stack install --stack-root $PWD/.stack-root/ --work-dir .stack-work-docker/ --local-bin-path artifacts/

It shows this

Downloading lts-6.10 build plan ...
Downloaded lts-6.10 build plan.
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...
Downloading package index from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz
Populating index cache ...
Populated index cache.
Did not find .cabal file for zot-0.0.3 with Git SHA of f9092bae03d0044d2b4e3f33609c9544e8c46b19
Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for zlib-lens-0.1.2.1 with Git SHA of b5797267b30c16afcccbd42fa0ce4b9fa58a7ca9
Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)

And after various similar lines, I get

Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for AC-Vector-2.3.2 with Git SHA of 32d97a60969fe6912b33597e7399c407b5f2e39a
Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
/src/.stack-work-docker/downloaded/9rVosC0PeQVv/: getDirectoryContents: does not exist (No such file or directory)

[lehtoj]

I am new to Haskell and today I attempted to install Haskell for Windows using Stack. However, I get similar errors as in this issue. Number of errors is too long to be copied here, so I provide you with a link instead.

While I have clearly run stack install outside a project, I get same errors on the first time I run stack setup.

Version 1.1.2, Git revision c6dac65 (3647 commits) x86_64 hpack-0.14.0
Downloaded from https://www.stackage.org/stack/windows-x86_64-installer

[mgsloan]

I was working on getting https://github.com/lukexi/rumpus built on windows yesterday, and also ran into this. I think it just happens when you don't have git. We need to identify when we shouldn't expect to find any SHAs for snapshot packages.

[Blaisorblade]

What I got from looking at the code and hadn't got from reading here: master won't print Right Nothing or Left ..., but that fix is not yet released.
The "Did not find .cabal file" message is still there and not fully explained.

[lehtoj]

Another observation is that when I use Stack to download contents of stack_root, mirror is https://s3.amazonaws.com/hackage.fpcomplete.com, size ends up being 262MB. But when someone else in another country uses Stack, download mirror is Github and contents end up being 451MB. Is the size difference explained by missing cabal/SHA files, or is there something else amiss?

[Blaisorblade]

@KeeperB5 interesting catch! Do you have the two actual URLs for one to look at?
If mirrors aren't mirrors, debugging this issue isn't going to be fun.

[lehtoj]

Well, my index is fetched from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz, but I believe the GitHub mirror had https://github.com/commercialhaskell/all-cabal-hashes.

How does Stack pick a mirror and do we have any way to manually select a mirror?

[sjakobi]

I've had the "Did not find .cabal file" message pop up in the middle of a build while trying to reproduce #2422:

~/t/postgres-tmp> stack build --trace
...
attoparsec-0.13.0.2: copy/register
Did not find .cabal file for aeson-0.11.2.0 with Git SHA of 2057e715a488cd9d2456fdb1c6c4b0d6142c539e
aeson-0.11.2.0: configure
aeson-0.11.2.0: build
aeson-0.11.2.0: copy/register
...

I can't rule out that I had just started using another stack version in another thread.

@mgsloan:

I was working on getting https://github.com/lukexi/rumpus built on windows yesterday, and also ran into this. I think it just happens when you don't have git.

Are you saying you got the "Right Nothing" lines with stack-HEAD? Or do you only see the "Did not find .cabal file" lines?

The "Did not find .cabal file" lines certainly appear on Linux with git installed.

[Blaisorblade]

Further experiments (below) suggest the hashes are part of the snapshot, which makes sense to ensure reproducibility. Hence, we get a log entry for each cabal file in the snapshot considered which was updated since then. Furthermore, I understand the fallback uses a newer cabal file, limiting build reproducibility in the sense of #2217.

Under these circumstances, I'd propose to clone the entirety of all-cabal-hashes as a quick fix (currently 215M). This is not fine long-term since bandwidth is limited and since the repo will grow indefinitely, but a better fix wouldn't be immediate.

I imagine one could add tags for each snapshot, and stack could fetch those tags when needed, but the tags would be needed remotely.

Experiment Results

Using lts-6.0, we request for instance a package at revision 5, while git log reveals we're at revision 7.
Trying out the snapshot in the OP shows the same hash (revision 6) for servant-yaml, while we're now at revision 8. Generally, all old messages reappear, and there are new ones.

Did not find .cabal file for opaleye-0.4.2.0 with Git SHA of 9b5608ac701b60a82f5cf985887d0d1450757864
Right Nothing

$ git show 9b5608ac701b60a82f5cf985887d0d1450757864|grep x-revi
x-revision: 5

$ git log -p current-hackage -- opaleye/0.4.2.0/opaleye.cabal
[...]
-x-revision: 6
+x-revision: 7

[Blaisorblade]

@rainbyte In your new issue, can you also add whether stack update fixes it?
I've just gotten that after rm -rf ~/.stack/indices/Hackage/git-update—somehow, stack did not figure out it needed to refetch the repo.

[Blaisorblade]

@KeeperB5: mirroring turns out to not be the problem, but here are some answers.
But since you were installing from scratch on Windows, here's the short of it:
TL;DR.: Install Git and ensure it is on the PATH so Stack can use it to update the index faster. Otherwise, Stack will fall back to downloading an index snapshot via HTTP from scratch each time.

Below's more info than you'd want, saving it for me.

Well, my index is fetched from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz, but I believe the GitHub mirror had https://github.com/commercialhaskell/all-cabal-hashes.

Those are the correct URLs, and they're related but not exactly mirrors, so the size difference is expected. https://github.com/commercialhaskell/all-cabal-hashes has extra metadata on top of https://github.com/commercialhaskell/all-cabal-files, which is a mirror in Git format of the tarball (which is a mirror of the Hackage index on the Amazon cloud).

How does Stack pick a mirror and do we have any way to manually select a mirror?

Basically, mirroring is not geographically based (except whatever optimizations Amazon does internally in S3). Make sure your stack can find your Git and you don't change your config, and stack will use git to update the index incrementally and faster (since you can download only the changes rather than a new snapshot), as described in these docs.

The relevant config is described here, https://docs.haskellstack.org/en/latest/yaml_configuration/#package-indices, but I'd avoid changing it usually.

With the default config, on both master and 1.1.2, the HTTP download (which is slower) is not used and the Git one is preferred, as long as Git is installed. So either stack doesn't find your Git or you changed your config.

Sources

Updating logic:

stack/src/Stack/PackageIndex.hs

Lines 214 to 223 in aac2c81

	updateIndex menv index =
	do let name = indexName index
	logUpdate mirror = $logSticky $ "Updating package index " <> indexNameText (indexName index) <> " (mirrored at " <> mirror <> ") ..."
	git <- isGitInstalled menv
	case (git, indexLocation index) of
	(True, ILGit url) -> logUpdate url >> updateIndexGit menv name index url
	(True, ILGitHttp url _) -> logUpdate url >> updateIndexGit menv name index url
	(_, ILHttp url) -> logUpdate url >> updateIndexHTTP name index url
	(False, ILGitHttp _ url) -> logUpdate url >> updateIndexHTTP name index url
	(False, ILGit url) -> logUpdate url >> throwM (GitNotAvailable name)

Default config (matching docs):

stack/src/Stack/Config.hs

Lines 219 to 228 in aac2c81

	configPackageIndices = fromFirst
	[PackageIndex
	{ indexName = IndexName "Hackage"
	, indexLocation = ILGitHttp
	"https://github.com/commercialhaskell/all-cabal-hashes.git"
	"https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz"
	, indexDownloadPrefix = "https://s3.amazonaws.com/hackage.fpcomplete.com/package/"
	, indexGpgVerify = False
	, indexRequireHashes = False
	}]

[lehtoj]

I've been using Visual Studio and SourceTree which don't need Git to be installed separately. Anyhow, I installed Git for Windows, deleted my stack_root directory, ran stack setup. Now it seems to work as intended, no more errors.

Perhaps install documentation should mention requirement for Git?

[Blaisorblade]

Perhaps install documentation should mention requirement for Git?

#2469

I've been using Visual Studio and SourceTree which don't need Git to be installed separately. Anyhow, I installed Git for Windows, deleted my stack_root directory, ran stack setup. Now it seems to work as intended, no more errors.

Note these were warnings. Should you get further warnings of this kind, you can mostly ignore them—and we'll have a fix by next release.
The real advantage with Git is that index updates will be much faster.

[lehtoj]

Noted. :) But the log does not really specify either way, Usually "The system cannot find the path specified." is treated as an error, so I just followed the suit since I didn't know better.

[dpwiz]

Receiving objects: 0% (3467/576096), 1.97 MiB | 19.00 KiB/s

Ouch! I never experienced the problems that prompted this fix but can't proceed now.
You'd better add something like --do-no-harm or --fix-2175 next time. Until then, gotta return to previous stack... 😔

[Blaisorblade]

@wiz Is that your usual connection speed?

[dpwiz]

I hope not. It downloaded eventually, but in a pinch that got me nervous.

[dpwiz]

Now there's real pain. My CI have this restricted uplink and should start from scratch each time 😔

[sjakobi]

@wiz: Can't you cache the ~/.stack directory?