package-tests LBI reading (still) doesn't work

[ezyang]

@phadej and I had a discussion about this on IRC and I thought I should record the problem for posterity.

When we run the package-tests test suite (and some other test suites), we need to call GHC to build some Haskell files against the Cabal library which we are currently testing against. In particular, we need to know what package databases to use; depending on whether or not sandboxes, or new-build, or Stack are used, these databases will vary. The canonical location that this information is stored is in the LocalBuildInfo that the bootstrapping Cabal (the Cabal that builds Cabal) writes out.

Now, here's the problem: package-tests is linked against the NEW Cabal, not the bootstrapping Cabal. Which means there is no reason that the new Cabal should be able to read the bootstrapped LocalBuildInfo. And indeed, in most cases it does not (case in point, Travis's invocation of package-tests fails to read the LBI, and so it just "reconfigures" to try to come up with the parameters necessary.)

How can we fix this? There are a few ways but none of them seem to be convenient:

1. If there was a way to link package-tests against the bootstrap Cabal, we could just use the bootstrap Cabal to read in the LBI. But this is too much for Cabal's brain to handle right now (it's not expressible in the current build-depends syntax)
2. If we make a Custom setup script it has access to the correct LBI and can pass it on. But we don't really want to make Cabal's setup script Custom (causes other problems, like the build of Setup subsequently picking up all of the local hs files because the include path coincides). Another possibility is to turn package tests into its own package; not an unreasonable proposition!

[23Skidoo]

Another possibility is to turn package tests into its own package; not an unreasonable proposition!

IMO this is the only proposal that'll keep us sane, and we want to do this anyway for #3667.

[Blaisorblade]

If there was a way to link package-tests against the bootstrap Cabal, we could just use the bootstrap Cabal to read in the LBI. But this is too much for Cabal's brain to handle right now (it's not expressible in the current build-depends syntax)

What about reading LBI through a separate process linked against the right Cabal version?
I'm taking this from skimming cabal-helper docs.

[ezyang]

Yes that would work. The problem is making sure it links against the right Cabal.

[Blaisorblade]

cabal-helper IIUC solves this, but Cabal can't depend on it. Not sure how simple that is; if it's nontrivial, splitting the package would be good.
Elsewhere I suggest adding that into cabal-install, but this again won't help bootstrapping.

So I guess I should keep my suggestion to #2395 and refrain from derailing; someday this problem should be solved as a side effect of #2395, but we're not there yet.

[ezyang]

If I understand correctly, the way cabal-helper decides what version of Cabal to build against is it attempts to parse the top of the LocalBuildInfo to get out a version, which it then compiles. How does it know where to find the Cabal library? It has a pile of heuristics compileHelper of places to look; the global db, the sandbox db, and if everything fails it just downloads and build the damn Cabal version itself.

This strategy is annoying and I'd prefer not to use it. There is a single obviously correct version of Cabal to use (the one you used to build the package) and we should figure out how to get that information to the right place.

[Blaisorblade]

I guess you just made an excellent argument for supporting this properly in cabal 😄

[ezyang]

Fixed now when I separated out cabal-testsuite (#4054)