Cabal package version macros

[snoyberg]

Original issue: https://github.com/fpco/fpco/issues/1167

Two changes:

1. Instead of inheriting the list of packages via environment variable, ide-backend should explicitly in its API receive a list (stack) of package databases.
2. ide-backend will generate a cabal_macros.h file based on all available packages and include this for all compilation (both exe building and normal compiling).

[snoyberg]

This issue is ready to be worked on.

[Mikolaj]

Re package databases:

1. I assume the package databases are given as a Maybe of a list of file paths.
2. I assume the package databases should be stacked on top of the global package db and the user package db should not be included.
3. I assume the package databases are specified only once, in session config, before session start and can't be changed afterwards.

Edit: actually, to make it easier to test, etc., I will implicitly add the user package db if the explicit list of package databases is Nothing.

Edit2: in the newest version of the code the UserPackageDB i controlled explicitly at session init, but the default again includes UserPackageDB.

[snoyberg]

I'm not sure about the global package database, I think instead you should not assume that you're adding it, or at least provide an option for that. We may need to generate dummy global databases to implement a separate feature.

If we need to change the package databases, we can always kill the session configs and start them up again, so (3) seems like a fair assumption. @jwiegley Do you agree?

[Mikolaj]

To be sure we use the same terminology, the global database is the one containing rts, ghc-prim, base. Below is a snippet from http://www.haskell.org/ghc/docs/7.2.2/html/users_guide/packages.html

GHC knows about two package databases in particular:

The global package database, which comes with your GHC installation, e.g. /usr/lib/ghc-6.12.1/package.conf.d.

A package database private to each user. On Unix systems this will be $HOME/.ghc/arch-os-version/package.conf.d, and on Windows it will be something like C:\Documents And Settings\user\ghc\package.conf.d. The ghc-pkg tool knows where this file should be located, and will create it if it doesn't exist (see Section 4.9.6, “Package management (the ghc-pkg command)”).

[snoyberg]

Yes, that's what I'm referring to. We want to ensure that there is only ever one copy of a package in scope, so in some cases (e.g., user installs a new copy of binary), we would want to modify the global database. An individual user clearly shouldn't be allowed to do that, so what we might do is create a new database including all packages from the global database and then remove (e.g.) binary.

[Mikolaj]

OK, great, so if you give me a Just packageDBStack, I just set it as the current stack to use. It's your responsibility to put in there something with 'rts', etc., in it. And if you give me Nothing, I implicitly set the global and the user databases (that's useful at least for our tests, possibly for yours too).

[Mikolaj]

Cabal raises "unexpected package db stack" when there's no special case of the default GlobalPackageDB among the package db stack. So we need to patch cabal to allow stacks without the default GlobalPackageDB. The patch should be trivial, just disabling the check (unless @dcoutts points out this should be done differently). The refactoring of ide-backend to allow 2 different cabals is already approved on our list, so we'll start it soon.

Edit.: specifying DBs to use is ready (including the API) and we are ready to fork cabal and remove the check for GlobalPackageDB.

[jwiegley]

@snoyberg I agree to the question asked above.

[Mikolaj]

There is a problem with package dbs: in 7.4.2

http://www.haskell.org/ghc/docs/7.4.2/html/users_guide/packages.html#package-databases

as opposed to 7.6

http://www.haskell.org/ghc/docs/7.6.1/html/users_guide/packages.html#package-databases

we cannot disable (or substitute) the global database (it's reflected in the API and that's not a problem of exposing some names). We'd need to backport the new functionality to 7.4.2, which unfortunately invalidates the estimates for this part (DBs) of the feature. I'm pinging @dcoutts in hopes he proves me wrong.

Edit:
12:51 < snoyberg> for the record, compiling without the default global package database does in fact work with GHC 7.4.2 and 7.6.3

[Mikolaj]

It appears it's impossible to build exes with Cabal, using the required macros, without patching Cabal. The reason is Cabal overwrites the macros file with it's own as a first step of building and the subsequent steps are are not exposed in the API. So, at least in the fist prototype, exes will be built with Cabal using a smaller macros file (referring to just the real dependencies) than what the GHC API uses during the compilation (all packages in the dbs, because at this point (without the macros) the dependencies are not yet known.). That should only break freak code that refers to macros of packages that it does not use (the code will compile via GHC API, but not via cabal buildExe).

Edit: a possible solution (not to be attempted right now and not without confirmation):
16:23 < dcoutts> mikolaj: what we could (but I'm not suggesting we do do that now) do is to do (in ide-backend compilation via GHC API) the same thing that Cabal is doing
16:24 < dcoutts> ie we load the dep graph using a big cabal_macros.h
16:24 < mikolaj> oh, I see
16:24 < dcoutts> and then generate the usual small cabal_macros.h for loading the files normally
16:24 < mikolaj> yes, that'd fix it
16:24 < dcoutts> mikolaj: we can file a ticket explaining this, but I don't think we need to do it
16:24 < mikolaj> we'd need to expose some more stuff in GHC
16:24 < dcoutts> ?
16:25 < mikolaj> because we currently can't construct manually the 'load'
16:25 < mikolaj> without a substantial overhead

[Mikolaj]

I've just noticed the macros can't be generated for package DBs that contain multiple versions of the same package (we get multiple macro definitions). So I will be choosing the newest version and I assume that's OK (e.g., the package DBs we receive only have one version of each package).

[snoyberg]

It is safe to assume one package version. We may allow users to override that assumption, but will warn that they must do so at their own risk.

[Mikolaj]

This feature is ready for integration and testing (all internal ide-backend tests pass (some are on experimental, the basic ones on master)).