Cabal will build and install broken library if other-modules is not complete

[ezyang]

This is a reopen of #121

Cabal computes the set of object files to link into an archive by using the exposed-modules/other-modules fields. This means that if an imported module is omitted, it isn't put into the archive.

That in and of itself isn't too bad (since the Cabal file is buggy), but THEN Cabal will happily proceed to install this broken archive, and then you'll get a mysterious linker error error when you try to use it. Disaster!

A number of resolutions were suggested in the previous ticket:

• Cabal should test install a library (i.e. make sure it loads) before it gets installed
• Cabal learns how to chase module dependencies through GHC (by some mechanism, maybe parsing -v output or running ghc -M)

Attached is a test case:
test.zip

[23Skidoo]

Cabal learns how to chase module dependencies through GHC (by some mechanism, maybe parsing -v output or running ghc -M)

The reason we don't do this is that running ghc -M before ghc --make slows the build down quite noticeably, especially on large projects (see #1455). I think this can be fixed by making ghc --make build the exe/library and emit the dependency graph in a single step.

[ezyang]

Yeah, because it has to redo all of the work parsing headers and transitively chasing everything down. I'm pretty sure this will require GHC cooperation to properly fix.

[ezyang]

I checked and it turns out that all the information you need is in the top-level interface files. So GHC should get a command akin to --abi-hash, which gets passed some number of modules, and which accurately reports the set of home modules which was depended upon.

[23Skidoo]

Would be also nice to have a command to get a list of extensions enabled in each module so that we could check that against default-extensions and other-extensions. This is sometimes required for correctness (e.g. with Template Haskell).

[ezyang]

This information is not currently in interface files. Can you expand on when it's necessary?

[23Skidoo]

This information is used during building, so I'd like to do this check on each cabal build and possibly also when doing cabal sdist/cabal check. Same for the other-modules/exposed-modules check.

[ezyang]

Oh I see, you mean the Template Haskell extension! I can't tell you if an arbitrary extension was enabled, but I CAN tell you if Template Haskell was enabled from an interface file.

Let me bring up another comment: we need a command line interface for this. This is a bit annoying, because the possibilities are basically endless. What should GHC accept as a flag?

[23Skidoo]

I can't tell you if an arbitrary extension was enabled, but I CAN tell you if Template Haskell was enabled from an interface file.

This is nice, but ideally I'd like to do this check for all extensions, though for most of them we can be more lenient (just print a warning), while omitting TH from other-extensions should be an error. Data from other-extensions is now used by the solver, so nudging people to keep it up to date is useful.

What should GHC accept as a flag?

If --make could be used together with -M (so that GHC would build the exe/library and output the dependency graph in one step) and the -M output was extended with information about extensions enabled in each module (this info can be printed in comments), that'd cover my use case.

If you want to implement this by parsing interface files, I guess I could use a command for dumping the dependency graph and a command for dumping a list of "extensions affecting compilation" (currently only TH, I think).

[ezyang]

Well, -M doesn't give extension info, does it?

If you want to implement this by parsing interface files, I guess I could use a command for dumping the dependency graph and a command for dumping a list of "extensions affecting compilation" (currently only TH, I think).

That's a bunch of commands. Would be nice to get some agreement on how to organize them, what their output should be, etc.

[23Skidoo]

@ezyang

Well, -M doesn't give extension info, does it?

My idea was that it could be changed to do that (in a backwards-compatible way, via Makefile comments). Perhaps this could be enabled via a flag to avoid breaking consumers not expecting comments in ghc -M output. So running our hypothetical ghc --make -dump-exts -M deps.d Foo.hs would both build the library and produce the following deps.d file:

# exts:
# Foo.hs : TemplateHaskell
# Bar.hs : MultiParamTypeClasses, RecursiveDo
# [...]

Foo.o : Bar.hi
Foo.o : Baz.hi
[...]

Hmm, actually, nothing requires us to output both pieces of info in the same file. So they of course could be separate.

Would be nice to get some agreement on how to organize them, what their output should be, etc.

For the dependency graph, the current ghc -M format is satisfactory. For dumping extensions, we can just print the module name and a comma-separated list of extensions, maybe in the Haskell literal format:

("Foo.hs", [TemplateHaskell])
("bar/Bar.hs", [MultiParamTypeClasses, RecursiveDo])

[23Skidoo]

For the dependency graph, the current ghc -M format is satisfactory.

Though there are some cases when ghc -M currently fails (DEPENDS pragma, CPP #include, TH's addDependentFile), see 23Skidoo/ghc-parmake#9 for discussion.

[ezyang]

I filed a ticket on GHC for simultaneous --make and -M: https://ghc.haskell.org/trac/ghc/ticket/11281#comment:1

I'm not really convinced by the API for getting out extensions.

[23Skidoo]

I'm not really convinced by the API for getting out extensions.

Care to elaborate?

[ezyang]

I dunno. (1) it seems annoying to parse and (2) it offends me aesthetically. Someone else should feel welcome to submit the GHC ticket requesting this though :)

[thomie]

Alternative solutions, that don't require teaching Cabal how to chase dependencies, or to get that information from the compiler.

"Cabal computes the set of object files to link into an archive by using the exposed-modules/other-modules fields.

• How about Cabal doesn't do that, but instead lets GHC do the linking. Wouldn't that solve this ticket? (maybe naive)
• Or, alternatively, add a flag -no-chase-dependencies to GHC. Similar to -hide-all-packages, it requires you to specify each module explicitly. If A.hs imports B.hs, you would do: ghc --make -no-chase-dependencies A.hs B.hs. Doing just ghc --make -no-chase-dependencies A.hs would be a compile error. Cabal already passes all modules it knows about to GHC, but GHC is currently just too smart to find the missing ones.

(GHC ticket caused by this bug not being fixed yet: https://ghc.haskell.org/trac/ghc/ticket/9009).

[ezyang]

thomie: It would, except for the bit where GHC does not actually know how to link static archives on not-Darwin systems. Which is a bit goofy and probably should be fixed! https://ghc.haskell.org/trac/ghc/ticket/11373#ticket

The second suggestion would work, and would be fairly easy to do.

[ttuegel]

I think we could solve this by

1. Mapping module names to source files in Cabal,
2. passing filenames, not module names, to GHC, and
3. clearing all the search paths on the GHC command line.

Cabal already knows how to map module names to source files for sdist. Am I missing an obvious problem with this solution?

[ezyang]

I can't think of any obvious problems! But it certainly has a few moving parts.