Sometimes buildExe compiles modules twice when once is enough

[Mikolaj]

BuildExe compiles a library from loaded modules and then compiles (possibly many) executables using the library (and thus avoiding recompilation of the modules for each executable in turn). In some cases (I can't see a clear pattern, though this seems to only happen for modules called Main, but in files not named "Main.hs"), buildExe compiles a module the second time, while it should take it from the library. E.g., in ghc-errors.hs test "Relative include paths (#156) with dynamic include path change" it produces the following progress messages list

[[1 of 1] Compiling B,[1 of 2] Compiling B,[2 of 2] Compiling Main]

while the correct list should be

[[1 of 1] Compiling B,[1 of 1] Compiling Main]

For comparison and illustration, for a single Main module in a Main.hs file, the progress messages, as currently produces, are correct and look as follows

[[1 of 1] Compiling Main,[1 of 1] Compiling Main]

The first compilation is as part of the library, the second as an exe. If buildExe is invoked with 4 executables to build, Main, Exe1, Exe2 and Exe3, and Exe* depend on the Main module, then Exe* don't need to recompile Main, but use it from the library, resulting in

[[1 of 1] Compiling Main,[1 of 1] Compiling Main,[1 of 1] Compiling Exe1,[1 of 1] Compiling Exe2,[1 of 1] Compiling Exe3]

[snoyberg]

Sorry if this is hijacking the issue, but I was just discussing something related with @dcoutts. If there are no source changes, would calling buildExe multiple times result in any duplicated compilation or linking? This could be a problem for us once we start implementing running binaries in our codebase.

[Mikolaj]

Originally, buildExe was intended to recompile only as much as needed (by keeping all old build directories intact between compilations). However, you discovered it leads to buggy behaviour and we implemented a quickfix (1.):

#119 (comment)

So, I guess, we should implement fix 2. or fix 3. (though I'm not sure it's valid, unless our code is only to work on Linux) and remove the quickfix 1. @edsko: do I get it right?

[edsko]

Backporting the fix only helps when running on a file system with high resolution filestamps. Some Linux file systems are in that category, but not all; and OSX's file system isn't either. That's why we didn't go ahead with backporting the fix.

[snoyberg]

Sorry, didn't realize you were waiting on my feedback. Option (2) from that issue sounds good.

[Mikolaj]

I've implemented and pushed a variant of fix 2. to #119 (and reverted the old workaround). Thanks to @edsko, I could get away with a much simpler actual fix code than anticipated. Nevertheless, debugging, preparing tests and testing took very long, as always, and I still can't say I understand all progress messages from the tests. The focus on convoluted and pathological cases in our tests doesn't help either. I'd need to extract all suspect tests, run them manually with ghc in verbose mode and match the obtained output with the progress lines from ide-backend. But I think the issue of recompilation in consecutive runs is fixed and if there are problems, they concern the initial issue --- unneeded compilation within a single run.

As a side effect of the better fix to #119, the initial example from the top of this issue "Relative include paths (#156) with dynamic include path change" now does not do any spurious recompilations. However the following test series still produce more progress messages than I can easily justify, so they are candidates for further investigation:

"Using the FFI with withIncludes and TargetsExclude"
"Build executable from ParFib.Main"
"Using C files"
"Switch from one to another"

[edsko]

See #237 (comment) .

[edsko]

I now even got a module compiled three times:

        We expected 4 progress messages, but got [[1 of 1] Compiling A,[1 of 2] Compiling A,[2 of 2] Compiling Main,[1 of 2] Compiling A,[2 of 2] Compiling Main]

Not sure what's going on..