qAddDependentFile doesn't cause re-compilation without extra-source-files

[pbrisbin]

Steps to reproduce

1. Set up a repro project with stack new --bare repro

2. Add a file templates/example.hamlet

   <h1>Hi

3. Update Lib.hs to

   {-# LANGUAGE TemplateHaskell #-}
   
   module Lib
      ( someFunc
      ) where
   
   import Text.Blaze.Html.Renderer.Pretty
   import Text.Hamlet
   
   someFunc :: IO ()
   someFunc = putStrLn $ renderHtml $(shamletFile "templates/example.hamlet")

   (Add the shakespeareand blaze-html dependencies as necessary.)

4. Start up stack build --file-watch

5. Modify the contents of templates/example.hamlet

Expected

Modifying templates/example.hamlet should trigger re-compilation.

Actual

It does not. However, if I add

extra-source-files:
  - templates/*

To package.yaml, it works as expected. This should not be necessary as this packaging field is only documented as related to including files in sdist, not as a requirement for qAddDependentFile to work.

Stack version

$ stack --version
Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2

Method of installation

https://aur.archlinux.org/packages/stack-static/

Notes

It's definitely an assumption on my part that this boils down to qAddDependentFile. So I apologize if that's not actually the case. Another way to phrase it could be, external files referenced in TemplateHaskell don't cause re-compilation....

I've reproduced this bug with pretty much every file-reading TH function in a standard Yesod app (parseRoutesFile, widgetFile, etc).

I wouldn't have guessed this was Stack's responsibility either, but cocreature in #haskell encouraged me to report this bug here.

[dbaynard]

Thanks for the report. You may wish to try the new release of stack but I suspect this hasn't changed Edit: see below.

this packaging field is only documented as related to including files in sdist

This is the Cabal documentation, right?

@snoyberg how should we proceed?

[snoyberg]

I'm on my phone and can't search, but I'm fairly certain this is a bug that's been fixed on master. There was a change in the format of output from GHC.

[pbrisbin]

@dbaynard

This is the Cabal documentation, right?

Yeah, I was referring to this: https://www.haskell.org/cabal/users-guide/developing-packages.html#pkg-field-extra-source-files

[dbaynard]

@pbrisbin Do check the new version, when you get a chance — that should fix it.

[pbrisbin]

Unfortunately, this still reproduces for me with 1.9.

(In the upper-right you can see me modifying the template, but no re-compilation happens in the lower-right -- modulo that first initial compilation.)

[dbaynard]

Ok, I can reproduce

[snoyberg]

The change that should fix this is only on the master branch, not on 1.9.

[ygale]

It's this commit: aae15e3

I wishfully submitted #4369, hoping that this small but important fix might be patched to 1.9.3.

[ygale]

Unfortunately, it appears that fix does not solve this issue.

In GHC 8.2 and earlier, the .dump-hi file contains addDependentFile lines for the TH files. That is no longer the case in GHC 8.4.

[ygale]

The workaround suggested by @pbrisbin to add the templates to extra-source-files does not work for me. Stack does then notice that the template file has changed, but still does not rebuild the Haskell module that uses the template.

Stack has lost the association between template file and Haskell module. This is because starting in GHC 8.4, GHC does not seem to report template files in the .dump-hi file, which is where stack looks for it.

If you add the template file to extra-source-files and also pass the -fforce-recomp GHC option, then stack will notice the change and rebuild all modules in the entire project. I can't find any way to get stack to rebuild just the affected module as in GHC 8.2 and below.

[ygale]

Hmm shakespeare explicitly reports its files to GHC as dependent files. This is starting to look like a GHC bug. I will investigate further.

[tolysz]

@ygale only the version shakespeare-2.0.19+ will report it.

[ygale]

@tolysz Ahh yes that was it. Thanks!!

In summary: To resolve this issue you need:

• stack >= 1.9.3
• shakespeare >= 2.0.19

As of this writing stack 1.9.3 is not yet released. For now, you can build stack 1.9.2 from the stable branch on github and it will work.

If you use Stackage LTS, upgrade to at least 12.14, currently the latest, to get the right version of shakespeare. Or specify shakespeare-2.0.19 in the extra-deps section of your stack.yaml.

A million thanks to the stack team for getting this fixed!