Touching preprocessor modules (alex, happy, c2hs) doesn't trigger a rebuild

[mtolly]

Steps to reproduce:

new-template.cabal:

name:                new-template
version:             0.1.0.0
build-type:          Simple
cabal-version:       >= 1.10

library
  exposed-modules:     A, B, C, D
  build-depends:       base >= 4.7 && < 5, array
  default-language:    Haskell2010
  ghc-options:         -fno-warn-tabs

stack.yaml:

flags: {}
packages:
- '.'
extra-deps: []
resolver: nightly-2015-06-28

A.hs:

module A where

B.chs:

module B where

C.x:

{
module C where
}

%wrapper "basic"

tokens :-

[\ ] ;

{ 
}

D.y:

{
module D where
}

%name parse
%tokentype { () }

%token
  '()' { () }

%%

Unit : '()' { () }

{ 
happyError = undefined
}

Expected:

touching any of the four modules triggers a rebuild.

Actual:

$ stack build
new-template-0.1.0.0: configure
Configuring new-template-0.1.0.0...
new-template-0.1.0.0: build
Building new-template-0.1.0.0...
Preprocessing library new-template-0.1.0.0...
[1 of 4] Compiling D                ( .stack-work/dist/x86_64-osx/Cabal-1.22.2.0/build/D.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.2.0/build/D.o )
[2 of 4] Compiling C                ( .stack-work/dist/x86_64-osx/Cabal-1.22.2.0/build/C.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.2.0/build/C.o )
[3 of 4] Compiling B                ( .stack-work/dist/x86_64-osx/Cabal-1.22.2.0/build/B.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.2.0/build/B.o )
[4 of 4] Compiling A                ( A.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.2.0/build/A.o )
In-place registering new-template-0.1.0.0...
new-template-0.1.0.0: install
Installing library in
/Users/mtolly/git/stackbugs/.stack-work/install/x86_64-osx/nightly-2015-06-28/7.10.1/lib/x86_64-osx-ghc-7.10.1/newte_4CYw9uk75Lc15H36S6c3aX
Registering new-template-0.1.0.0...
$ touch A.hs
$ stack build
new-template-0.1.0.0-1476e217c0863a8960511bcc73ce9ef2: unregistering (local file changes)
new-template-0.1.0.0: build
Building new-template-0.1.0.0...
Preprocessing library new-template-0.1.0.0...
[4 of 4] Compiling A                ( A.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.2.0/build/A.o )
In-place registering new-template-0.1.0.0...
new-template-0.1.0.0: install
Installing library in
/Users/mtolly/git/stackbugs/.stack-work/install/x86_64-osx/nightly-2015-06-28/7.10.1/lib/x86_64-osx-ghc-7.10.1/newte_4CYw9uk75Lc15H36S6c3aX
Registering new-template-0.1.0.0...
$ touch B.chs
$ stack build
$ touch C.x
$ stack build
$ touch D.y
$ stack build --verbose
Version 0.1.1.0
2015-07-01 16:11:18.16841: [debug] Checking for project config at: /Users/mtolly/git/stackbugs/stack.yaml @(stack-0.1.1.0:Stack.Config src/Stack/Config.hs:489:9)
2015-07-01 16:11:18.168702: [debug] Loading project config file stack.yaml @(stack-0.1.1.0:Stack.Config src/Stack/Config.hs:512:13)
2015-07-01 16:11:18.176584: [debug] Run process: ghc --info @(stack-0.1.1.0:System.Process.Read src/System/Process/Read.hs:253:3)
2015-07-01 16:11:18.236065: [debug] Run process: ghc --numeric-version @(stack-0.1.1.0:System.Process.Read src/System/Process/Read.hs:253:3)
2015-07-01 16:11:18.265797: [debug] Run process: ghc-pkg --no-user-package-db field --simple-output Cabal id @(stack-0.1.1.0:System.Process.Read src/System/Process/Read.hs:253:3)
2015-07-01 16:11:18.293125: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack-0.1.1.0:System.Process.Read src/System/Process/Read.hs:253:3)
2015-07-01 16:11:18.32044: [debug] Checking resolver: nightly-2015-06-28 @(stack-0.1.1.0:Stack.Build.Source src/Stack/Build/Source.hs:76:13)
2015-07-01 16:11:18.637582: [debug] Run process: ghc-pkg --global --no-user-package-db dump --expand-pkgroot @(stack-0.1.1.0:System.Process.Read src/System/Process/Read.hs:253:3)
2015-07-01 16:11:18.683137: [debug] Run process: ghc-pkg --user --no-user-package-db --package-db /Users/mtolly/.stack/snapshots/x86_64-osx/nightly-2015-06-28/7.10.1/pkgdb/ dump --expand-pkgroot @(stack-0.1.1.0:System.Process.Read src/System/Process/Read.hs:253:3)
2015-07-01 16:11:18.757945: [debug] Run process: ghc-pkg --user --no-user-package-db --package-db /Users/mtolly/git/stackbugs/.stack-work/install/x86_64-osx/nightly-2015-06-28/7.10.1/pkgdb/ dump --expand-pkgroot @(stack-0.1.1.0:System.Process.Read src/System/Process/Read.hs:253:3)
2015-07-01 16:11:19.40135: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack-0.1.1.0:System.Process.Read src/System/Process/Read.hs:253:3)
$

[snoyberg]

Assigning to @chrisdone. I think this should just be a case of needing a few more built-in module extensions to match Cabal's behavior.

[chrisdone]

We have this for C file types.

[chrisdone]

And we have this for common Haskell preprocessors.

[mtolly]

Do note that Cabal supports preprocessing any extension via a custom setup. For example here's a project that uses embedded Ruby as a preprocessor for .erb files: https://gist.github.com/mtolly/265e41428d448049978d

This has the same issue (changing A.hs triggers rebuild, changing B.erb does not).

[chrisdone]

We can't use the Cabal library directly inside stack to read the dist/setup-config file due to version mismatches. It's not a good situation.

One possible hack is to, after running the configure step, write a Tmp.hs file which imports Distribution.Simple and prints to stdout the information that we want, and to run that file with runhaskell and consume the output to be written to the file modification time cache. This is icky because if I want to get all the "files" of a project in getPackage, the results ought to differ depending on whether I've configured or not.

[snoyberg]

Honestly, I think we may want to just have this as a documented limitation of stack for now. It can always be worked around by adding an explicit extra-source-files, unless I'm mistaken.

[snoyberg]

To clarify: I'm talking about custom preprocessors. Having support for the standard preprocessors out-of-the-box should hopefully be easy to do and cover the common use cases.

[chrisdone]

Indeed, I think extra-source-files is a good compromise.

[chrisdone]

This should do the trick:

• Added to the FAQ about custom preprocessors.
• Implemented support for .y, etc. and all the common ones, cribbed from Cabal in 8b63316.
• Implemented extra helpful messages in 9c61cc9 when resolving files. Example:

chris@retina:~/265e41428d448049978d$ stack build

Unable to find a known candidate for the Cabal entry "B", but did find: B.erb. If you are using a custom preprocessor for this module with its own file extension, consider adding the file(s) to your .cabal under extra-source-files.

It only shows this warning when it finds a corresponding B.<something> file in a source directory.

[chrisdone]

@mtolly Care to test it out?

[snoyberg]

LGTM. @mtolly if you see problems with this, please reopen.

[mtolly]

Everything looks good, thanks much!

[gergoerdi]

I'm using UUAGC and added the following to my .cabal file:

extra-source-files:
    src/Syn.ag
    src/TyChk.ag
    uuagc_options

However, stack build still complains about these files missing:

Unable to find a known candidate for the Cabal entry "Syn", but did find: Syn.ag. If you are using a custom preprocessor for this module with its own file extension, consider adding the file(s) to your .cabal under extra-source-files.
Unable to find a known candidate for the Cabal entry "TyChk", but did find: TyChk.ag. If you are using a custom preprocessor for this module with its own file extension, consider adding the file(s) to your .cabal under extra-source-files.

[mgsloan]

@gergoerdi Hi! Is there an easily downloadable reproduction of this issue? Feel free to open a new issue.

I've taken a look at stack's code, but having something concrete to work with would be good. I don't straightforwardly see logic for taking into account extra-source-files for these warnings. Perhaps the semantics is that if it's listed in extra-source-files and the name matches the module, omit the warning? Seems iffy, but I suppose it'd work in many cases.

[adituv]

@mgsloan I have just come across this issue; my cut-down reproduction is at adituv/protobuf@9f6987c

[mgsloan]

@adituv Interesting, thanks for the example! Should I expect a compilation error if I put non protobuf text in TestProto.proto? Currently it seems to build fine even after a clean. I've opened #3491 feel free to continue discussion there.

[adituv]

@mgsloan I removed my implementation and added a dummy implementation in Setup.hs; there should be no compilation errors no matter the contents of that file.