Package (streamly) with *.m file in c-sources fails to build with GHC 9.8.2/Cabal-3.10.2.0

[AliceRixte]

The streamly library fails to build on the latest nightly resolver. On this related issue in the streamly repository : composewell/streamly#2844 you will find a PR that reproduces the fail on a CI.

cabal build works fine on the file generated by stack, so it looks like this is a stack issue.

Steps to reproduce

1. Use a stack.yaml file with one of the following resolvers :
   resolver: nightly-2024-06-20
resolver: nightly-2024-09-18
2. Use a package.yaml file with the following dependency :
   - streamly >= 0.10.1
3. Run command stack build.

Expected

streamly builds properly

Actual

The build fails after having successfully built all the files from streamly :

streamly    > <<ghc: 124151944520 bytes, 2919 GCs, 119349929/206603200 avg/max bytes residency (44 samples), 574M in use, 0.001 INIT (0.001 elapsed), 60.617 MUT (97.917 elapsed), 23.990 GC (24.379 elapsed) :ghc>>

streamly    > ar: .stack-work/dist/aarch64-osx/ghc-9.8.2/build/src/Streamly/Internal/FileSystem/Event/Darwin.o: No such file or directory

Stack version

stack --version
Version 3.1.1, Git revision d8cef4ea052430ea09ce7836c41e68b52d04fd73 aarch64 hpack-0.37.0

Method of installation

• Via GHCup

Platform

Mac OS Sonoma 14.4.1

[mpilgrem]

@AliceRixte, thanks for reporting. I've started by seeing if I can build a local version of the package on my machine (os Windows 11), with:

stack unpack streamly
cd streamly-0.10.1
# create stack.yaml with snapshot: nightly-2024-10-02
stack build

Result: builds fine. EDIT: Same result with nightly-2024-06-20 and nightly-09-18.

However, I see from your CI script that you are depending on an unreleased version: streamly-0.11.0.

[mpilgrem]

So, next, I cloned the streamly repository (at commit 5c0a10afa331539e69e8955c781b045ae5238599) and tried to build the package (streamly-0.11.0) with a stack-ghc-9.8.2.yaml:

snapshot: nightly-2024-10-02 # GHC 9.8.2
packages:
- . # streamly
- core # streamly-core

git clone https://github.com/composewell/streamly.git
cd streamly
# create stack-ghc-9.8.2.yaml
stack --stack-yaml stack-ghc-9.8.2.yaml build

Result: builds fine.

I'll try building a package that depends on that package.

[mpilgrem]

So, I created a simple application (stack new streamly-test) with a dependency on those packages, with package.yaml (extract):

dependencies:
- base >= 4.7 && < 5
- streamly
- streamly-core

and stack.yaml:

snapshot: nightly-2024-10-02 # GHC 9.8.2
packages:
- .
- ../streamly # streamly-0.11.0
- ../streamly/core # streamly-core-0.3.0

Result of stack build is: builds fine.

On my Windows 11 machine, I can't reproduce your problem. Perhaps it is macOS-specific. I do not immediately have a macOS machine to hand. However, I can't think of a reason why Stack itself would do anything different on macOS to what it does on Windows 11.

[AliceRixte]

@mpilgrem Thank you for your answer. I'm sorry, I should have been more clear : in the continous integration branch mentioned in composewell/streamly#2844 , we already made clear that :

1. This is a Mac OS specific issue. I also have a Windows 11 computer, and the build is fine for me on that computer too, for the exact same project.
2. This is a problem that appeared after LTS 22.36 for ghc-9.6.6, as the build is fine on this LTS, even on Mac OS.
3. The only resolvers where this bug where reproduced are the ones I mentioned before, but I haven't tried any other nightly resolver after LTS 22.36.

[AliceRixte]

Also notice that the error log mentions a file called "src/Streamly/Internal/FileSystem/Event/Darwin.o", which considering the name I would guess is a Streamly file that deals specifically with Mac OS. @harendra-kumar might be able to confirm this.

[harendra-kumar]

This is a macOS specific issue, other builds are working fine with stack. Here is the failure output on macOS:

2024-09-27 01:56:28.745387: [info] streamly           > [66 of 74] Compiling Streamly.Internal.FileSystem.Event.Darwin
2024-09-27 01:56:30.678949: [info] streamly           > [67 of 74] Compiling Streamly.Internal.FileSystem.Event

...

2024-09-27 01:56:37.923535: [warn] streamly           > <<ghc: 129265195656 bytes, 3196 GCs, 116917740/198910864 avg/max bytes residency (44 samples), 541M in use, 0.006 INIT (0.006 elapsed), 78.238 MUT (130.819 elapsed), 27.692 GC (28.054 elapsed) :ghc>>
2024-09-27 01:56:38.023732: [warn] streamly           > ar: .stack-work/dist/x86_64-osx/ghc-9.8.2/build/src/Streamly/Internal/FileSystem/Event/Darwin.o: No such file or directory
2024-09-27 01:56:38.085362: [debug] Loading project config file stack.yaml
2024-09-27 01:56:38.091484: [debug] Use of Casa server enabled: (CasaRepoPrefix "https://casa.stackage.org/", 1280).
2024-09-27 01:56:38.099974: [debug] (SQL) SELECT COUNT(*) FROM "last_performed" WHERE ("action"=?) AND ("timestamp">=?); [PersistInt64 1,PersistUTCTime 2024-09-26 01:56:38.099783 UTC]
Error: -27 01:56:38.101167: [error] 
Error: [S-7282]
       Stack failed to execute the build plan.
       
       While executing the build plan, Stack encountered the error:
       
       [S-7011]
       While building package streamly-0.11.0 (scroll up to its section to see the error) using:
       /Users/runner/work/streamly/streamly/.stack-work/dist/x86_64-osx/ghc-9.8.2/setup/setup --verbose=1 --builddir=.stack-work/dist/x86_64-osx/ghc-9.8.2 build lib:streamly --ghc-options ""
       Process exited with code: ExitFailure 1 
Error: Command [stack --no-terminal --resolver nightly-2024-09-26 --stack-yaml stack.yaml build -v] failed. Exiting.
Error: Process completed with exit code 1.

The file Darwin.hs includes configuration generated by autoconf. The build works fine with cabal though.

[mpilgrem]

I'll have access to a macOS/AArch64 machine later this month and can investigate further then. Given current nightly, Stack builds with GHC 9.8.2 and, by default, its Cabal-3.10.2.0 boot package. I am pretty sure that Stack will be sending the same commands to the Cabal library on Windows 11 and macOS, so my current hypothesis is this is a macOS-specific Cabal bug.

Do you know what version of Cabal (the library) your Cabal (the tool) (that works) is based on?

If Cabal (the tool) (that works) is using a more recent version of Cabal (the library) than ships with GHC 9.8.2, Stack can be set to use a different version of Cabal to build. You set the required Cabal version as an extra dep and, in the project package's Cabal file, use a custom-setup stanza that specifies how Cabal's Setup.hs should be built, for example:

custom-setup
  setup-depends:
    base,
    Cabal > 3.10.2.0

EDIT: My other suggestion is to see if 'full verbosity' reveals anything. The following will turn on all of Stack's, Cabal's and GHC's verbosity:

stack --verbose build --cabal-verbose --ghc-options=-v

[mpilgrem]

I now have access to a macOS/AArch64 machine. Building does fail. Cabal has the following command (output reformatted, for clarity - all the files are in Cabal's .stack-work/dist/aarch64-osx/ghc-9.8.2/build directory) - something seems to be causing Cabal to try to add a file to an archive that does not exist in the src sub-directory:

EDIT: It turned out that GHC 9.6.6/Cabal-3.10.3.0 had the same command as GHC 9.8.2/Cabal-3.10.2.0, so I looked elsewhere to try to understand why the src sub-directory had different contents.

Running: /usr/bin/ar 
-r 
-s 
.../objs-35394/libHSstreamly-0.10.1-EmCphv71pdaAJx9ykRXuZy.a 
.../Streamly/Internal/Data/IsMap/HashMap.o 
.../Streamly/Internal/Data/Cont.o 
.../Streamly/Internal/Data/Channel.o 
.../Streamly/Internal/Data/Stream/MkType.o 
.../Streamly/Internal/Control/Concurrent.o 
.../Streamly/Internal/Control/ForkLifted.o 
.../Streamly/Internal/Data/Atomics.o 
.../Streamly/Internal/Data/IOFinalizer/Lifted.o
.../Streamly/Internal/Data/Stream/Prelude.o 
.../Streamly/Internal/Data/Unfold/Prelude.o 
.../Streamly/Internal/Data/Fold/Prelude.o 
.../Streamly/Internal/Unicode/Utf8.o 
.../Streamly/Internal/Unicode/Char.o 
.../Streamly/Internal/Network/Socket.o 
.../Streamly/Internal/Network/Inet/TCP.o 
.../Streamly/Data/Stream/MkType.o 
.../Streamly/Data/Stream/Prelude.o 
.../Streamly/Data/Fold/Prelude.o 
.../Streamly/Network/Socket.o 
.../Streamly/Network/Inet/TCP.o 
.../Streamly/Data/Array/Foreign.o 
.../Streamly/Data/Fold/Tee.o 
.../Streamly/Prelude.o
.../Streamly/Internal/Data/SVar.o 
.../Streamly/Internal/Data/Stream/SVar.o 
.../Streamly/Internal/Data/Stream/Serial.o 
.../Streamly/Internal/Data/Stream/Zip.o 
.../Streamly/Internal/Data/Stream/Async.o 
.../Streamly/Internal/Data/Stream/Parallel.o 
.../Streamly/Internal/Data/Stream/Ahead.o 
.../Streamly/Internal/Data/Stream/ZipAsync.o
.../Streamly/Internal/Data/Stream/IsStream.o 
.../Streamly/Internal/FileSystem/Event/Darwin.o         <<< This is interesting! 
.../Streamly/Internal/FileSystem/Event.o 
.../Streamly/Internal/Data/Channel/Types.o 
.../Streamly/Internal/Data/Channel/Dispatcher.o 
.../Streamly/Internal/Data/Channel/Worker.o 
.../Streamly/Internal/Data/Stream/Concurrent/Channel/Type.o 
.../Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.o 
.../Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.o 
.../Streamly/Internal/Data/Stream/Concurrent/Channel/Append.o 
.../Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.o 
.../Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.o 
.../Streamly/Internal/Data/Stream/Concurrent/Channel.o 
.../Streamly/Internal/Data/Stream/Concurrent.o 
.../Streamly/Internal/Data/Stream/Time.o 
.../Streamly/Internal/Data/Stream/Lifted.o 
.../Streamly/Internal/Data/Fold/Async.o 
.../Streamly/Internal/Data/Fold/Concurrent/Channel/Type.o 
.../Streamly/Internal/Data/Fold/Concurrent/Channel.o 
.../Streamly/Internal/Data/Fold/Concurrent.o 
.../Streamly/Internal/Data/Unfold/Exception.o 
.../Streamly/Internal/Data/Unfold/SVar.o 
.../Streamly/Internal/Data/Stream/Common.o 
.../Streamly/Internal/Data/Stream/IsStream/Type.o 
.../Streamly/Internal/Data/Stream/IsStream/Generate.o 
.../Streamly/Internal/Data/Stream/IsStream/Eliminate.o 
.../Streamly/Internal/Data/Stream/IsStream/Transform.o 
.../Streamly/Internal/Data/Stream/IsStream/Expand.o 
.../Streamly/Internal/Data/Stream/IsStream/Reduce.o 
.../Streamly/Internal/Data/Stream/IsStream/Exception.o 
.../Streamly/Internal/Data/Stream/IsStream/Lift.o 
.../Streamly/Internal/Data/Stream/IsStream/Top.o 
.../Streamly/Internal/Data/Stream/IsStream/Combinators.o 
.../Streamly/Internal/Data/Stream/IsStream/Common.o 
.../Streamly/Internal/Data/Stream/IsStream/Enumeration.o 
.../Streamly/Internal/Data/Stream/SVar/Generate.o 
.../Streamly/Internal/Data/Stream/SVar/Eliminate.o 
.../Streamly/Internal/Data/Fold/SVar.o 
.../Streamly/Internal/Data/SVar/Worker.o 
.../Streamly/Internal/Data/SVar/Dispatch.o 
.../Streamly/Internal/Data/SVar/Pull.o 
.../src/Streamly/Internal/FileSystem/Event/Darwin.o.  <<< This argument is the cause!

[mpilgrem]

@harendra-kumar, perhaps this is the cause of the problem? streamly.cabal has:

    if os(darwin)
          include-dirs:  src/Streamly/Internal
          c-sources: src/Streamly/Internal/FileSystem/Event/Darwin.m
          exposed-modules: Streamly.Internal.FileSystem.Event.Darwin

A *.m file is not valid in c-sources. (I think *.m is Objective-C, not C.) This was enforced in Cabal-3.10.2.0 (GHC 9.8.1, 9.8.2), but then tripped up packages that had wrongly-specified Cabal files that Cabal had previously tolerated - see https://github.com/haskell/cabal/blob/master/release-notes/Cabal-3.10.2.1.md.

[mpilgrem]

@AliceRixte, @harendra-kumar if I rename the problematic Darwin.m as Darwin.c and update the Cabal file accordingly, the package can be built with GHC 9.8.2/Cabal-3.10.2.0.