cabal 3.4.0.0 is broken on Windows 7 (use_process_jobs vs. hsc2hs.exe) #7309

bitc · 2021-02-28T23:12:39Z

Describe the bug

cabal 3.4.0.0 uses the use_process_jobs flag for all created processes. See: #6529 and #6536

The problem is that hsc2hs.exe also uses the use_process_jobs flag to create its sub-processes (like when it calls gcc.exe)

See: https://github.com/haskell/hsc2hs/blob/24100ea521596922d3edc8370b3d9f7b845ae4cf/Common.hs#L43

This is a problem because on Windows 7:

A process can be associated only with a single job. [...] If a process is associated with a job, all child processes it creates are associated with that job by default.

which is documented by Microsoft here: https://docs.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-assignprocesstojobobject (Note that on Windows 8 and above there is no problem)

What ends up happening is that inside hsc2exe.exe, inside createProcess, the call to the Win32 function AssignProcessToJobObject fails:

"C:\ghc-8.10.3\bin\hsc2hs.exe" "@C:\Users\bitc\myproject\dist-newstyle\build\x86_64-windows\ghc-8.10.3\myproject-0.6.2.3\build\My\Module\hsc258F.txt"
hsc2hs.exe: C:\ghc-8.10.3\lib\../mingw/bin\gcc.exe: createProcess: permission denied (Permission denied)

Explanation: hsc2hs.exe is running inside a "job" (that was created by cabal.exe). hsc2hs.exe launches gcc.exe and now gcc.exe is also running inside the same "job". Immediately after launching gcc.exe, hsc2hs.exe creates a new "job" and tries to put gcc.exe into this new job (AssignProcessToJobObject) but this call fails because gcc.exe already is part of a "job" (the one it inherited from hsc2hs.exe).

AssignProcessToJobObject is called here: https://github.com/haskell/process/blob/5a0cbd46eca6d30b78726688058b7fd258a2253d/cbits/runProcess.c#L742

Solution

When cabal calls hsc2hs.exe it should set use_process_jobs = False, so that hsc2hs.exe will be able to use this flag. This is valid because hsc2hs.exe doesn't use exec, it properly awaits on its own children.

When cabal calls ghc.exe, gcc.exe, or other tools, it may set use_process_jobs = True (because they themselves don't set the use_process_jobs flag)

Another Solution

It may be possible to utilize CREATE_BREAKAWAY_FROM_JOB somehow (either in cabal or in hsc2hs). See: https://docs.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-assignprocesstojobobject

Alternative Solution

cabal should officially drop support for Windows 7 (a shame)

System information

Windows 7
cabal 3.4.0.0

Thank you

The text was updated successfully, but these errors were encountered:

jneira · 2021-03-01T08:38:13Z

Hi, i've tried to reproduce it in windows 7 with no luck. I've tried to build with cabal-3.4.0.0 v2-build, haskell-language-server (with lot of dependencies, i guess some of them have .hsc files) but just in case i've built the package process-1.6.11.0 (that has a .hsc module for windows) using msys2 and Win32-2.11.1.0 (with several .hsc files) using a dos console.
Maybe it depends of the concrete windows 7 version? some other variable of the environment? the concrete project?

bitc · 2021-03-01T10:00:46Z

Here are fully reproducible instructions:

OS: Windows 7 Professional (running inside VirtualBox 6.1)

Download:

Extract all 4 to C:\bug\ (you will need 7-zip or another program to extract them).

Note: The cabal-install download link I used is a bit weird, it contains only a single file cabal. Rename it to cabal.exe and put it inside a C:\bug\cabal\ directory.

Now you have these directories:

C:\bug\ghc-8.10.3\
C:\bug\cabal\
C:\bug\msys64\
C:\bug\zlib-0.6.2.3\

Now open windows cmd.exe and run:

set PATH=C:\bug\msys64\usr\bin;%PATH%
set CABAL_DIR=C:\bug\cabal_dir
cd C:\bug\zlib-0.6.2.3\
C:\bug\cabal\cabal.exe --version        # Should say:  cabal-install version 3.4.0.0
C:\bug\cabal\cabal.exe v2-update
C:\bug\cabal\cabal.exe v2-build -j1 -v3 --with-ghc=C:\bug\ghc-8.10.3\bin\ghc.exe

Output

found hpc in C:\bug\ghc-8.10.3\bin\hpc.exe
"C:\bug\ghc-8.10.3\bin\hpc.exe" "version"
C:\bug\ghc-8.10.3\bin\hpc.exe is version 0.68
looking for tool hsc2hs near compiler in C:\bug\ghc-8.10.3\bin
candidate locations: ["C:\\bug\\ghc-8.10.3\\bin\\hsc2hs.exe"]
found hsc2hs in C:\bug\ghc-8.10.3\bin\hsc2hs.exe
"C:\bug\ghc-8.10.3\bin\hsc2hs.exe" "--version"
C:\bug\ghc-8.10.3\bin\hsc2hs.exe is version 0.68.7
Searching for HsColour in path.
Cannot find HsColour on the path
Searching for jhc in path.
Cannot find jhc on the path
Searching for ld in path.
Found ld at C:\bug\ghc-8.10.3\lib\../mingw/bin\ld.exe
Environment: [("","ExitCode=00000000"),("ALLUSERSPROFILE","C:\\ProgramData"),("A
PPDATA","C:\\Users\\plum\\AppData\\Roaming"),("CABAL_DIR","C:\\bug\\cabal_dir"),
("COMMONPROGRAMFILES","C:\\Program Files\\Common Files"),("COMMONPROGRAMFILES(X8
6)","C:\\Program Files (x86)\\Common Files"),("COMMONPROGRAMW6432","C:\\Program
Files\\Common Files"),("COMPUTERNAME","PLUM-PC"),("COMSPEC","C:\\Windows\\system
32\\cmd.exe"),("FP_NO_HOST_CHECK","NO"),("HASKELL_DIST_DIR","C:\\bug\\zlib-0.6.2
.3\\dist-newstyle\\build\\x86_64-windows\\ghc-8.10.3\\zlib-0.6.2.3"),("HOMEDRIVE
","C:"),("HOMEPATH","\\Users\\plum"),("LOCALAPPDATA","C:\\Users\\plum\\AppData\\
Local"),("LOGONSERVER","\\\\PLUM-PC"),("MPCONFIG_PRODUCTAPPDATAPATH","C:\\Progra
mData\\Microsoft\\Windows Defender"),("MPCONFIG_PRODUCTCODENAME","AntiSpyware"),
("MPCONFIG_PRODUCTPATH","C:\\Program Files\\Windows Defender"),("MPCONFIG_PRODUC
TUSERAPPDATAPATH","C:\\Users\\plum\\AppData\\Local\\Microsoft\\Windows Defender"
),("MPCONFIG_REPORTINGGUID","9DB624F5-EE2E-44E8-B7E2-1B9AA3F0A9A6"),("NUMBER_OF_
PROCESSORS","2"),("OS","Windows_NT"),("PATH","C:\\bug\\msys64\\usr\\bin;C:\\Wind
ows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\Win
dowsPowerShell\\v1.0\\;C:\\bug\\cabal_dir\\bin"),("PATHEXT",".COM;.EXE;.BAT;.CMD
;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC"),("PROCESSOR_ARCHITECTURE","AMD64"),("PROCES
SOR_IDENTIFIER","Intel64 Family 6 Model 94 Stepping 3, GenuineIntel"),("PROCESSO
R_LEVEL","6"),("PROCESSOR_REVISION","5e03"),("PROGRAMDATA","C:\\ProgramData"),("
PROGRAMFILES","C:\\Program Files"),("PROGRAMFILES(X86)","C:\\Program Files (x86)
"),("PROGRAMW6432","C:\\Program Files"),("PROMPT","$P$G"),("PSMODULEPATH","C:\\W
indows\\system32\\WindowsPowerShell\\v1.0\\Modules\\"),("PUBLIC","C:\\Users\\Pub
lic"),("SESSIONNAME","Console"),("SYSTEMDRIVE","C:"),("SYSTEMROOT","C:\\Windows"
),("TEMP","C:\\Users\\plum\\AppData\\Local\\Temp"),("TMP","C:\\Users\\plum\\AppD
ata\\Local\\Temp"),("USERDOMAIN","plum-PC"),("USERDOMAIN_ROAMINGPROFILE","plum-P
C"),("USERNAME","plum"),("USERPROFILE","C:\\Users\\plum"),("WINDIR","C:\\Windows
"),("WINDOWS_TRACING_FLAGS","3"),("WINDOWS_TRACING_LOGFILE","C:\\BVTBin\\Tests\\
installpackage\\csilogfile.log"),("ZLIB_DATADIR","C:\\bug\\zlib-0.6.2.3\\.")]
"C:\bug\ghc-8.10.3\bin\ghc.exe" "-hide-all-packages" "-c" "C:\Users\plum\AppData
\Local\Temp\ghc2484.c" "-o" "C:\Users\plum\AppData\Local\Temp\ghc2485.o" "-hide-
all-packages"
"C:\bug\ghc-8.10.3\lib\../mingw/bin\ld.exe" "-x" "-r" "C:\Users\plum\AppData\Loc
al\Temp\ghc2485.o" "-o" "C:\Users\plum\AppData\Local\Temp\ghc2570.o"
Searching for pkg-config in path.
Cannot find pkg-config on the path
looking for tool runghc near compiler in C:\bug\ghc-8.10.3\bin
candidate locations: ["C:\\bug\\ghc-8.10.3\\bin\\runghc.exe"]
found runghc in C:\bug\ghc-8.10.3\bin\runghc.exe
"C:\bug\ghc-8.10.3\bin\runghc.exe" "--version"
C:\bug\ghc-8.10.3\bin\runghc.exe is version 8.10.3
Searching for strip in path.
Found strip at C:\bug\ghc-8.10.3\mingw\bin\strip.exe
"C:\bug\ghc-8.10.3\mingw\bin\strip.exe" "--version"
C:\bug\ghc-8.10.3\mingw\bin\strip.exe is version 2.32
Searching for tar in path.
Cannot find tar on the path
Searching for uhc in path.
Cannot find uhc on the path
Source component graph: component lib
Configured component graph:
    component zlib-0.6.2.3-inplace
        include base-4.14.1.0
        include bytestring-0.10.12.0
Linked component graph:
    unit zlib-0.6.2.3-inplace
        include base-4.14.1.0
        include bytestring-0.10.12.0
        Codec.Compression.GZip=zlib-0.6.2.3-inplace:Codec.Compression.GZip,Codec
.Compression.Zlib=zlib-0.6.2.3-inplace:Codec.Compression.Zlib,Codec.Compression.
Zlib.Internal=zlib-0.6.2.3-inplace:Codec.Compression.Zlib.Internal,Codec.Compres
sion.Zlib.Raw=zlib-0.6.2.3-inplace:Codec.Compression.Zlib.Raw
Ready component graph:
    definite zlib-0.6.2.3-inplace
        depends base-4.14.1.0
        depends bytestring-0.10.12.0
Using Cabal-3.4.0.0 compiled by ghc-8.8
Using compiler: ghc-8.10.3
Using install prefix: C:\bug\cabal_dir
Executables installed in: C:\bug\cabal_dir\bin
Libraries installed in:
C:\bug\cabal_dir\x86_64-windows-ghc-8.10.3\zlib-0.6.2.3-inplace
Dynamic Libraries installed in: C:\bug\cabal_dir\x86_64-windows-ghc-8.10.3
Private executables installed in:
C:\bug\cabal_dir\zlib-0.6.2.3-inplace\x86_64-windows-ghc-8.10.3\zlib-0.6.2.3
Data files installed in:
C:\bug\cabal_dir\x86_64-windows-ghc-8.10.3\zlib-0.6.2.3
Documentation installed in:
C:\bug\cabal_dir\doc\x86_64-windows-ghc-8.10.3\zlib-0.6.2.3
Configuration files installed in: C:\bug\cabal_dir\etc
No alex found
Using ar found on system at: C:\bug\ghc-8.10.3\lib\../mingw/bin\ar.exe
No c2hs found
No cpphs found
No doctest found
Using gcc version 9.2.0 found on system at:
C:\bug\ghc-8.10.3\lib\../mingw/bin\gcc.exe
Using ghc version 8.10.3 given by user at: C:\bug\ghc-8.10.3\bin\ghc.exe
Using ghc-pkg version 8.10.3 given by user at:
C:\bug\ghc-8.10.3\bin\ghc-pkg.exe
No ghcjs found
No ghcjs-pkg found
No greencard found
Using haddock version 2.24.0 found on system at:
C:\bug\ghc-8.10.3\bin\haddock.exe
No happy found
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.68 found on system at: C:\bug\ghc-8.10.3\bin\hpc.exe
Using hsc2hs version 0.68.7 found on system at:
C:\bug\ghc-8.10.3\bin\hsc2hs.exe
No hscolour found
No jhc found
Using ld found on system at: C:\bug\ghc-8.10.3\lib\../mingw/bin\ld.exe
No pkg-config found
Using runghc version 8.10.3 found on system at:
C:\bug\ghc-8.10.3\bin\runghc.exe
Using strip version 2.32 found on system at:
C:\bug\ghc-8.10.3\mingw\bin\strip.exe
No tar found
No uhc found
"C:\bug\ghc-8.10.3\lib\../mingw/bin\gcc.exe" "C:\Users\plum\AppData\Local\Temp\g
hc25E0.c" "-o" "C:\Users\plum\AppData\Local\Temp\ghc25E1" "-D__GLASGOW_HASKELL__
=810" "-Dmingw32_BUILD_OS=1" "-Dx86_64_BUILD_ARCH=1" "-Dmingw32_HOST_OS=1" "-Dx8
6_64_HOST_ARCH=1" "-IC:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-
8.10.3\zlib-0.6.2.3\build\autogen" "-IC:\bug\zlib-0.6.2.3\dist-newstyle\build\x8
6_64-windows\ghc-8.10.3\zlib-0.6.2.3\build\cbits" "-I.\cbits" "-I." "-IC:\bug\gh
c-8.10.3\lib\bytestring-0.10.12.0\include" "-IC:\bug\ghc-8.10.3\lib\base-4.14.1.
0\include" "-IC:\bug\ghc-8.10.3\lib\integer-gmp-1.0.3.0\include" "-IC:\bug\ghc-8
.10.3\lib/include" "-LC:\bug\ghc-8.10.3\lib\bytestring-0.10.12.0" "-LC:\bug\ghc-
8.10.3\lib\deepseq-1.4.4.0" "-LC:\bug\ghc-8.10.3\lib\array-0.5.4.0" "-LC:\bug\gh
c-8.10.3\lib\base-4.14.1.0" "-LC:\bug\ghc-8.10.3\lib\integer-gmp-1.0.3.0" "-LC:\
bug\ghc-8.10.3\lib\ghc-prim-0.6.1" "-LC:\bug\ghc-8.10.3\lib/rts"
Using internal setup method with build-type Simple and args:
["build","--verbose=3","--builddir=C:\\bug\\zlib-0.6.2.3\\dist-newstyle\\build\\
x86_64-windows\\ghc-8.10.3\\zlib-0.6.2.3"]
Component build order: library
Environment: [("","ExitCode=00000000"),("ALLUSERSPROFILE","C:\\ProgramData"),("A
PPDATA","C:\\Users\\plum\\AppData\\Roaming"),("CABAL_DIR","C:\\bug\\cabal_dir"),
("COMMONPROGRAMFILES","C:\\Program Files\\Common Files"),("COMMONPROGRAMFILES(X8
6)","C:\\Program Files (x86)\\Common Files"),("COMMONPROGRAMW6432","C:\\Program
Files\\Common Files"),("COMPUTERNAME","PLUM-PC"),("COMSPEC","C:\\Windows\\system
32\\cmd.exe"),("FP_NO_HOST_CHECK","NO"),("HASKELL_DIST_DIR","C:\\bug\\zlib-0.6.2
.3\\dist-newstyle\\build\\x86_64-windows\\ghc-8.10.3\\zlib-0.6.2.3"),("HOMEDRIVE
","C:"),("HOMEPATH","\\Users\\plum"),("LOCALAPPDATA","C:\\Users\\plum\\AppData\\
Local"),("LOGONSERVER","\\\\PLUM-PC"),("MPCONFIG_PRODUCTAPPDATAPATH","C:\\Progra
mData\\Microsoft\\Windows Defender"),("MPCONFIG_PRODUCTCODENAME","AntiSpyware"),
("MPCONFIG_PRODUCTPATH","C:\\Program Files\\Windows Defender"),("MPCONFIG_PRODUC
TUSERAPPDATAPATH","C:\\Users\\plum\\AppData\\Local\\Microsoft\\Windows Defender"
),("MPCONFIG_REPORTINGGUID","9DB624F5-EE2E-44E8-B7E2-1B9AA3F0A9A6"),("NUMBER_OF_
PROCESSORS","2"),("OS","Windows_NT"),("PATH","C:\\bug\\msys64\\usr\\bin;C:\\Wind
ows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\Win
dowsPowerShell\\v1.0\\;C:\\bug\\cabal_dir\\bin"),("PATHEXT",".COM;.EXE;.BAT;.CMD
;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC"),("PROCESSOR_ARCHITECTURE","AMD64"),("PROCES
SOR_IDENTIFIER","Intel64 Family 6 Model 94 Stepping 3, GenuineIntel"),("PROCESSO
R_LEVEL","6"),("PROCESSOR_REVISION","5e03"),("PROGRAMDATA","C:\\ProgramData"),("
PROGRAMFILES","C:\\Program Files"),("PROGRAMFILES(X86)","C:\\Program Files (x86)
"),("PROGRAMW6432","C:\\Program Files"),("PROMPT","$P$G"),("PSMODULEPATH","C:\\W
indows\\system32\\WindowsPowerShell\\v1.0\\Modules\\"),("PUBLIC","C:\\Users\\Pub
lic"),("SESSIONNAME","Console"),("SYSTEMDRIVE","C:"),("SYSTEMROOT","C:\\Windows"
),("TEMP","C:\\Users\\plum\\AppData\\Local\\Temp"),("TMP","C:\\Users\\plum\\AppD
ata\\Local\\Temp"),("USERDOMAIN","plum-PC"),("USERDOMAIN_ROAMINGPROFILE","plum-P
C"),("USERNAME","plum"),("USERPROFILE","C:\\Users\\plum"),("WINDIR","C:\\Windows
"),("WINDOWS_TRACING_FLAGS","3"),("WINDOWS_TRACING_LOGFILE","C:\\BVTBin\\Tests\\
installpackage\\csilogfile.log"),("ZLIB_DATADIR","C:\\bug\\zlib-0.6.2.3\\.")]
"C:\bug\ghc-8.10.3\bin\ghc-pkg.exe" "init" "C:\bug\zlib-0.6.2.3\dist-newstyle\bu
ild\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\package.conf.inplace" "-v2"
GHC package manager version 8.10.3
writing cache C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\
zlib-0.6.2.3\package.conf.inplace\package.cache
creating
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild
creating
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild\autogen
creating
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild\autogen
Preprocessing library for zlib-0.6.2.3..
creating
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild\Codec\Compression\Zlib
creating
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild\Codec\Compression
creating
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild\Codec
creating
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild\Codec\Compression
creating
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild\Codec\Compression\Zlib
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild\Codec\Compression\Zlib\hsc270D.txt
contents: <<<
--cc=C:\\bug\\ghc-8.10.3\\lib\\../mingw/bin\\gcc.exe
--ld=C:\\bug\\ghc-8.10.3\\lib\\../mingw/bin\\gcc.exe
--cflag=-D__GLASGOW_HASKELL__=810
--cflag=-Dmingw32_BUILD_OS=1
--cflag=-Dx86_64_BUILD_ARCH=1
--cflag=-Dmingw32_HOST_OS=1
--cflag=-Dx86_64_HOST_ARCH=1
--cflag=-Icbits
--cflag=-IC:\\bug\\zlib-0.6.2.3\\dist-newstyle\\build\\x86_64-windows\\ghc-8.10.
3\\zlib-0.6.2.3\\build\\cbits
--cflag=-IC:\\bug\\zlib-0.6.2.3\\dist-newstyle\\build\\x86_64-windows\\ghc-8.10.
3\\zlib-0.6.2.3\\build\\autogen
--cflag=-IC:\\bug\\zlib-0.6.2.3\\dist-newstyle\\build\\x86_64-windows\\ghc-8.10.
3\\zlib-0.6.2.3\\build\\global-autogen
--cflag=-include
--cflag=C:\\bug\\zlib-0.6.2.3\\dist-newstyle\\build\\x86_64-windows\\ghc-8.10.3\
\zlib-0.6.2.3\\build\\autogen\\cabal_macros.h
--cflag=-IC:\\bug\\ghc-8.10.3\\lib\\bytestring-0.10.12.0\\include
--cflag=-IC:\\bug\\ghc-8.10.3\\lib\\base-4.14.1.0\\include
--cflag=-IC:\\bug\\ghc-8.10.3\\lib\\integer-gmp-1.0.3.0\\include
--cflag=-IC:\\bug\\ghc-8.10.3\\lib/include
--lflag=-LC:\\bug\\ghc-8.10.3\\lib\\bytestring-0.10.12.0
--lflag=-LC:\\bug\\ghc-8.10.3\\lib\\deepseq-1.4.4.0
--lflag=-LC:\\bug\\ghc-8.10.3\\lib\\array-0.5.4.0
--lflag=-LC:\\bug\\ghc-8.10.3\\lib\\base-4.14.1.0
--lflag=-lwsock32
--lflag=-luser32
--lflag=-lshell32
--lflag=-lmsvcrt
--lflag=-lmingw32
--lflag=-lmingwex
--lflag=-lshlwapi
--lflag=-LC:\\bug\\ghc-8.10.3\\lib\\integer-gmp-1.0.3.0
--lflag=-LC:\\bug\\ghc-8.10.3\\lib\\ghc-prim-0.6.1
--lflag=-luser32
--lflag=-lmingw32
--lflag=-lmingwex
--lflag=-LC:\\bug\\ghc-8.10.3\\lib/rts
--lflag=-lm
--lflag=-lwsock32
--lflag=-lgdi32
--lflag=-lwinmm
--lflag=-ldbghelp
--lflag=-lpsapi
-o
C:\\bug\\zlib-0.6.2.3\\dist-newstyle\\build\\x86_64-windows\\ghc-8.10.3\\zlib-0.
6.2.3\\build\\Codec\\Compression\\Zlib\\Stream.hs
Codec\\Compression\\Zlib\\Stream.hsc
>>>
C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86_64-windows\ghc-8.10.3\zlib-0.6.2.3\b
uild\Codec\Compression\Zlib\hsc270D.txt
Environment: [("","ExitCode=00000000"),("ALLUSERSPROFILE","C:\\ProgramData"),("A
PPDATA","C:\\Users\\plum\\AppData\\Roaming"),("CABAL_DIR","C:\\bug\\cabal_dir"),
("COMMONPROGRAMFILES","C:\\Program Files\\Common Files"),("COMMONPROGRAMFILES(X8
6)","C:\\Program Files (x86)\\Common Files"),("COMMONPROGRAMW6432","C:\\Program
Files\\Common Files"),("COMPUTERNAME","PLUM-PC"),("COMSPEC","C:\\Windows\\system
32\\cmd.exe"),("FP_NO_HOST_CHECK","NO"),("HASKELL_DIST_DIR","C:\\bug\\zlib-0.6.2
.3\\dist-newstyle\\build\\x86_64-windows\\ghc-8.10.3\\zlib-0.6.2.3"),("HOMEDRIVE
","C:"),("HOMEPATH","\\Users\\plum"),("LOCALAPPDATA","C:\\Users\\plum\\AppData\\
Local"),("LOGONSERVER","\\\\PLUM-PC"),("MPCONFIG_PRODUCTAPPDATAPATH","C:\\Progra
mData\\Microsoft\\Windows Defender"),("MPCONFIG_PRODUCTCODENAME","AntiSpyware"),
("MPCONFIG_PRODUCTPATH","C:\\Program Files\\Windows Defender"),("MPCONFIG_PRODUC
TUSERAPPDATAPATH","C:\\Users\\plum\\AppData\\Local\\Microsoft\\Windows Defender"
),("MPCONFIG_REPORTINGGUID","9DB624F5-EE2E-44E8-B7E2-1B9AA3F0A9A6"),("NUMBER_OF_
PROCESSORS","2"),("OS","Windows_NT"),("PATH","C:\\bug\\msys64\\usr\\bin;C:\\Wind
ows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\Win
dowsPowerShell\\v1.0\\;C:\\bug\\cabal_dir\\bin"),("PATHEXT",".COM;.EXE;.BAT;.CMD
;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC"),("PROCESSOR_ARCHITECTURE","AMD64"),("PROCES
SOR_IDENTIFIER","Intel64 Family 6 Model 94 Stepping 3, GenuineIntel"),("PROCESSO
R_LEVEL","6"),("PROCESSOR_REVISION","5e03"),("PROGRAMDATA","C:\\ProgramData"),("
PROGRAMFILES","C:\\Program Files"),("PROGRAMFILES(X86)","C:\\Program Files (x86)
"),("PROGRAMW6432","C:\\Program Files"),("PROMPT","$P$G"),("PSMODULEPATH","C:\\W
indows\\system32\\WindowsPowerShell\\v1.0\\Modules\\"),("PUBLIC","C:\\Users\\Pub
lic"),("SESSIONNAME","Console"),("SYSTEMDRIVE","C:"),("SYSTEMROOT","C:\\Windows"
),("TEMP","C:\\Users\\plum\\AppData\\Local\\Temp"),("TMP","C:\\Users\\plum\\AppD
ata\\Local\\Temp"),("USERDOMAIN","plum-PC"),("USERDOMAIN_ROAMINGPROFILE","plum-P
C"),("USERNAME","plum"),("USERPROFILE","C:\\Users\\plum"),("WINDIR","C:\\Windows
"),("WINDOWS_TRACING_FLAGS","3"),("WINDOWS_TRACING_LOGFILE","C:\\BVTBin\\Tests\\
installpackage\\csilogfile.log"),("ZLIB_DATADIR","C:\\bug\\zlib-0.6.2.3\\.")]
"C:\bug\ghc-8.10.3\bin\hsc2hs.exe" "@C:\bug\zlib-0.6.2.3\dist-newstyle\build\x86
_64-windows\ghc-8.10.3\zlib-0.6.2.3\build\Codec\Compression\Zlib\hsc270D.txt"
hsc2hs.exe: C:\bug\ghc-8.10.3\lib\../mingw/bin\gcc.exe: createProcess: permissio
n denied (Permission denied)

Note that it completely hangs at the end, and if you open Windows task manager you will see that there is a "zombie" gcc.exe process.

jneira · 2021-03-01T13:40:36Z

Many thanks for the detailed instructions, in my end i've tested and succesfullt built zlib:

Windows 7 Professional Service Pack 1 (same as yours afaics)
cabal-3.4.0.0 official final release from https://downloads.haskell.org/cabal/cabal-install-3.4.0.0/
with no msys in PATH using the dos console, where gcc doesn't find any executable
- not needed afaik to build packages that dont use a configure step like zlib (or Win32)
- for the package needing configure (process in my case), i am using msys2-x86_64-20180531 and the build is succesful within the msys2 console
  - just in case gcc version is realgcc.exe (Rev1, Built by MSYS2 project) 7.2.0
ghc-8.10.4 and ghc-8.10.3 (both downloaded from https://downloads.haskell.org)
building zlib i've got with cabal get zlib-0.6.2.3
without changing CABAL_DIR
relevant entries config in %APPDATA%\cabal\config

store-dir: D:\csd
overwrite-policy: always
install-method: copy
installdir: D:\bin

bitc · 2021-03-02T11:01:50Z

cabal-3.4.0.0 official final release from https://downloads.haskell.org/cabal/cabal-install-3.4.0.0/

I tried this binary and it does not exhibit the bug. It is because it was built with "process" package < 1.6.9

See:

cabal/Cabal/Distribution/Compat/Process.hs

Line 33 in be18bb7

#if MIN_VERSION_process(1,6,9)

This is confirmed when I run strings cabal.exe | grep process-:

process-1.6.5.0

The rc4 binary I linked above: https://oleg.fi/cabal-install-3.4.0.0-rc4/cabal-install-3.4.0.0-x86_64-windows.zip shows the bug.

Using the same "strings" trick I can see that it indeed uses process-1.6.9.0

Also, I compiled cabal-install myself and it has the bug (same as the rc4 binary).

Turns out MSYS is not needed, here are simpler instructions to reproduce:

Make sure to use the exact cabal-install rc4 binary given here, or build cabal-install yourself using process >= 1.6.9

Download:

Extract all 3 to C:\bug\ (you will need 7-zip or another program to extract them).

Note: The cabal-install download link I used is a bit weird, it contains only a single file cabal. Rename it to cabal.exe and put it inside a C:\bug\cabal\ directory.

Now you have these directories:

C:\bug\ghc-8.10.3\
C:\bug\cabal\
C:\bug\zlib-0.6.2.3\

Now open windows cmd.exe and run:

set CABAL_DIR=C:\bug\cabal_dir
cd C:\bug\zlib-0.6.2.3\
C:\bug\cabal\cabal.exe --version        # Should say:  cabal-install version 3.4.0.0
C:\bug\cabal\cabal.exe v2-update
C:\bug\cabal\cabal.exe v2-build -j1 -v3 --with-ghc=C:\bug\ghc-8.10.3\bin\ghc.exe

The output is the same as I pasted above.

Thank you

jneira · 2021-03-02T13:26:27Z

Note: The cabal-install download link I used is a bit weird, it contains only a single file cabal. Rename it to cabal.exe and put it inside a C:\bug\cabal\ directory.

That error in packaging for rc's ended to be in the final release too 😟, but it was corrected

I tried this binary and it does not exhibit the bug. It is because it was built with "process" package < 1.6.9

Well, fortunately the bug is gone in the definitive release without having any issue reported at that point 😅
I think we could close this one, changing the title to
cabal 3.4.0.0-rc4 is broken on Windows 7 (use_process_jobs vs. hsc2hs.exe) to make clear the error is present in the release candidate

bitc · 2021-03-02T13:37:32Z

Well, then the real bug is that the official cabal 3.4.0.0 final release windows binary was compiled with an old version of "process".

This means that windows users aren't getting the fix that was added in #6536, which means that they "may see sporatic build failures" (quote from that pull request).

I don't agree that this has anything to do with rc4/final. Like I mentioned previously, if you compile cabal-install 3.4.0.0 (final) yourself, the default is that it will use a recent version of "process", and then you will still encounter the bug (on Windows 7).

jneira · 2021-03-02T13:45:13Z

the default is that it will use a recent version of "process", and then you will still encounter the bug (on Windows 7).

oh, sorry, i thought the newer one doesnt trigger the error but it is the other way around

bgamari · 2021-03-02T16:50:24Z

Great diagnosis, @bitc. Indeed I hadn't appreciated that a process can be associated with only a single job. This significantly complicates matters for us, sadly. I really don't know how we can reliably track potential forking under this restriction. We could try to set use_process_jobs only in cases where we know that the subprocess itself forks but this is a somewhat tricky invariant to maintain.

What I do not see in the MSDN documentation is what behavior Windows 8 and later will exhibit when one attempts to associate a process with multiple jobs. Given that jobs cannot be "nested" I suppose the last assignment wins? If so, this sounds like it also carries the potential for brokenness (e.g. if a process crashes before its children have finished; that being said, we already cut some necessary corners in the face of crashing since we cannot robustly determine which process within a job is the true child of the fork).

bgamari · 2021-03-02T16:54:37Z

@Mistuke, perhaps you know the answer to the question I pose in the second paragraph of my previous post?

bgamari · 2021-03-02T17:32:33Z

Actually, never mind. To its credit, the documentation does specify the behavior for later versions of Windows:

A process can be associated with more than one job in a hierarchy of nested jobs. ... By default, all child processes are associated with the immediate job and every job in the parent job chain. To create a child process that is not part of the same job chain, call the CreateProcess function with the CREATE_BREAKAWAY_FROM_JOB flag. The child process breaks away from every job in the job chain unless a job in the chain does not allow breakaway. In this case, the child process does not break away from that job or any job above it in the job chain. For more information, see Nested Jobs.

bgamari · 2021-03-02T17:56:26Z

I have opened ghc#19473 to track this on the GHC side, as this issue affects far more than just Cabal. Ultimately I'm not entirely sure what to do here. Windows 7 was EoL'd in July 2020, so it doesn't seem terribly high priority to retain support for it. Moreover, as far as I can tell it is essentially impossible to robustly support fork without nested job objects (supported only in Windows 8 and later).

As far as I can tell there are no good options for dealing with WIndows 7:

Solution

When cabal calls hsc2hs.exe it should set use_process_jobs = False, so that hsc2hs.exe will be able to use this flag. This is valid because hsc2hs.exe doesn't use exec, it properly awaits on its own children.

When cabal calls ghc.exe, gcc.exe, or other tools, it may set use_process_jobs = True (because they themselves don't set the use_process_jobs flag)

Unfortunately this isn't true. ghc does use use_process_jobs when it calls gcc and ld. Moreover, I don't think we want to get into the business of tracking which tools might exec in Cabal (or anywhere, for that matter). The only reason that use_process_jobs = True isn't the default is concern that there was a performance overhead (which appears not to be the case).

Another Solution

It may be possible to utilize CREATE_BREAKAWAY_FROM_JOB somehow (either in cabal or in hsc2hs). See: https://docs.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-assignprocesstojobobject

Yes, this is possible. However, the result is potentially fragile in the case of crashing. I would only want to do this where nested jobs truly aren't supported (which we would need to determine at runtime since it is common to ship around binaries on Windows).

Mistuke · 2021-03-02T18:30:02Z

It may be possible to utilize CREATE_BREAKAWAY_FROM_JOB somehow (either in cabal or in hsc2hs). See: https://docs.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-assignprocesstojobobject

Yes, this is possible. However, the result is potentially fragile in the case of crashing. I would only want to do this where nested jobs truly aren't supported (which we would need to determine at runtime since it is common to ship around binaries on Windows).

Yes nested jobs on older Windows versions are indeed somewhat broken. I will have to think about this somewhat, but I think it's safe for process to specify CREATE_BREAKAWAY_FROM_JOB by default.

The problem we're trying to catch is for programs that use spawn and exec. All Haskell program are well behaved in this regard and do CreateProcess. So what that means is waiting on any Haskell program is already correct. Breaking away from the job means that nested Haskell process chains just see the child Haskell program. Which is fine. I don't believe we have any chain where a POSIX program calls back into a Haskell program.

cabal will wait on hsc2hs which waits on all posix programs. Cabal will wait on GHC which can wait on hsc2hs or all the posix children. So breaking away from the job is OK,

One thing to check is if resource cleanup still works as expected.... i.e. if cabal sends a sigkill do the children of GHC still die? (it should since we specificy kill child on parent exit I believe.. but needs checking).

Mistuke · 2021-03-02T19:30:40Z

So you can think of any Haskell program as a synchronization barrier, each Haskell program can safely break away from the parent without affecting correctness (we don't use jobs for resource restrictions (aka cgroups)). Remember that the issue is programs that use exec because the exact semantics is impossible under the NT process model.

So I'm convinced it'll work for execution correctness. But resource leaks should be checked out.

jneira · 2021-06-15T08:19:26Z

Maybe it worths translating here the last comment of the ghc issue:

Ultimately the priority of this ticket comes down to whether we want to support Windows 7 or not. As I mention above, the status quo works fine with "modern" Windows releases. Strictly speaking Windows 7 was EoL'd in July 2020, so we would be well-justified in officially dropping it.

jneira · 2021-11-22T11:50:39Z

While cabal-install continue being buildable with a ghc version which support process < 1.6.9.0 (ghc < 8.8.4) a workaround could be build cabal-install from source with cabal install cabal-install --constraint="process < 1.6.9.0" -w ghc-8.6.5

I've just tried with cabal head and it seems to work

fgaz added the platform: windows label Mar 1, 2021

jneira added blocked: ghc type: bug labels Jun 15, 2021

jneira added the can-workaround There is a (maybe partial) workaround for the issue or missing feature label Nov 22, 2021

jneira mentioned this issue Dec 1, 2021

Fix cabal on Windows 7 #7844

Merged

3 tasks

jneira linked a pull request Dec 1, 2021 that will close this issue

Fix cabal on Windows 7 #7844

Merged

3 tasks

mergify bot closed this as completed in #7844 Dec 3, 2021

Mikolaj mentioned this issue Jan 20, 2022

Cabal run leaves children running when killed #7914

Closed

robx mentioned this issue Jul 22, 2022

Interrupting cabal-3.8.0.0.20220526 and master with ctrl+c makes it hang on windows #8208

Open

thomasjm mentioned this issue Aug 19, 2022

stuck when compiling haskell-fswatch/hfsnotify#104

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cabal 3.4.0.0 is broken on Windows 7 (use_process_jobs vs. hsc2hs.exe) #7309

cabal 3.4.0.0 is broken on Windows 7 (use_process_jobs vs. hsc2hs.exe) #7309

bitc commented Feb 28, 2021

jneira commented Mar 1, 2021

bitc commented Mar 1, 2021 •

edited

Loading

jneira commented Mar 1, 2021 •

edited

Loading

bitc commented Mar 2, 2021

jneira commented Mar 2, 2021

bitc commented Mar 2, 2021

jneira commented Mar 2, 2021

bgamari commented Mar 2, 2021

bgamari commented Mar 2, 2021

bgamari commented Mar 2, 2021

bgamari commented Mar 2, 2021

Mistuke commented Mar 2, 2021

Mistuke commented Mar 2, 2021

jneira commented Jun 15, 2021

jneira commented Nov 22, 2021 •

edited

Loading

cabal 3.4.0.0 is broken on Windows 7 (use_process_jobs vs. hsc2hs.exe) #7309

cabal 3.4.0.0 is broken on Windows 7 (use_process_jobs vs. hsc2hs.exe) #7309

Comments

bitc commented Feb 28, 2021

jneira commented Mar 1, 2021

bitc commented Mar 1, 2021 • edited Loading

jneira commented Mar 1, 2021 • edited Loading

bitc commented Mar 2, 2021

jneira commented Mar 2, 2021

bitc commented Mar 2, 2021

jneira commented Mar 2, 2021

bgamari commented Mar 2, 2021

bgamari commented Mar 2, 2021

bgamari commented Mar 2, 2021

bgamari commented Mar 2, 2021

Mistuke commented Mar 2, 2021

Mistuke commented Mar 2, 2021

jneira commented Jun 15, 2021

jneira commented Nov 22, 2021 • edited Loading

bitc commented Mar 1, 2021 •

edited

Loading

jneira commented Mar 1, 2021 •

edited

Loading

jneira commented Nov 22, 2021 •

edited

Loading