Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support coming GHC 9.8.1 (GHC 9.8.1-rc1 is available) #16

Closed
mpilgrem opened this issue Sep 30, 2023 · 12 comments · Fixed by #17
Closed

Support coming GHC 9.8.1 (GHC 9.8.1-rc1 is available) #16

mpilgrem opened this issue Sep 30, 2023 · 12 comments · Fixed by #17

Comments

@mpilgrem
Copy link
Member

@hsyl20, may I once again call on your assistance with the above? Based on my experiments with GHC 9.8.1-rc1, it looks to me that the *.hi format has changed again, after GHC 9.6.3.

@hsyl20
Copy link
Collaborator

hsyl20 commented Oct 2, 2023

I'll have a look at this week.

@hsyl20
Copy link
Collaborator

hsyl20 commented Oct 3, 2023

That was a minor change. Fixed in #17

@mpilgrem mpilgrem reopened this Nov 2, 2023
@mpilgrem
Copy link
Member Author

mpilgrem commented Nov 2, 2023

@hsyl20, I am trying to build pantry with the official release of GHC 9.8.1 and a version of Stack that depends on hi-file-parser-0.1.5.0, and I'm getting these messages:

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry\Internal\Stackage.hi Decoding failure:
         Invalid dictionary index: 4168842332

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry.hi Decoding failure:
         Invalid dictionary index: 2574345280

I am wondering: could there have been a further change to the *.hi file format that is somehow evading the CI of this library (which is reporting 'green' for GHC 9.8.1).

@mpilgrem
Copy link
Member Author

mpilgrem commented Nov 2, 2023

@hsyl20, I built Stack against a version of hi-file-parser-0.1.5.0 with enableDebug = True, and got this sort of debub output (extracts only, gaps represented by ...). Hope that helps:

Magic: 1face64
Version: 9081
Ways: ""
Dict ptr: 6248
Dictionary size: 478
Dictionary: ["pantry-0.9.2-JkhjEPHMYEj8HkTp1e9qOb", ...
...
Module: "Prelude"
Usage type: 0
Unit type: 0
Dict index: 5
Dict index: 250
Module: "Pantry.Types"
Usage type: 1
Dict index: 2
Home module: "Pantry.Hackage"
Dict index: 0
Dict index: 331
Dict index: 331
Dict index: 332
Dict index: 333
Dict index: 331
Dict index: 4168842332

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry\Internal\Stackage.hi Decoding failure:
         Invalid dictionary index: 4168842332
...
Magic: 1face64
Version: 9081
Ways: ""
Dict ptr: 88102
Dictionary size: 1412
Dictionary: ["pantry-0.9.2-JkhjEPHMYEj8HkTp1e9qOb", ...
...
Home module: "Pantry.HTTP"
Dict index: 0
Dict index: 396
Usage type: 1
Dict index: 5
Home module: "Pantry.Hackage"
Dict index: 0
Dict index: 397
Dict index: 398
Dict index: 399
Dict index: 400
Dict index: 401
Dict index: 402
Dict index: 403
Dict index: 404
Dict index: 405
Dict index: 406
Dict index: 407
Dict index: 408
Dict index: 409
Dict index: 2574345280

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry.hi Decoding failure:
         Invalid dictionary index: 2574345280

@mpilgrem
Copy link
Member Author

mpilgrem commented Nov 2, 2023

I suppose it could be possible that GHC 9.8.1 can produce corrupt *.hi files. The corresponding parts of the debug output when building pantry with GHC 9.6.3 are:

...
Usage type: 1
Dict index: 2
Home module: "Pantry.Hackage"
Dict index: 0
Dict index: 334
Dict index: 334
Dict index: 335
Dict index: 336
Dict index: 337
Dict index: 338
...
Usage type: 1
Dict index: 5
Home module: "Pantry.Hackage"
Dict index: 0
Dict index: 400
Dict index: 401
Dict index: 402
Dict index: 403
Dict index: 404
Dict index: 405
Dict index: 406
Dict index: 406
Dict index: 407
Dict index: 408
Dict index: 409
Dict index: 410
Dict index: 411
Dict index: 412
Dict index: 413
Dict index: 414
Dict index: 415
Dict index: 416

@mpilgrem
Copy link
Member Author

mpilgrem commented Nov 3, 2023

I think I can rule out the (unlikely) possibility that it is GHC 9.8.1 that is producing malformed *.hi files. I used this test (built against package ghc-9.8.1):

module Main (main) where

import Control.Monad ( forM )
import GHC.Iface.Binary ( CheckHiWay (..), TraceBinIFace (..), readBinIface )
import GHC.Platform
         ( Platform (..), ArchOS (..), Arch (..), OS (..), genericPlatform )
import GHC.Platform.Profile ( Profile (..) )
import GHC.Types.Name.Cache ( NameCache (..), initNameCache )
import GHC.Unit.Module.ModIface ( ModIface, ModIface_ (..) )
import GHC.Unit.Types ( GenModule (..) )
import GHC.Utils.Outputable ( SDoc, printSDocLn, traceSDocContext )
import GHC.Utils.Ppr ( Mode (..) )
import System.IO ( stdout )

main :: IO ()
main = do
  let fps =
        [ "D:\\Users\\mike\\Code\\GitHub\\commercialhaskell\\pantry\\.stack-work\\dist\\2b10cf2c\\build\\Pantry\\Internal\\Stackage.hi"
        , "D:\\Users\\mike\\Code\\GitHub\\commercialhaskell\\pantry\\.stack-work\\dist\\2b10cf2c\\build\\Pantry.hi"
        ]
  nc <- initNameCache 'A' []
  modIfaces <- forM fps (readBinIface' nc)
  let ns = map (moduleName . mi_module) modIfaces
  print ns

readBinIface' :: NameCache -> FilePath -> IO ModIface
readBinIface' nc fp =
  readBinIface profile nc CheckHiWay (TraceBinIFace traceBinIFace) fp

profile :: Profile
profile = Profile platform mempty

platform :: Platform
platform = genericPlatform
  { platformArchOS = archOS }

archOS :: ArchOS
archOS = ArchOS ArchX86_64 OSMinGW32

traceBinIFace :: SDoc -> IO ()
traceBinIFace = printSDocLn traceSDocContext (PageMode True) stdout

The output of the exectuable is (that is, it does not throw an error):

❯ stack exec -- hiTest-exe

Warning: Stack has not been tested with GHC versions 9.8 and above, and using 9.8.1, this may fail.
Magic: Wanted 33214052,
       got    33214052
Version: Wanted 9081,
         got    9081
Way: Wanted ,
     got
Magic: Wanted 33214052,
       got    33214052
Version: Wanted 9081,
         got    9081
Way: Wanted ,
     got
[ModuleName "Pantry.Internal.Stackage",ModuleName "Pantry"]

@mpilgrem
Copy link
Member Author

mpilgrem commented Nov 4, 2023

Modifying my test to provide more information, the first instance gives me this:

UsageHomeModule ModuleName "Pantry.Hackage"
[ (HackageTarballResult,e1b26ec099457bfa71b89d24afac1cae)
, (HackageTarballResult,3130b9a905b4346529568b9a8f954565)
, (forceUpdateHackageIndex,50666e761ee4bf6215ce10c4cec342ce)
, (getHackageTarball,b53840208942067b1c88be8c9c67906f)
, (htrFreshPackageInfo,c4b8e8e27696394290fddd7c3dbe2e76)
, (htrPackage,fb6b3fab39801649ab2cb84cb8b22078)
]

and the elements of the directory are:

331: , "HackageTarballResult"
332: , "forceUpdateHackageIndex"
333: , "getHackageTarball"
334: , "htrFreshPackageInfo"
335: , "htrPackage"

However, it seems 334 is being read as 331 and 335 is being read as 4168842332???

@mpilgrem
Copy link
Member Author

mpilgrem commented Nov 4, 2023

To rule out that I had introduced a bug since #17 was merged, I built Stack against hi-file-parser commit 4797274, and experienced the same problem.

@mpilgrem
Copy link
Member Author

mpilgrem commented Nov 4, 2023

The first and second problem with added debug information from hi-file-parser:

Usage type: 1
Dict index: 2
Home module: "Pantry.Hackage"
Dict index: 0
Fingerprint:  "8048068b3e99c1d8832be56c1865429f"
Ignoring:  ()
Fingerprint:  "73d1081b486cee23657eb1c107af3f4d"
Ignoring:  ()
List (with LEB128): 6
Dict index: 331
Fingerprint:  "e1b26ec099457bfa71b89d24afac1cae"
Ignoring:  ()
Dict index: 331
Fingerprint:  "3130b9a905b4346529568b9a8f954565"
Ignoring:  ()
Dict index: 332
Fingerprint:  "50666e761ee4bf6215ce10c4cec342ce"
Ignoring:  ()
Dict index: 333
Fingerprint:  "b53840208942067b1c88be8c9c67906f"
Ignoring:  ()
Dict index: 331
Fingerprint:  "14ec4b8e8e276963942"
Ignoring:  ()
Dict index: 4168842332

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry\Internal\Stackage.hi Decoding failure:
         Invalid dictionary index: 4168842332
Usage type: 1
Dict index: 5
Home module: "Pantry.Hackage"
Dict index: 0
Fingerprint:  "8048068b3e99c1d8832be56c1865429f"
Ignoring:  ()
Fingerprint:  "73d1081b486cee23657eb1c107af3f4d"
Ignoring:  ()
List (with LEB128): 18
Dict index: 397
Fingerprint:  "fa9c7e16290dad7418a8252b836a5e75"
Ignoring:  ()
Dict index: 398
Fingerprint:  "e6879306c0e103c727c3120b40a50720"
Ignoring:  ()
Dict index: 399
Fingerprint:  "7c8e1a69a972a661437c87821b9415bd"
Ignoring:  ()
Dict index: 400
Fingerprint:  "b0de1a49a54c7e324c788f164f5dc731"
Ignoring:  ()
Dict index: 401
Fingerprint:  "f937d72150e494f225c0e3c8f2e945c2"
Ignoring:  ()
Dict index: 402
Fingerprint:  "8f82da48126adb635270b2f99deca1e3"
Ignoring:  ()
Dict index: 403
Fingerprint:  "c611f3918cfdefa95ce07f5776b7de3d"
Ignoring:  ()
Dict index: 403
Fingerprint:  "6f3b376f3d495477589fe4ee05c816e8"
Ignoring:  ()
Dict index: 404
Fingerprint:  "7c67e6444f5f388efe9d5fa6aa80b60d"
Ignoring:  ()
Dict index: 405
Fingerprint:  "167f6ae9113d1783150c807aee25cd8e"
Ignoring:  ()
Dict index: 406
Fingerprint:  "42818d6ba53c3e7f3a366d97e496343c"
Ignoring:  ()
Dict index: 407
Fingerprint:  "c4b08bfa666623eeda4df0a69c5f124e"
Ignoring:  ()
Dict index: 408
Fingerprint:  "b53840208942067b1c88be8c9c67906f"
Ignoring:  ()
Dict index: 409
Fingerprint:  "af5100401a891d5e209f87f4b46bbbf3"
Ignoring:  ()
Dict index: 410
Fingerprint:  "be37359c6d8be7093ffcf6567db3a46b"
Ignoring:  ()
Dict index: 411
Fingerprint:  "e40dacb6b6364edff90a529aa3a929cf"
Ignoring:  ()
Dict index: 412
Fingerprint:  "19dfb6b3fab39801649"
Ignoring:  ()
Dict index: 2574345280

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry.hi Decoding failure:
         Invalid dictionary index: 2574345280

It seems that something is going wrong with the penultimate Fingerprint in the lists.

@mpilgrem
Copy link
Member Author

mpilgrem commented Nov 4, 2023

@hsyl20, I think I have it! The instance of Binary for NameSpace has changed; it is no longer guaranteed to be a single byte:

GHC 9.6.3:

instance Binary NameSpace where
    put_ bh VarName =
            putByte bh 0
    put_ bh DataName =
            putByte bh 1
    put_ bh TvName =
            putByte bh 2
    put_ bh TcClsName =
            putByte bh 3
    get bh = do
            h <- getByte bh
            case h of
              0 -> return VarName
              1 -> return DataName
              2 -> return TvName
              _ -> return TcClsName

GHC 9.8.1:

instance Binary NameSpace where
    put_ bh VarName =
            putByte bh 0
    put_ bh DataName =
            putByte bh 1
    put_ bh TvName =
            putByte bh 2
    put_ bh TcClsName =
            putByte bh 3
    put_ bh (FldName parent) = do
            putByte bh 4
            put_ bh parent
    get bh = do
            h <- getByte bh
            case h of
              0 -> return VarName
              1 -> return DataName
              2 -> return TvName
              3 -> return TcClsName
              _ -> do
                parent <- get bh
                return $ FldName { fldParent = parent }

I think that is throwing off this line in hi-file-parser, which assumes getWord8:

void (getList (getTuple (getWord8 *> getFastString) getFP)) -- usg_entities

@mpilgrem
Copy link
Member Author

mpilgrem commented Nov 4, 2023

I am going to attempt a fix ...

mpilgrem added a commit that referenced this issue Nov 4, 2023
Fix #16 Reflect change in `instance Binary NameSpace`
@hsyl20
Copy link
Collaborator

hsyl20 commented Nov 6, 2023

Good work @mpilgrem! Sorry for the delay, I was on vacation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants