Binary module interface

.gitignore

@@ -29,3 +29,4 @@ tags
 /etc/scripts/stack-scripts.cabal
 .hspec-failures
 better-cache/
+/subs/*/*.cabal

ChangeLog.md

@@ -60,9 +60,13 @@ Major changes:
       is uniquely identified by a commit id and an Hadrian "flavour" (Hadrian is
       the newer GHC build system), hence `compiler` can be set to use a GHC
       built from source with `ghc-git-COMMIT-FLAVOUR`
+* `stack.yaml` now supports a `configure-options`, which are passed directly to
+  the `configure` step in the Cabal build process. See
+  [#1438](https://github.com/commercialhaskell/stack/issues/1438)
 
 * Remove support for building GHCJS itself. Future releases of Stack
   may remove GHCJS support entirely.
+* Support for lock files for pinning exact project dependency versions
 
 Behavior changes:
 * `stack.yaml` now supports `snapshot`: a synonym for `resolver`. See [#4256](https://github.com/commercialhaskell/stack/issues/4256)
@@ -116,6 +120,17 @@ Behavior changes:
   means that Stack will no longer have to force reconfigures as often. See
   [#3554](https://github.com/commercialhaskell/stack/issues/3554).
 
+* When building a package, Stack takes a lock on the dist directory in
+  use to avoid multiple runs of Stack from trampling each others'
+  files. See
+  [#2730](https://github.com/commercialhaskell/stack/issues/2730).
+
+* Stack will check occassionally if there is a new version available and prompt
+  the user to upgrade. This will not incur any additional network traffic, as
+  it will piggy-back on the existing Hackage index updates. You can set
+  `recommend-stack-upgrade: false` to bypass this. See
+  [#1681](https://github.com/commercialhaskell/stack/issues/1681).
+
 Other enhancements:
 
 * Defer loading up of files for local packages. This allows us to get

doc/lock_files.md

@@ -18,11 +18,12 @@ set of input files. There are a few problems with making this work:
 
 To address this, we follow the (fairly standard) approach of having a
 _lock file_. The goal of the lock file is to cache completed
-information about all packages and snapshot files so that:
+locations of project, snapshot packages and snapshots themselves so that:
 
 * These files can be stored in source control
 * Users on other machines can reuse these lock files and get identical
-  build plans
+  build plans given that the used local packages and local snapshots are
+  the same on those machines
 * Rerunning `stack build` in the future is deterministic in the build
   plan, not depending on mutable state in the world like Hackage
   revisions
@@ -31,8 +32,6 @@ information about all packages and snapshot files so that:
       to perform the build. However, by deterministic, we mean it
       either performs the same build or fails, never accidentally
       doing something different.
-* Stack can quickly determine the build plan in the common case of no
-  changes to `stack.yaml` or snapshot files
 
 This document explains the contents of a lock file, how they are used,
 and how they are created and updated.
@@ -42,11 +41,7 @@ and how they are created and updated.
 Relevant to this discussion, the `stack.yaml` file specifies:
 
 * Resolver (the parent snapshot)
-* Compiler override
 * `extra-deps`
-* Flags
-* GHC options
-* Hidden packages
 
 The resolver can either specify a compiler version or another snapshot
 file. This snapshot file can contain the same information referenced
@@ -55,12 +50,7 @@ above for a `stack.yaml`, with the following differences:
 * The `extra-deps` are called `packages`
 * Drop packages can be included
 
-Some of this information is, by its nature, complete. For example, the
-"flags" field cannot be influenced by anything outside of the file
-itself.
-
-On the other hand, some information in these files can be
-incomplete. Consider:
+Some information in these files can be incomplete. Consider:
 
 ```yaml
 resolver: lts-13.9
@@ -128,24 +118,16 @@ parsing of the additional files in the common case of no changes.
 
 The lock file contains the following information:
 
-* The full snapshot definition information, including completed
-  package locations for both `extra-deps` and packages in
+* Completed package locations for both `extra-deps` and packages in
   snapshot files
     * **NOTE** This only applies to _immutable_ packages. Mutable
       packages are not included in the lock file.
 * Completed information for the snapshot locations
-* A hash of the `stack.yaml` file
-* The snapshot hash, to bypass the need to recalculate this on each
-  run of Stack
 
 It looks like the following:
 
 ```yaml
 # Lock file, some message about the file being auto-generated
-stack-yaml:
-  sha256: XXXX
-  size: XXXX # in bytes
-
 snapshots:
   # Starts with the snapshot specified in stack.yaml,
   # then continues with the snapshot specified in each
@@ -163,33 +145,22 @@ snapshots:
       url: https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/13/9.yaml
       sha256: 83de9017d911cf7795f19353dba4d04bd24cd40622b7567ff61fc3f7223aa3ea
 
-compiler: ghc-X.Y.Z
-
 packages:
-  acme-missiles:
-    location:
-      # QUESTION: any reason we need to specify which snapshot file it came from? I don't think so...
-      original: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz
-      completed:
-        size: 1442
-        url: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz
-        cabal-file:
-          size: 613
-          sha256: 2ba66a092a32593880a87fb00f3213762d7bca65a687d45965778deb8694c5d1
-        name: acme-missiles
-        version: '0.3'
-        sha256: e563d8b524017a06b32768c4db8eff1f822f3fb22a90320b7e414402647b735b
-        pantry-tree:
-          size: 226
-          sha256: 614bc0cca76937507ea0a5ccc17a504c997ce458d7f2f9e43b15a10c8eaeb033
-    flags: ...
-    hidden: true/false
-    ghc-options: [...]
+- original: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz
+  completed:
+    size: 1442
+    url: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz
+    cabal-file:
+      size: 613
+      sha256: 2ba66a092a32593880a87fb00f3213762d7bca65a687d45965778deb8694c5d1
+    name: acme-missiles
+    version: '0.3'
+    sha256: e563d8b524017a06b32768c4db8eff1f822f3fb22a90320b7e414402647b735b
+    pantry-tree:
+      size: 226
+      sha256: 614bc0cca76937507ea0a5ccc17a504c997ce458d7f2f9e43b15a10c8eaeb033
 ```
 
-**NOTE** The `original` fields may seem superfluous at first. See the
-update procedure below for an explanation.
-
 ## Creation
 
 Whenever a `stack.yaml` file is loaded, Stack checks for a lock file
@@ -206,36 +177,9 @@ If the lock file does not exist, it will be created by:
 * Completing all missing information
 * Writing out the new `stack.yaml.lock` file
 
-## Dirtiness checking
-
-If the `stack.yaml.lock` file exists, its last modification time is
-compared against the last modification time of the `stack.yaml` file
-and any local snapshot files. If any of those files is more recent
-than the `stack.yaml` file, and the file hashes in the lock file
-do not match the files on the filesystem, then the update procedure is
-triggered. Otherwise, the `stack.yaml.lock` file can be used as the
-definition of the snapshot.
-
 ## Update procedure
 
-The simplest possible implementation is: ignore the lock file entirely
-and create a new one followign the creation steps above. There's a
-significant downside to this, however: it may cause a larger delta in
-the lock file than intended, by causing more packages to be
-updates. For example, many packages from Hackage may have their
-Hackage revision information updated unnecessarily.
-
-The more complicated update procedure is described below. **QUESTION**
-Do we want to go the easy way at first and later implement the more
-complicated update procedure?
-
-1. Create a map from original package location to completed package
-   location in the lock file
-2. Load up each snapshot file
-3. For each incomplete package location:
-    * Lookup the value in the map created in (1)
-    * If present: use that completed information
-    * Otherwise: complete the information using the same completion
-      procedure from Pantry as in "creation"
-
-This should minimize the number of changes to packages incurred.
+When loading a Stack project all completed package or snapshot locations
+(even when they were completed using information from a lock file) get
+collected to form a new lock file in memory and compare against the one
+on disk, writing if there are any differences.

doc/yaml_configuration.md

@@ -585,6 +585,24 @@ on an options change, but this behavior can be changed back with the following:
 rebuild-ghc-options: true
 ```
 
+### configure-options
+
+Options which are passed to the configure step of the Cabal build process.
+These can either be set by package name, or using the `$everything`,
+`$targets`, and `$locals` special keys. These special keys have the same
+meaning as in `ghc-options`.
+
+```yaml
+configure-options:
+  $everything:
+  - --with-gcc
+  - /some/path
+  my-package:
+  - --another-flag
+```
+
+(Since 2.0)
+
 ### ghc-variant
 
 (Since 0.1.5)
@@ -1072,4 +1090,12 @@ Build output when disabled:
 ...
 ```
 
+### recommend-stack-upgrade
+
+When Stack notices that a new version of Stack is available, should it notify the user?
+
+```yaml
+recommend-stack-upgrade: true
+```
+
 Since 2.0

package.yaml

@@ -1,5 +1,5 @@
 name: stack
-version: '1.10.0'
+version: '2.0.0'
 synopsis: The Haskell Tool Stack
 description: ! 'Please see the README.md for usage information, and
   the wiki on Github for more details.  Also, note that
@@ -123,6 +123,7 @@ dependencies:
 - yaml
 - zip-archive
 - zlib
+- binary
 when:
 - condition: os(windows)
   then:
@@ -183,6 +184,8 @@ library:
   - Stack.IDE
   - Stack.Init
   - Stack.Ls
+  - Stack.Lock
+  - Stack.ModuleInterface
   - Stack.New
   - Stack.Nix
   - Stack.Options.BenchParser
@@ -302,6 +305,7 @@ tests:
     dependencies:
     - QuickCheck
     - hspec
+    - raw-strings-qq
     - stack
     - smallcheck
 flags:

snapshot-lts-12.yaml

@@ -10,6 +10,8 @@ packages:
 - yaml-0.10.4.0@rev:0 #for hpack-0.31
 - persistent-2.9.2@rev:0
 - persistent-sqlite-2.9.3@rev:0
+- github: snoyberg/filelock
+  commit: 97e83ecc133cd60a99df8e1fa5a3c2739ad007dc
 
 drop-packages:
 # See https://github.com/commercialhaskell/stack/pull/4712

snapshot-nightly.yaml

@@ -4,6 +4,8 @@ name: snapshot-for-building-stack-with-ghc-8.6.2
 packages:
 - persistent-2.9.2@rev:0
 - persistent-sqlite-2.9.3@rev:0
+- github: snoyberg/filelock
+  commit: 97e83ecc133cd60a99df8e1fa5a3c2739ad007dc
 
 drop-packages:
 # See https://github.com/commercialhaskell/stack/pull/4712

snapshot.yaml

@@ -19,6 +19,8 @@ packages:
 - process-1.6.3.0@sha256:fc77cfe75a9653b8c54ae455ead8c06cb8adc4d7a340984d84d8ca880b579919,2370 #because of https://github.com/haskell/process/pull/101
 - persistent-2.9.2@rev:0
 - persistent-sqlite-2.9.3@rev:0
+- github: snoyberg/filelock
+  commit: 97e83ecc133cd60a99df8e1fa5a3c2739ad007dc
 
 drop-packages:
 # See https://github.com/commercialhaskell/stack/pull/4712

src/Stack/Build.hs

@@ -244,16 +244,18 @@ loadPackage
   :: (HasBuildConfig env, HasSourceMap env)
   => PackageLocationImmutable
   -> Map FlagName Bool
-  -> [Text]
+  -> [Text] -- ^ GHC options
+  -> [Text] -- ^ Cabal configure options
   -> RIO env Package
-loadPackage loc flags ghcOptions = do
+loadPackage loc flags ghcOptions cabalConfigOpts = do
   compiler <- view actualCompilerVersionL
   platform <- view platformL
   let pkgConfig = PackageConfig
         { packageConfigEnableTests = False
         , packageConfigEnableBenchmarks = False
         , packageConfigFlags = flags
         , packageConfigGhcOptions = ghcOptions
+        , packageConfigCabalConfigOpts = cabalConfigOpts
         , packageConfigCompilerVersion = compiler
         , packageConfigPlatform = platform
         }

src/Stack/Build/ConstructPlan.hs

@@ -118,7 +118,7 @@ type M = RWST -- TODO replace with more efficient WS stack on top of StackT
 
 data Ctx = Ctx
     { baseConfigOpts :: !BaseConfigOpts
-    , loadPackage    :: !(PackageLocationImmutable -> Map FlagName Bool -> [Text] -> M Package)
+    , loadPackage    :: !(PackageLocationImmutable -> Map FlagName Bool -> [Text] -> [Text] -> M Package)
     , combinedMap    :: !CombinedMap
     , ctxEnvConfig   :: !EnvConfig
     , callStack      :: ![PackageName]
@@ -171,7 +171,7 @@ instance HasEnvConfig Ctx where
 constructPlan :: forall env. HasEnvConfig env
               => BaseConfigOpts
               -> [DumpPackage] -- ^ locally registered
-              -> (PackageLocationImmutable -> Map FlagName Bool -> [Text] -> RIO EnvConfig Package) -- ^ load upstream package
+              -> (PackageLocationImmutable -> Map FlagName Bool -> [Text] -> [Text] -> RIO EnvConfig Package) -- ^ load upstream package
               -> SourceMap
               -> InstalledMap
               -> Bool
@@ -231,8 +231,8 @@ constructPlan baseConfigOpts0 localDumpPkgs loadPackage0 sourceMap installedMap
 
     mkCtx econfig globalCabalVersion sources mcur pathEnvVar' = Ctx
         { baseConfigOpts = baseConfigOpts0
-        , loadPackage = \x y z -> runRIO econfig $
-            applyForceCustomBuild globalCabalVersion <$> loadPackage0 x y z
+        , loadPackage = \w x y z -> runRIO econfig $
+            applyForceCustomBuild globalCabalVersion <$> loadPackage0 w x y z
         , combinedMap = combineMap sources installedMap
         , ctxEnvConfig = econfig
         , callStack = []
@@ -426,7 +426,7 @@ addDep name = do
                             -- names. This code does not feel right.
                             let version = installedVersion installed
                                 askPkgLoc = liftRIO $ do
-                                  mrev <- getLatestHackageRevision name version
+                                  mrev <- getLatestHackageRevision YesRequireHackageIndex name version
                                   case mrev of
                                     Nothing -> do
                                       -- this could happen for GHC boot libraries missing from Hackage
@@ -469,7 +469,7 @@ tellExecutablesUpstream name retrievePkgLoc loc flags = do
     when (name `Set.member` wanted ctx) $ do
         mPkgLoc <- retrievePkgLoc
         forM_ mPkgLoc $ \pkgLoc -> do
-            p <- loadPackage ctx pkgLoc flags []
+            p <- loadPackage ctx pkgLoc flags [] []
             tellExecutablesPackage loc p
 
 tellExecutablesPackage :: InstallLocation -> Package -> M ()
@@ -505,7 +505,7 @@ installPackage name ps minstalled = do
     case ps of
         PSRemote pkgLoc _version _fromSnaphot cp -> do
             planDebug $ "installPackage: Doing all-in-one build for upstream package " ++ show name
-            package <- loadPackage ctx pkgLoc (cpFlags cp) (cpGhcOptions cp)
+            package <- loadPackage ctx pkgLoc (cpFlags cp) (cpGhcOptions cp) (cpCabalConfigOpts cp)
             resolveDepsAndInstall True (cpHaddocks cp) ps package minstalled
         PSFilePath lp -> do
             case lpTestBench lp of
@@ -662,7 +662,7 @@ addPackageDeps package = do
         eres <- addDep depname
         let getLatestApplicableVersionAndRev :: M (Maybe (Version, BlobKey))
             getLatestApplicableVersionAndRev = do
-              vsAndRevs <- runRIO ctx $ getHackagePackageVersions UsePreferredVersions depname
+              vsAndRevs <- runRIO ctx $ getHackagePackageVersions YesRequireHackageIndex UsePreferredVersions depname
               pure $ do
                 lappVer <- latestApplicableVersion range $ Map.keysSet vsAndRevs
                 revs <- Map.lookup lappVer vsAndRevs
@@ -1057,7 +1057,7 @@ pprintExceptions exceptions stackYaml stackRoot parentMap wanted' prunedGlobalDe
     pprintExtra (name, (version, BlobKey cabalHash cabalSize)) =
       let cfInfo = CFIHash cabalHash (Just cabalSize)
           packageIdRev = PackageIdentifierRevision name version cfInfo
-       in "- " <+> fromString (T.unpack (utf8BuilderToText (RIO.display packageIdRev)))
+       in fromString ("- " ++ T.unpack (utf8BuilderToText (RIO.display packageIdRev)))
 
     allNotInBuildPlan = Set.fromList $ concatMap toNotInBuildPlan exceptions'
     toNotInBuildPlan (DependencyPlanFailures _ pDeps) =