Consider ghcup for windows #70

hasufell · 2021-08-30T09:58:37Z

Right now, when I define

jobs:
  build-cabal:

    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macOS-latest, windows-latest]
        ghc: ['8.10.7', '9.0.1']
        cabal: ['3.4.0.0']

This will fail on windows only, because 8.10.7 isn't in choco yet. It's quite confusing, because different installers with different release schedules are used under the hood.

I merged https://gitlab.haskell.org/haskell/ghcup-hs/-/merge_requests/159 which would allow the installation procedure on windows to be:

curl.exe -O https://www.haskell.org/ghcup/sh/bootstrap-haskell.ps1
.\bootstrap-haskell.ps1 -InstallDir 'C:\' -InBash -Minimal

Leaving you with an msys2 installation (in C:\ghcup\msys64) and the ghcup binary (in C:\ghcup\bin), nothing more.

Installations of msys2 packages would be carried out via:

C:\ghcup\msys64\usr\bin\bash.exe -lc 'pacman --noconfirm -S package'

The text was updated successfully, but these errors were encountered:

hasufell · 2021-08-30T10:40:30Z

Here's an example workflow:

hazelweakly · 2021-09-01T16:31:25Z

I think the windows images from GitHub still use chocolatey under the hood, so I would still default to that (for fastest setup time in the cloud), but I'm looking forward to being able to fall back to ghcup on Windows and possibly even switch to using it first if the action images start using ghcup for windows too.

PRs are more than welcome for the functionality, but otherwise I'll try to get around to it when time permits over the next few weeks :)

(Edit: in case it wasn't clear, I deeply appreciate the windows support in ghcup. It's something I've been looking forward to for a long time and I'm super stoked about it. Thanks for all your work in the Haskell tooling space!)

hasufell · 2021-09-01T17:19:40Z

for fastest setup time in the cloud

Good point. I timed that and from my tests ghcup is twice as fast: https://github.com/hasufell/shortbytestring/runs/3487178253?check_suite_focus=true#step:3:1

Takes 2m5s and installs ghcup, cabal, ghc and msys2

      Set-ExecutionPolicy Bypass -Scope Process -Force;[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072;Invoke-Command -ScriptBlock ([ScriptBlock]::Create((Invoke-WebRequest https://www.haskell.org/ghcup/sh/bootstrap-haskell.ps1 -UseBasicParsing))) -ArgumentList $false

Takes 3m59s and installs cabal, ghc and msys2

    choco install -y haskell-dev

Does the haskell action do anything different that would make chocolatey faster?

jneira · 2021-09-03T09:48:01Z

I think the windows images from GitHub still use chocolatey under the hood,

confirmed: https://github.com/actions/virtual-environments/blob/main/images/win/scripts/Installers/Install-Haskell.ps1

linux/macos uses the package manager (ppa/hvr) for ghc and ghcup for cabal, it seems the ghc installed by ghcup is bigger than the other one (???):

# We are using apt-get to install ghc, not ghcup,
# because ghc installed through ghcup takes up too much disk space (2GB versus 1GB through apt-get)

https://github.com/actions/virtual-environments/blob/ec9ac010f114131adacb246cf87c8127c19aa0d9/images/linux/scripts/installers/haskell.sh#L33

I think @maerwald will find that interesting 😉

hasufell · 2021-09-03T09:52:52Z

linux/macos uses the package manager (ppa/hvr) for ghc and ghcup for cabal, it seems the ghc installed by ghcup is bigger than the other one (???):

I'm not sure how that's relevant to the windows discussion?

(The reason is that the apt repository has debug symbols and documentation separate. The GHC build system doesn't allow that split. It has to be done manually.)

jneira · 2021-09-03T09:57:32Z

I'm not sure how that's relevant to the windows discussion?

Well if linux/macos already would use ghcup for ghc (even as fallback) it would be more appealing for gha unify all the three os's through ghcup

jneira · 2021-09-03T09:59:29Z

In fact upstreaming the actual haskell actions ghc/cabal/stack setup logic to gha vms would be great by itself

hasufell · 2021-09-03T10:07:46Z

Well if linux/macos already would use ghcup for ghc (even as fallback) it would be more appealing for gha unify all the three os's through ghcup

I don't understand. I thought haskell/actions already does use ghcup for linux and mac. I have no idea about actions/virtual-environments and currently don't have plans to use it. This ticket is about this action only.

jneira · 2021-09-03T10:17:14Z

hmm actions/virtual-environments are the actual vms where gha (so haskell/actions) are being run. Afaik (maybe @jared-w could correct me if i am wrong) those vms have already installed the haskell toolchain and haskell/actions does a fallback over such preinstalled toolchain (as it can be obsolete for some time)

but yeah we can ignore upstream 🙂

hasufell · 2021-09-03T10:21:20Z

hmm actions/virtual-environments are the actual vms where gha (so haskell/actions) are being run.

Ah. I wasn't aware. Well, then my suggestion is to simply tell ghcup where the pre-installed msys2 is. Then the installation will take only a couple of seconds.

jneira · 2021-09-03T10:30:38Z

as it can be obsolete for some time

Also it does not install all ghc versions, only 2 latest Haskell Major.Minor versions so haskell/action installs the other ones if requested. But the build would be faster if you use the preinstalled ones and they will be more up to date if they use ghcup :-)

hasufell · 2021-09-03T14:09:28Z

So we can use the pre-installed msys2 via:

Set-ExecutionPolicy Bypass -Scope Process -Force;[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072;Invoke-Command -ScriptBlock ([ScriptBlock]::Create((Invoke-WebRequest https://www.haskell.org/ghcup/sh/bootstrap-haskell.ps1 -UseBasicParsing))) -ArgumentList $false, $true, $true, $false, $false, $false, $false, C:\, "", C:\msys64, C:\cabal

This takes 15 seconds to install ghcup: https://github.com/hasufell/shortbytestring/runs/3506845639?check_suite_focus=true#step:3:1

hasufell · 2021-09-03T15:14:47Z

actions/runner-images#4016

newhoggy · 2021-09-05T07:24:32Z

Also worth considering that Chocolatey is fairly unreliable. For example, I just witnessed this failure on Windows:

Run haskell/actions/setup@v1
Preparing to setup a Haskell environment
Installing ghc version 8.10.4
  Attempting to install ghc 8.10.4 using chocolatey
  ::stop-commands::SetupHaskellStopCommands
  C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe choco install ghc --version 8.10.4 -m --no-progress -r
  Installing the following packages:
  ghc
  By installing you accept licenses for the packages.
  [NuGet] Could not connect to the feed specified at 'https://community.chocolatey.org/api/v2/'. Please verify that the package source (located in the Package Manager Settings) is valid and ensure your network connectivity.
  ghc not installed. An error occurred during installation:
   Unable to resolve dependency 'cabal (≥ 3.2.0.0)'.
  ghc package files install completed. Performing other installation steps.
  The install of ghc was NOT successful.
  ghc not installed. An error occurred during installation:
   Unable to resolve dependency 'cabal (≥ 3.2.0.0)'.
  
  Chocolatey installed 0/1 packages. 1 packages failed.
   See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
  
  Failures
   - ghc (exited 1) - ghc not installed. An error occurred during installation:
   Unable to resolve dependency 'cabal (≥ 3.2.0.0)'.
  C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe choco install ghc --version 8.10.4 -m --no-progress -r --pre
  Installing the following packages:
  ghc
  By installing you accept licenses for the packages.
  ghc not installed. An error occurred during installation:
   The operation has timed out
  ghc package files install completed. Performing other installation steps.
  The install of ghc was NOT successful.
  ghc not installed. An error occurred during installation:
   The operation has timed out
  
  Chocolatey installed 0/1 packages. 1 packages failed.
   See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
  
  Failures
   - ghc (exited 1) - ghc not installed. An error occurred during installation:
   The operation has timed out
  ::SetupHaskellStopCommands::
Error: All install methods for ghc 8.10.4 failed

Reliability trumps installation speed in CI.

jneira · 2021-09-05T10:15:27Z

yeah in hls we experienced that issue with choco lately 😕

hasufell · 2021-09-10T17:53:01Z

So this has been merged upstream at actions/virtual-environments:

ghcup is used on ubuntu
profiling libs and share folder are removed to reduce disk size
ghcup is intalled on windows per default (but chocolatey still installs GHCs)

newhoggy · 2021-09-25T07:37:46Z

@hasufell How do I use the ghcup installed on Windows by default?

hasufell · 2021-09-25T07:48:30Z

@hasufell How do I use the ghcup installed on Windows by default?

Hmm, maybe the PATH update didn't make it: actions/runner-images@a9ac553

GHCup on windows is installed usually at C:\\ghcup\\bin.

Furthermore, the script should have set GHCUP_MSYS2=C:\\msys64 and CABAL_DIR=C:\\cabal. If that didn't happen, then there's a problem with the image. Maybe it doesn't run refreshenv?

newhoggy · 2021-09-25T08:01:34Z

I don't see those environment variables either.

hasufell · 2021-09-25T08:04:21Z

I don't see those environment variables either.

Can you try without haskell actions?

newhoggy · 2021-09-25T08:04:28Z

I do see the binary installed at the location you specified:

newhoggy · 2021-09-25T08:47:08Z

It's the same without Haskell actions.

newhoggy · 2021-09-25T08:47:42Z

Is there a way I can run refreshenv myself from within Github Actions?

hasufell · 2021-09-25T10:06:25Z

I'll have to investigate

newhoggy · 2021-11-08T10:23:31Z

@hasufell Thanks for the PR!

hasufell · 2022-01-21T16:28:35Z

ghcup is now pre-installed on windows machines as well and should be in PATH etc.

newhoggy · 2022-01-29T00:05:00Z

This is amazing. Thanks!

andreasabel · 2022-03-10T10:16:08Z

@jared-w : Do you think the time is ripe for switching to ghcup under Windows?
I am trying to bump my CIs to GHC 9.2.2 and they fail on Windows because chocolatey does not have GHC 9.2.1 yet (but ghcup has it!).

jneira · 2022-03-10T10:25:41Z

Some times choco had the new version before ghcup 😉 . I usually open an issue in Mistuke/GhcChoco#18 and usually @Mistuke adds it very quickly when it is not.

But afaics 9.2.1 was added to chocolatey: https://community.chocolatey.org/packages/ghc
9.2.2 is not and i guess it will not, as it is broken for windows

hasufell · 2022-03-10T10:34:43Z

9.2.2 is not and i guess it will not, as it is broken for windows

Yes: https://gitlab.haskell.org/ghc/ghc/-/issues/21196

It seems you can get it to work when msys2 is in PATH though, but you might hit other problems.

Mistuke · 2022-03-10T10:41:23Z

Some times choco had the new version before ghcup 😉 . I usually open an issue in Mistuke/GhcChoco#18 and usually @Mistuke adds it very quickly when it is not.

Yes it's coming. I'm just adding a workaround for the broken packaging. I tend to not like to push broken compilers :)

hasufell · 2022-03-10T10:56:08Z

I tend to not like to push broken compilers :)

Many GHC releases are defunct on one platform or another. E.g. M1 is broken on most GHC versions. I don't do downstream patching.

Mistuke · 2022-03-10T11:01:49Z

I tend to not like to push broken compilers :)

Many GHC releases are defunct on one platform or another. E.g. M1 is broken on most GHC versions. I don't do downstream patching.

That's fine :) I have routinely worked around broken packaging and will continue to do so.

hasufell · 2022-03-10T11:59:07Z

I tend to not like to push broken compilers :)

Many GHC releases are defunct on one platform or another. E.g. M1 is broken on most GHC versions. I don't do downstream patching.

That's fine :) I have routinely worked around broken packaging and will continue to do so.

Hopefully that won't be required soon anymore, since GHC should have some dogfood tests as you suggested in the ticket.

I've also offered my help of testing releases manually before they are announced via ghcup, since this is probably the 3rd time I've discovered such issues close after release. But GHC devs decided against human testing.

jneira · 2022-03-10T13:11:36Z

I've also offered my help of testing releases manually before they are announced via ghcup, since this is probably the 3rd time I've discovered such issues close after release. But GHC devs decided against human testing.

yeah, given the complexity of ghc and how frequently minor version breaks things it should have rcs for minor versions, even with a smaller window time

hasufell · 2022-03-10T19:48:02Z

@andreasabel

Btw... ghcup is pre-installed on all github environments, so this should work on all platforms without haskell/actions:

jobs:
  build:

    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macOS-latest, windows-latest]
        ghc: ['8.0.2', '8.2.2', '8.4.4', '8.6.5', '8.8.4', '8.10.7', '9.0.2', '9.2.1']
        cabal: ['3.6.2.0']

    steps:
    - name: Install ghc/cabal
      run: |
        ghcup install ghc --set ${{ matrix.ghc }}
        ghcup install cabal ${{ matrix.cabal }}
      shell: bash

And in fact, that's what I do in filepath: https://github.com/haskell/filepath/blob/4638d2a6a375064c270fd0a3e249d922859b4f9b/.github/workflows/test.yaml#L58

So users who want ghcup release channels, can already do so and don't need this action... hence closing.

jneira · 2022-03-10T21:49:42Z

I still think use ghcup in this action would make sense. You can also use chocolatey or hvr/ppa outside this action but it allows workflows abstract away from the concrete channels and the future changes in them.
It will make handling across all os's more uniform.

andreasabel · 2022-03-15T09:17:21Z

@hasufell

And in fact, that's what I do in filepath: haskell/filepath@4638d2a/.github/workflows/test.yaml#L58

This CI does not demonstrate caching of dependencies. How do communicate the cabal store directory in a stable way without setup-haskell?
This would be a typical caching phase:

    - uses: actions/cache@v2
      with:
        path: ${{ steps.setup-haskell.outputs.cabal-store }}
        # The file `plan.json` contains the build information.
        key: ${{ runner.os }}-${{ env.GHC_VER }}-${{ env.CABAL_VER }}-${{ hashFiles('**/plan.json') }}

newhoggy · 2022-03-15T09:32:23Z

I feel like cabal should have a command to print where the cabal store is since it already knows where it is.

andreasabel · 2022-03-15T09:36:40Z

I feel like cabal should have a command to print where the cabal store is since it already knows where it is.

Is there already a cabal issue for this? If not please report, linking here!

jneira · 2022-03-15T09:46:11Z

It seems to me that the incoming cabal status could give that info: haskell/cabal#7500

An usual workaround is tell cabal use a specific store dir, changing the global config using cabal init and use that specific path in the cache step. Not ideal but you have to do it for windows anyways to workaround the max path length issue.

https://github.com/haskell/cabal/blob/906a8bcabc307c6c607cf1b9826a6b329e3ceb7f/.github/workflows/windows.yml#L37

However chocolatey already sets the store dir in C:\sr and ghcup in X:\cabal\store and the setup action gives it to you.

hasufell · 2022-03-15T10:07:08Z

@hasufell

And in fact, that's what I do in filepath: haskell/filepath@4638d2a/.github/workflows/test.yaml#L58

This CI does not demonstrate caching of dependencies. How do communicate the cabal store directory in a stable way without setup-haskell? This would be a typical caching phase:
    - uses: actions/cache@v2
      with:
        path: ${{ steps.setup-haskell.outputs.cabal-store }}
        # The file `plan.json` contains the build information.
        key: ${{ runner.os }}-${{ env.GHC_VER }}-${{ env.CABAL_VER }}-${{ hashFiles('**/plan.json') }}

I would actually suggest using https://github.com/haskell-works/cabal-cache ...it allows syncing to a directory (instead of S3). Then you would tell actions/cache@v2 to just cache that directory. That may also save some cache space, because it only caches the needed libraries from the resolution.

I can code up an example if I have a bit more time.

andreasabel · 2023-02-18T07:02:46Z

Work on this is going on in a fork: main...wz1000:actions:wip/783

andreasabel · 2023-03-23T22:37:28Z

WIP PR:

use ghcup on windows (clone of #192) #206

This was referenced Sep 6, 2021

Improve haskell ghcup actions/runner-images#4016

Closed

Install ghcup on windows actions/runner-images#4044

Merged

raehik mentioned this issue Sep 17, 2021

Add Windows CI camfort/camfort#140

Closed

hasufell mentioned this issue Oct 12, 2021

Expose ghcup binary to PATH on windows actions/runner-images#4264

Merged

hasufell closed this as completed Mar 10, 2022

jneira mentioned this issue Mar 11, 2022

Make ghc installation consistent across all os's (use ghcup as fallback for windows) #97

Closed

hasufell mentioned this issue Nov 20, 2022

Overhaul metadata merging and add 'ghcup config add-release-channel URI' haskell/ghcup-hs#647

Closed

andreasabel added platform: windows re: ghcup Concerning setup using ghcup labels Dec 22, 2022

andreasabel reopened this Feb 18, 2023

andreasabel mentioned this issue Feb 18, 2023

ghcup not found on windows-2022 version 20230214.1 #191

Closed

andreasabel mentioned this issue Feb 27, 2023

use ghcup from PATH on windows #192

Closed

andreasabel added the help wanted Extra attention is needed label Mar 23, 2023

andreasabel mentioned this issue May 1, 2023

use ghcup on windows (clone of #192) #206

Merged

5 tasks

andreasabel closed this as completed in #206 May 1, 2023

andreasabel added this to the 2.3.8 milestone May 1, 2023

Consider ghcup for windows #70

Consider ghcup for windows #70

Comments

hasufell commented Aug 30, 2021 • edited Loading

hasufell commented Aug 30, 2021

hazelweakly commented Sep 1, 2021 • edited Loading

hasufell commented Sep 1, 2021

jneira commented Sep 3, 2021 • edited Loading

hasufell commented Sep 3, 2021

jneira commented Sep 3, 2021

jneira commented Sep 3, 2021

hasufell commented Sep 3, 2021

jneira commented Sep 3, 2021

hasufell commented Sep 3, 2021

jneira commented Sep 3, 2021

hasufell commented Sep 3, 2021

hasufell commented Sep 3, 2021

newhoggy commented Sep 5, 2021

jneira commented Sep 5, 2021

hasufell commented Sep 10, 2021

newhoggy commented Sep 25, 2021

hasufell commented Sep 25, 2021 • edited Loading

newhoggy commented Sep 25, 2021

hasufell commented Sep 25, 2021

newhoggy commented Sep 25, 2021

newhoggy commented Sep 25, 2021

newhoggy commented Sep 25, 2021

hasufell commented Sep 25, 2021

newhoggy commented Nov 8, 2021 • edited Loading

hasufell commented Jan 21, 2022

newhoggy commented Jan 29, 2022

andreasabel commented Mar 10, 2022

jneira commented Mar 10, 2022 • edited Loading

hasufell commented Mar 10, 2022 • edited Loading

Mistuke commented Mar 10, 2022

hasufell commented Mar 10, 2022

Mistuke commented Mar 10, 2022

hasufell commented Mar 10, 2022

jneira commented Mar 10, 2022

hasufell commented Mar 10, 2022 • edited Loading

jneira commented Mar 10, 2022

andreasabel commented Mar 15, 2022

newhoggy commented Mar 15, 2022

andreasabel commented Mar 15, 2022

jneira commented Mar 15, 2022 • edited Loading

hasufell commented Mar 15, 2022

andreasabel commented Feb 18, 2023

andreasabel commented Mar 23, 2023

hasufell commented Aug 30, 2021 •

edited

Loading

hazelweakly commented Sep 1, 2021 •

edited

Loading

jneira commented Sep 3, 2021 •

edited

Loading

hasufell commented Sep 25, 2021 •

edited

Loading

newhoggy commented Nov 8, 2021 •

edited

Loading

jneira commented Mar 10, 2022 •

edited

Loading

hasufell commented Mar 10, 2022 •

edited

Loading

hasufell commented Mar 10, 2022 •

edited

Loading

jneira commented Mar 15, 2022 •

edited

Loading