-
-
Notifications
You must be signed in to change notification settings - Fork 367
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate release CI to github #3406
Changes from all commits
44c3b23
848b44d
d82b34f
a27e1f4
df58681
c9a1e2c
eb204de
e580d72
c10af27
c40c83b
53cbee7
4ddf848
e453d4b
3a5d5de
344681f
8d37ed4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,109 @@ | ||
# release CI for FreeBSD | ||
compute_engine_instance: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What does this file do? Does it need documentation? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a cirrus CI config. Cirrus CI provides FreeBSD runners for free. Documentation is here: https://cirrus-ci.org/guide/writing-tasks/ |
||
image_project: freebsd-org-cloud-dev | ||
image: family/freebsd-13-1 | ||
platform: freebsd | ||
disk: 100 # Gb | ||
hasufell marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
build_task: | ||
timeout_in: 120m | ||
only_if: $CIRRUS_TAG != '' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This variable doesn't seem to be set anywhere? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's set by cirrus CI. It's an automatic variable, documented. |
||
env: | ||
AWS_ACCESS_KEY_ID: ENCRYPTED[dc5896620ebc12e98e6bbe96f72c5a2fe3785f439b7b2346797355f8d329a4bfd8ef6e58086bfc014be0d914424101cd] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Comment with where these come from. Are they the same as the ones from the github secrets? I guess there's no way we can fetch them from there easily. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, they're created via cirrus CI web interface. It's documented in cirrus docs. |
||
AWS_SECRET_ACCESS_KEY: ENCRYPTED[6501cd594aca08c6c67cc679dd6f6d30db0cd44a81cceddebf32bb3d0a37f9af19cd71ddb7169d3f7b284a7829969f9e] | ||
S3_HOST: ENCRYPTED[d3fef1b5850e85d80dd1684370b53183df2218f2d36509108a2703371afd9ebd3f9596ad4de52487c15ea29baed606b7] | ||
TARBALL_EXT: "tar.xz" | ||
ARCH: 64 | ||
ARTIFACT: "x86_64-freebsd" | ||
DISTRO: "na" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why isn't it "FreeBSD"? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The platform is FreeBSD, so we don't use the DISTRO value. We use it only for Linux platform. We set |
||
RUNNER_OS: "FreeBSD" | ||
ADD_CABAL_ARGS: "--enable-split-sections" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why? I see this crops up in a few places but we need to document it. I also wonder whether it would be better to pass this via creating a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think this needs further documentation. It follows the existing gitlab workflow: https://gitlab.haskell.org/haskell/haskell-language-server/-/blob/master/.gitlab/ci.sh#L48 The GHC user guide documents the flag: https://downloads.haskell.org/~ghc/9.2.5/docs/html/users_guide/phases.html?highlight=split%20sections#ghc-flag--split-sections There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Definitely not. |
||
GITHUB_WORKSPACE: ${CIRRUS_WORKING_DIR} | ||
CABAL_CACHE_NONFATAL: "yes" | ||
matrix: | ||
- name: build-ghc-8.10.7 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We need a list of places to update when you change the set of GHC versions, sigh. |
||
env: | ||
GHC_VERSION: 8.10.7 | ||
- name: build-ghc-9.0.2 | ||
env: | ||
GHC_VERSION: 9.0.2 | ||
- name: build-ghc-9.2.5 | ||
env: | ||
GHC_VERSION: 9.2.5 | ||
- name: build-ghc-9.2.7 | ||
env: | ||
GHC_VERSION: 9.2.7 | ||
install_script: pkg install -y hs-cabal-install git bash misc/compat10x misc/compat11x misc/compat12x gmake patchelf tree | ||
script: | ||
- tzsetup Etc/GMT | ||
- adjkerntz -a | ||
- bash .github/scripts/build.sh | ||
- tar caf out.tar.xz out/ store/ | ||
binaries_artifacts: | ||
path: "out.tar.xz" | ||
|
||
|
||
bindist_task: | ||
name: bindist | ||
depends_on: | ||
- build-ghc-8.10.7 | ||
- build-ghc-9.0.2 | ||
- build-ghc-9.2.5 | ||
- build-ghc-9.2.7 | ||
timeout_in: 120m | ||
only_if: $CIRRUS_TAG != '' | ||
env: | ||
TARBALL_EXT: "tar.xz" | ||
ARCH: 64 | ||
ARTIFACT: "x86_64-freebsd" | ||
DISTRO: "na" | ||
RUNNER_OS: "FreeBSD" | ||
GITHUB_WORKSPACE: ${CIRRUS_WORKING_DIR} | ||
install_script: pkg install -y hs-cabal-install git bash misc/compat10x misc/compat11x misc/compat12x gmake patchelf tree unzip | ||
script: | ||
- tzsetup Etc/GMT | ||
- adjkerntz -a | ||
|
||
- curl -o binaries-8.10.7.tar.xz -L https://api.cirrus-ci.com/v1/artifact/build/${CIRRUS_BUILD_ID}/build-ghc-8.10.7/binaries/out.tar.xz | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this the approved way to get artifacts between Cirrus jobs? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure what you mean. There's no better way afais. I did research on it. |
||
- tar xvf binaries-8.10.7.tar.xz | ||
- rm -f binaries-8.10.7.tar.xz | ||
|
||
- curl -o binaries-9.0.2.tar.xz -L https://api.cirrus-ci.com/v1/artifact/build/${CIRRUS_BUILD_ID}/build-ghc-9.0.2/binaries/out.tar.xz | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe this should be in a script so we can at least loop over the versions... this is going to be quite painful to update. |
||
- tar xvf binaries-9.0.2.tar.xz | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should this be using TARBALL_EXT? TBH I'm not sure what the point of having TARBALL_EXT as a variable is 🤔 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So the bash scripts don't error on unset variables. |
||
- rm -f binaries-9.0.2.tar.xz | ||
|
||
- curl -o binaries-9.2.5.tar.xz -L https://api.cirrus-ci.com/v1/artifact/build/${CIRRUS_BUILD_ID}/build-ghc-9.2.5/binaries/out.tar.xz | ||
- tar xvf binaries-9.2.5.tar.xz | ||
- rm -f binaries-9.2.5.tar.xz | ||
|
||
- curl -o binaries-9.2.7.tar.xz -L https://api.cirrus-ci.com/v1/artifact/build/${CIRRUS_BUILD_ID}/build-ghc-9.2.7/binaries/out.tar.xz | ||
- tar xvf binaries-9.2.7.tar.xz | ||
- rm -f binaries-9.2.7.tar.xz | ||
|
||
- bash .github/scripts/bindist.sh | ||
bindist_artifacts: | ||
path: "./out/*.tar.xz" | ||
|
||
test_task: | ||
name: test | ||
depends_on: | ||
- bindist | ||
timeout_in: 120m | ||
only_if: $CIRRUS_TAG != '' | ||
env: | ||
TARBALL_EXT: "tar.xz" | ||
ARCH: 64 | ||
ARTIFACT: "x86_64-freebsd" | ||
DISTRO: "na" | ||
RUNNER_OS: "FreeBSD" | ||
GITHUB_WORKSPACE: ${CIRRUS_WORKING_DIR} | ||
install_script: pkg install -y hs-cabal-install git bash misc/compat10x misc/compat11x misc/compat12x gmake patchelf tree unzip | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can this be a reference to a script? Since it's the same every time, perhaps we want |
||
script: | ||
- tzsetup Etc/GMT | ||
- adjkerntz -a | ||
|
||
- curl -O -L https://api.cirrus-ci.com/v1/artifact/build/${CIRRUS_BUILD_ID}/bindist/bindist.zip | ||
- unzip bindist.zip | ||
|
||
- bash .github/scripts/test.sh | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
#!/bin/bash | ||
|
||
set -eux | ||
|
||
. .github/scripts/env.sh | ||
. .github/scripts/common.sh | ||
|
||
# ensure ghcup | ||
if ! command -v ghcup ; then | ||
install_ghcup | ||
fi | ||
|
||
# create tarball/zip | ||
case "${TARBALL_EXT}" in | ||
zip) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why are these two so different? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Zip is used on windows |
||
HLS_VERSION="$(grep '^version:' haskell-language-server.cabal | awk '{ print $2 }')" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd be tempted to make this a parameter and have a separate step for getting it. It's a little surprising that it gets pulled out in here, it's not where I'd look! |
||
( | ||
cd "$CI_PROJECT_DIR/out/${ARTIFACT}" | ||
zip "$CI_PROJECT_DIR/out/haskell-language-server-${HLS_VERSION}-${ARTIFACT}.zip" haskell-language-server-* | ||
) | ||
;; | ||
tar.xz) | ||
# we need to control the order, so the hls wrapper binary is installed | ||
# from the oldest version in the list | ||
: "${GHCS:="$(cd "$CI_PROJECT_DIR/out/${ARTIFACT}" && rm -f ./*.json && for ghc in * ; do printf "%s\n" "$ghc" ; done | sort -r | tr '\n' ' ')"}" | ||
emake --version | ||
emake GHCUP=ghcup ARTIFACT="${ARTIFACT}" GHCS="${GHCS}" bindist | ||
emake GHCUP=ghcup ARTIFACT="${ARTIFACT}" bindist-tar | ||
emake GHCUP=ghcup GHCS="${GHCS}" clean-ghcs | ||
;; | ||
*) | ||
fail "Unknown TARBALL_EXT: ${TARBALL_EXT}" | ||
;; | ||
esac |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
#!/bin/sh | ||
|
||
set -eux | ||
|
||
. .github/scripts/env.sh | ||
|
||
if [ -e "$HOME/.brew" ] ; then | ||
( | ||
cd "$HOME/.brew" | ||
git fetch --depth 1 | ||
git reset --hard origin/master | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is where we pin to a specific commit, right? should that be a parameter? |
||
) | ||
else | ||
git clone --depth=1 https://github.com/Homebrew/brew "$HOME/.brew" | ||
fi | ||
export PATH="$HOME/.brew/bin:$HOME/.brew/sbin:$PATH" | ||
|
||
mkdir -p $CI_PROJECT_DIR/.brew_cache | ||
export HOMEBREW_CACHE=$CI_PROJECT_DIR/.brew_cache | ||
mkdir -p $CI_PROJECT_DIR/.brew_logs | ||
export HOMEBREW_LOGS=$CI_PROJECT_DIR/.brew_logs | ||
mkdir -p /private/tmp/.brew_tmp | ||
export HOMEBREW_TEMP=/private/tmp/.brew_tmp | ||
|
||
#brew update | ||
brew install ${1+"$@"} | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
#!/bin/bash | ||
|
||
set -eux | ||
|
||
. .github/scripts/env.sh | ||
. .github/scripts/common.sh | ||
|
||
uname -a | ||
uname -p | ||
uname | ||
pwd | ||
env | ||
|
||
# ensure ghcup | ||
if ! command -v ghcup ; then | ||
install_ghcup | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you have a reason not to want to use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, it's unreliable. |
||
fi | ||
|
||
# ensure cabal-cache | ||
download_cabal_cache "$HOME/.local/bin/cabal-cache" | ||
|
||
|
||
# build | ||
ecabal update | ||
ghcup install ghc "${GHC_VERSION}" | ||
ghcup set ghc "${GHC_VERSION}" | ||
"ghc-${GHC_VERSION}" --info | ||
"ghc" --info | ||
|
||
mkdir -p "$CI_PROJECT_DIR/out/${ARTIFACT}" | ||
mkdir -p "$CI_PROJECT_DIR/out/plan.json" | ||
|
||
case "$(uname)" in | ||
MSYS_*|MINGW*) | ||
args=( -O2 -w "ghc-$GHC_VERSION" --project-file cabal.project --disable-profiling --disable-tests --enable-executable-stripping ${ADD_CABAL_ARGS}) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Doesn't For the others, I again wonder whether it would be simpler to setup a |
||
|
||
# Shorten binary names | ||
# due to MAX_PATH issues on windows | ||
sed -i.bak -e 's/haskell-language-server/hls/g' \ | ||
-e 's/haskell_language_server/hls/g' \ | ||
haskell-language-server.cabal cabal.project | ||
sed -i.bak -e 's/Paths_haskell_language_server/Paths_hls/g' \ | ||
src/**/*.hs exe/*.hs | ||
|
||
# shellcheck disable=SC2068 | ||
build_with_cache ${args[@]} exe:hls exe:hls-wrapper | ||
cp dist-newstyle/cache/plan.json "$CI_PROJECT_DIR/out/plan.json/${ARTIFACT}-ghc-${GHC_VERSION}-plan.json" | ||
|
||
# shellcheck disable=SC2068 | ||
cp "$(cabal list-bin -v0 ${args[@]} exe:hls)" "$CI_PROJECT_DIR/out/${ARTIFACT}/haskell-language-server-${GHC_VERSION}${ext}" | ||
# shellcheck disable=SC2068 | ||
cp "$(cabal list-bin -v0 ${args[@]} exe:hls-wrapper)" "$CI_PROJECT_DIR/out/${ARTIFACT}/haskell-language-server-wrapper${ext}" | ||
;; | ||
*) | ||
sed -i.bak -e '/DELETE MARKER FOR CI/,/END DELETE/d' cabal.project # see comment in cabal.project | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have no idea. This is ported from existing CI. |
||
emake --version | ||
emake GHCUP=ghcup CABAL_CACHE_BIN=cabal-cache.sh S3_HOST="${S3_HOST}" S3_KEY="${ARTIFACT}" GHC_VERSION="${GHC_VERSION}" hls-ghc | ||
;; | ||
esac | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
#!/usr/bin/env bash | ||
|
||
case "$(uname -s)" in | ||
MSYS_*|MINGW*) | ||
ext=".exe" | ||
;; | ||
*) | ||
ext="" | ||
;; | ||
esac | ||
|
||
if [ "${CABAL_CACHE_DISABLE}" = "yes" ] ; then | ||
echo "cabal-cache disabled (CABAL_CACHE_DISABLE set)" | ||
elif [ "${CABAL_CACHE_NONFATAL}" = "yes" ] ; then | ||
time "cabal-cache${ext}" "$@" || echo "cabal-cache failed (CABAL_CACHE_NONFATAL set)" | ||
else | ||
time "cabal-cache${ext}" "$@" | ||
exit $? | ||
fi | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add yourself to CODEOWNERS for this and the other files, so you get review pings?