GH-21: Add release CI #122

kou · 2024-09-12T08:45:17Z

This is based on
https://github.com/apache/arrow-julia/tree/main/dev/release and https://github.com/apache/arrow-adbc/tree/main/dev/release .

Workflow:

Cut a RC:

Update PkgVersion in arrow/doc.go
Run dev/release/release_rc.sh
1. dev/release/release_rc.sh pushes v${version}-rc${rc} tag
2. .github/workflows/rc.yml creates apache-arrow-go-${version}.tar.gz{,.sha256,.sha512}
3. .github/workflows/rc.yml uploads apache-arrow-go-${version}.tar.gz{,.sha256,.sha512} to GitHub Releases
4. dev/release/release_rc.sh downloads apache-arrow-go-${version}.tar.gz from GitHub Releases
5. dev/release/release_rc.sh signs apache-arrow-go-${version}.tar.gz as apache-arrow-go-${version}.tar.gz.asc
6. dev/release/release_rc.sh uploads apache-arrow-go-${version}.tar.gz.asc to GitHub Releases
Start a vote

(GitHub Actions instead of
https://dist.apache.org/repos/dist/dev/arrow/ is used like ADBC.)

Verify a RC:

Run dev/release/verify_rc.sh

Release an approved RC:

Run dev/release/release.sh
1. dev/release/release.sh pushes v${version} tag that refers that same commit ID as v${version}-rc${rc}
2. dev/release/release.sh downloads apache-arrow-go-${version}.tar.gz{,.asc,.sha256,.sha512} from GitHub Actions
3. dev/release/release.sh uploads apache-arrow-go-${version}.tar.gz{,.asc,.sha256,.sha512} to https://dist.apache.org/repos/dist/release/arrow
4. dev/release/release.sh removes old releases from https://dist.apache.org/repos/dist/release/arrow
Add this release to ASF's report database: https://reporter.apache.org/addrelease.html?arrow

Follow-up tasks:

Add support for running integration test in the verification script
Add support for releasing a release note
Add support for creating "v${version}"'s GitHub Releases
- We can copy "v${version}-rc${rc}"'s GitHub Releases and adjust it for the official release

kou · 2024-09-12T08:46:04Z

Example RC:

raulcd

Some minor comments but looks good! Great work @kou I've tested the verification script locally against the uploaded RC to your fork and have validated successfully.

dev/release/release_rc.sh

raulcd · 2024-09-13T09:06:45Z

arrow/doc.go

-const PkgVersion = "18.0.0-SNAPSHOT"
+const PkgVersion = "18.0.0"


this doesn't have to change for this PR, only when releasing, right? or do we want this to be 17.0.0 and increment on the new release?
I think we should probably continue using SNAPSHOT and just remove the SNAPSHOT bit when releasing.

this doesn't have to change for this PR, only when releasing, right?

No.

or do we want this to be 17.0.0 and increment on the new release?

No.

My suggestions:

Don't use -SNAPSHOT suffix for unreleased version

I think that it's needless because we don't create a nightly (or something) package from main

I think that the -SNAPSHOT suffix name is Maven convention: https://maven.apache.org/maven-release/maven-release-plugin/usage/prepare-release.html

If we want to use -XXX suffix for unreleased version, other name such as -dev is better

We don't update this just after we release a new version

Because we can't know the next version (18.0.1? 18.1.0? 19.0.0?) just after we release a new version

We update this just before we release a new version, when we add a new compatible feature (minor version up) or when we add an incompatible change (major version up)

sounds good to me to update just before releasing

.github/workflows/rc.yml

dev/release/README.md

dev/release/release.sh

zeroshade

just a few nitpicks from me

zeroshade · 2024-09-13T17:52:48Z

.github/workflows/rc.yml

+            sed \
+              -e 's/^const PkgVersion = "//' \
+              -e 's/"$//')
+          rc=100


why rc=100?

I just want to use RC that is never used for normal RCs.
We can use 1000, $(date %Y%m%d) or something instead of 100 here.

ah, gotcha. $(date %Y%m%d) is probably best i think?

OK. I'll use it.

zeroshade · 2024-09-13T18:20:17Z

dev/release/verify_rc.sh

+    go clean -modcache || :
+    if [ "${VERIFY_SUCCESS}" = "yes" ]; then
+      rm -rf "${VERIFY_TMPDIR}"
+    else
+      echo "Failed to verify release candidate. See ${VERIFY_TMPDIR} for details."
+    fi


instead of calling go clean -modcache we could try just setting the GOCACHE and GOMODCACHE env vars to point to directories in the tmpdir. That way we still get clean caches, but we don't blow up the users cache dirs.

Removing cache isn't the main purpose of it.
It's for avoiding a permission problem. See also: apache/arrow#38222

BTW, do we need to set GOCACHE/GOMODCACHE explicitly when we set GOPATH? Can GOCACHE/GOMODCACHE be set automatically under GOPATH?

Ah gotcha.

BTW, do we need to set GOCACHE/GOMODCACHE explicitly when we set GOPATH? Can GOCACHE/GOMODCACHE be set automatically under GOPATH?

GOMODCACHE should generally be ${GOPATH}/pkg/mod, we might not need to change it explicitly since we set GOPATH.

GOCACHE can be anywhere though and typically is set to ~/.cache/go-build, if we're not running go clean -cache or otherwise cleaning the build cache, and only using go clean -modcache then we're probably fine here without issue.

OK. Let's use go clean -modcache here.

zeroshade · 2024-09-13T18:21:18Z

dev/release/verify_rc.sh

+  if [ -d "${TOP_SOURCE_DIR}/arrow-testing/data" ]; then
+    cp -a "${TOP_SOURCE_DIR}/arrow-testing" "${ARCHIVE_BASE_NAME}/"
+  else
+    git clone \
+      https://github.com/apache/arrow-testing.git \
+      "${ARCHIVE_BASE_NAME}/arrow-testing"
+  fi
+  if [ -d "${TOP_SOURCE_DIR}/parquet-testing/data" ]; then
+    cp -a "${TOP_SOURCE_DIR}/parquet-testing" "${ARCHIVE_BASE_NAME}/"
+  else
+    git clone \
+      https://github.com/apache/parquet-testing.git \
+      "${ARCHIVE_BASE_NAME}/parquet-testing"
+  fi
+
+  ARROW_TEST_DATA="${ARCHIVE_BASE_NAME}/arrow-testing/data"
+  export ARROW_TEST_DATA
+  PARQUET_TEST_DATA="${ARCHIVE_BASE_NAME}/parquet-testing/data"
+  export PARQUET_TEST_DATA
+}


Need to add PARQUET_TEST_BAD_DATA now as per #124

OK. I'll add it.

Fix apacheGH-21 This is based on https://github.com/apache/arrow-julia/tree/main/dev/release and https://github.com/apache/arrow-adbc/tree/main/dev/release . Workflow: Cut a RC: 1. Update `PkgVersion` in `arrow/doc.go` 2. Run `dev/release/release_rc.sh` 1. `dev/release/release_rc.sh` pushes `v${version}-rc${rc}` tag 2. `.github/workflows/rc.yml` creates `apache-arrow-go-${version}.tar.gz{,.sha256,.sha512}` 3. `.github/workflows/rc.yml` uploads `apache-arrow-go-${version}.tar.gz{,.sha256,.sha512}` to GitHub Releases 4. `dev/release/release_rc.sh` downloads `apache-arrow-go-${version}.tar.gz` from GitHub Releases 5. `dev/release/release_rc.sh` signs `apache-arrow-go-${version}.tar.gz` as `apache-arrow-go-${version}.tar.gz.asc` 6. `dev/release/release_rc.sh` uploads `apache-arrow-go-${version}.tar.gz.asc` to GitHub Releases 3. Start a vote (GitHub Actions instead of https://dist.apache.org/repos/dist/dev/arrow/ is used like ADBC.) Verify a RC: 1. Run `dev/release/verify_rc.sh` Release an approved RC: 1. Run `dev/release/release.sh` 1. `dev/release/release.sh` pushes `v${version}` tag that refers that same commit ID as `v${version}-rc${rc}` 2. `dev/release/release.sh` downloads `apache-arrow-go-${version}.tar.gz{,.asc,.sha256,.sha512}` from GitHub Actions 3. `dev/release/release.sh` uploads `apache-arrow-go-${version}.tar.gz{,.asc,.sha256,.sha512}` to https://dist.apache.org/repos/dist/release/arrow 4. `dev/release/release.sh` removes old releases from https://dist.apache.org/repos/dist/release/arrow 2. Add this release to ASF's report database: https://reporter.apache.org/addrelease.html?arrow Follow-up tasks: * Add support for running integration test in the verification script * Add support for releasing a release note * Add support for creating "v${version}"'s GitHub Releases * We can copy "v${version}-rc${rc}"'s GitHub Releases and adjust it for the official release

Co-authored-by: Raúl Cumplido <[email protected]>

raulcd

Thanks @kou

zeroshade · 2024-09-16T20:43:07Z

.github/workflows/rc.yml

+        with:
+          submodules: recursive
+      - name: Prepare for tag
+        if: github.ref_type == 'tag'


should this be ${{ github.ref_type }} ?

We can omit ${{ ... }}: https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idif

When you use expressions in an if conditional, you can, optionally, omit the ${{ }} expression syntax because GitHub Actions automatically evaluates the if conditional as an expression. However, this exception does not apply everywhere.

You must always use the ${{ }} expression syntax or escape with '', "", or () when the expression starts with !, since ! is reserved notation in YAML format.

kou · 2024-09-17T01:49:00Z

I'll merge this.

kou requested a review from raulcd September 13, 2024 05:10

raulcd reviewed Sep 13, 2024

View reviewed changes

zeroshade reviewed Sep 13, 2024

View reviewed changes

kou and others added 5 commits September 14, 2024 06:11

Apply suggestions from code review

8e4c9bd

Co-authored-by: Raúl Cumplido <[email protected]>

Improve style

a0e1de0

Update ignores

af2470b

Add PARQUET_TEST_BAD_DATA

c15bec7

kou force-pushed the release branch from 7de52c7 to c15bec7 Compare September 13, 2024 21:12

raulcd approved these changes Sep 16, 2024

View reviewed changes

zeroshade approved these changes Sep 16, 2024

View reviewed changes

zeroshade reviewed Sep 16, 2024

View reviewed changes

Use today instead of "100" for RC

293a07c

kou merged commit c6d372a into apache:main Sep 17, 2024
23 checks passed

kou deleted the release branch September 17, 2024 02:11

kou mentioned this pull request Sep 17, 2024

Prepare release procedure #18

Closed

kou mentioned this pull request Nov 25, 2024

Add release CI apache/arrow-java#20

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-21: Add release CI #122

GH-21: Add release CI #122

kou commented Sep 12, 2024

kou commented Sep 12, 2024

raulcd left a comment

raulcd Sep 13, 2024

kou Sep 13, 2024

raulcd Sep 16, 2024

zeroshade left a comment

zeroshade Sep 13, 2024

kou Sep 13, 2024

zeroshade Sep 16, 2024

kou Sep 17, 2024

zeroshade Sep 13, 2024

kou Sep 13, 2024

zeroshade Sep 16, 2024

kou Sep 17, 2024

zeroshade Sep 13, 2024

kou Sep 13, 2024

raulcd left a comment

zeroshade Sep 16, 2024

kou Sep 17, 2024

kou commented Sep 17, 2024

		const PkgVersion = "18.0.0-SNAPSHOT"
		const PkgVersion = "18.0.0"

GH-21: Add release CI #122

GH-21: Add release CI #122

Conversation

kou commented Sep 12, 2024

kou commented Sep 12, 2024

raulcd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeroshade left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raulcd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou commented Sep 17, 2024