Add GitHub Actions Workflows #1062

bdice · 2022-12-05T16:11:16Z

Description

This PR adds GitHub Actions workflows to dask-cuda.

Task list

Coverage required for this PR:

Python tests
Codecov
Style checks

Future work required:

Deploy sdist/wheels to PyPI

codecov-commenter · 2022-12-07T02:00:23Z

Codecov Report

Base: 0.00% // Head: 0.00% // No change to project coverage 👍

Coverage data is based on head (ce9a14f) compared to base (8c87288).
Patch coverage: 0.00% of modified lines in pull request are covered.

❗ Current head ce9a14f differs from pull request most recent head 5712a8b. Consider uploading reports for the commit 5712a8b to get more accurate results

Additional details and impacted files

@@             Coverage Diff              @@
##           branch-23.02   #1062   +/-   ##
============================================
  Coverage          0.00%   0.00%           
============================================
  Files                26      26           
  Lines              3439    3433    -6     
============================================
+ Misses             3439    3433    -6

Impacted Files	Coverage Δ
dask_cuda/benchmarks/local_cudf_groupby.py	`0.00% <ø> (ø)`
dask_cuda/cuda_worker.py	`0.00% <0.00%> (ø)`
dask_cuda/local_cuda_cluster.py	`0.00% <0.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

bdice · 2022-12-07T03:24:44Z

Thanks @pentschev for helping with the libnuma questions and @ajschmidt8 for updating the base images. Tests appear to be passing now. I’m going to open this for review but it should not be merged until after the 22.12 release, when we will migrate most repos to GitHub Actions. Can someone with privileges mark this DO NOT MERGE?

.github/workflows/build.yaml

.github/workflows/pr.yaml

ci/release/update-version.sh

- `strategy.fail-fast` is only necessary for matrix builds - `runs-on` can be changed to use GH hosted runners since the respective builds are quick - GH hosted runners are cheaper and are usually available a bit quicker than our self-hosted nodes - `env` vars aren't necessary

the environment variables standardized by these steps are only used in the `gha-tools` scripts that upload/download artifacts from S3. since these jobs don't upload/download artifacts from S3, these enironment variables are not necessary

`shellcheck` prefers `$()` vs. backticks

using a more generic matching pattern will help prevent potential issues in the future if the tags somehow get out of sync

and remove `external_contributors` since its not a valid entry

ajschmidt8 · 2023-01-01T21:00:00Z

I don't think the latest pytest failures are related to the changes in our PR.

They also seem to be appearing in #1069.

pentschev · 2023-01-03T09:26:50Z

I don't think the latest pytest failures are related to the changes in our PR.

They also seem to be appearing in #1069.

They're not related, I opened #1070 to track that.

pentschev

Thanks @bdice and @ajschmidt8 for the hard work you've put on this. I've added a few comments.

pentschev · 2023-01-03T10:54:51Z

conda/recipes/dask-cuda/meta.yaml


 # Usage:
 #   conda build -c conda-forge .
 {% set data = load_file_data("pyproject.toml") %}

 {% set version = environ.get('GIT_DESCRIBE_TAG', '0.0.0.dev').lstrip('v') + environ.get('VERSION_SUFFIX', '') %}
 {% set number = environ.get('GIT_DESCRIBE_NUMBER', 0) %}
-{% set py_version = environ.get('CONDA_PY', 36) %}
+{% set py_version = environ['CONDA_PY'] %}


I understand probably now we are sure that CONDA_PY is set, but what if there's a regression, wouldn't be better to keep a safe default like other variables have?

This is set by conda, not by RAPIDS. This should be safe -- and if it's not, we want to have a hard failure. Also we want to avoid setting defaults that need continually updated in the recipes as we change Python version support.

.github/workflows/build.yaml

pentschev · 2023-01-03T12:01:23Z

ci/test_python.sh

+    echo "FAILED: Local benchmark with explicit comms"
+fi
+
+exit ${SUITEERROR}


This will only capture the latest error, which may or may not be fine depending on how this is handled outside. Since we can't return multiple error codes on a single script, I think it might make sense to:

Have one script per executed test (in this particular case we would need 3, the first pytest and two benchmarks); or

Logging the error code for each failing test and returning a generic error code that the caller of this script will be able to parse (e.g., 0 for "success" and 1 for "error(s) occurred").

We have plans to revisit and simplify some of this logic in follow-up PRs, but ultimately I think we only care to distinguish zero (success) from non-zero (failure) exit codes for the purposes of GitHub Actions calling this script.

pentschev · 2023-01-03T12:02:13Z

dependencies.yaml

+  - rapidsai
+  - rapidsai-nightly


Do we really want both rapidsai and rapidsai-nightly here?

Yes. The tests require cudf (among other RAPIDS packages), and the pinning of cudf=23.02 requires rapidsai-nightly. For any dependencies on other RAPIDS packages, we want to test against nightlies so we are aware of breakage before release time.

dask-cuda/dependencies.yaml

Lines 87 to 88 in 297accb

- cudf=23.02

- dask-cudf=23.02

ajschmidt8 · 2023-01-03T22:35:12Z

rerun tests

ajschmidt8 · 2023-01-03T22:54:16Z

.github/workflows/build.yaml

+      - name: Build wheel
+        run: ci/build_python_pypi.sh
+      - name: Publish distribution 📦 to PyPI
+        if: inputs.build_type == 'branch'


Suggested change

if: inputs.build_type == 'branch'

if: inputs.build_type == 'nightly'

For parity with the Jenkins setup, we should only publish this once a day (at night).

We'll also need to add this repo to the RAPIDS nightly pipeline below

https://github.com/rapidsai/actions/blob/main/.github/workflows/nightly-pipeline.yaml

.github/workflows/pr.yaml

ajschmidt8 · 2023-01-04T15:04:36Z

/merge

Add GitHub Actions Workflows [skip gpuci]

56f84a4

github-actions bot added the gpuCI gpuCI issue label Dec 5, 2022

bdice added 9 commits December 5, 2022 10:15

Merge remote-tracking branch 'upstream/branch-23.02' into gha

47678e8

Add style check script.

f645d8a

Update coverage config.

151568f

Add numactl-libs-cos7-ARCH to testing dependencies.

dbfeb96

Try devel instead of libs for numactl.

52fe124

Add dask-cudf to testing dependencies.

6ea6272

Add needs.

b89b7f8

Update CODEOWNERS.

4503a1f

Run CI with newer image containing libnuma.

7737c68

bdice marked this pull request as ready for review December 7, 2022 03:24

bdice requested a review from a team as a code owner December 7, 2022 03:24

madsbk added 5 - DO NOT MERGE Hold off on merging; see PR for details improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Dec 7, 2022

bdice changed the title ~~Add GitHub Actions Workflows [skip gpuci]~~ Add GitHub Actions Workflows Dec 7, 2022

bdice added 2 commits December 7, 2022 12:50

Retest with gpuCI enabled.

5bf3c68

Minor updates.

05b601c

github-actions bot added the conda conda issue label Dec 13, 2022

bdice added 8 commits December 14, 2022 11:01

Require CONDA_PY.

712b565

Add dependencies.yaml to update-version.sh.

18412ae

Add basic wheel build job.

c49f3a6

Add environment variables.

d5c08da

Fix YAML syntax.

20b154b

Fix YAML again.

1d2b497

Use fetch-depth: 0.

a0e2299

Add version suffix.

bb0e90f

bdice added 3 commits December 21, 2022 15:39

Use source.

9145f2a

Use bash.

ef673b9

Add wheel build workflow to build.yaml.

1e4dad2

ajschmidt8 reviewed Dec 21, 2022

View reviewed changes

.github/workflows/build.yaml Outdated Show resolved Hide resolved

.github/workflows/pr.yaml Outdated Show resolved Hide resolved

.github/workflows/pr.yaml Outdated Show resolved Hide resolved

ci/release/update-version.sh Outdated Show resolved Hide resolved

ajschmidt8 and others added 10 commits January 1, 2023 12:15

rm standardize repository information steps

8a07c21

the environment variables standardized by these steps are only used in the `gha-tools` scripts that upload/download artifacts from S3. since these jobs don't upload/download artifacts from S3, these enironment variables are not necessary

update bash command substitution syntax

5840b12

`shellcheck` prefers `$()` vs. backticks

rm PyPI upload step from PR workflow

de20ceb

rm references to CURRENT_SHORT_TAG in `update-version.sh

7c2b129

using a more generic matching pattern will help prevent potential issues in the future if the tags somehow get out of sync

rerun update-version.sh to sync versions

c7c484f

rm extraneous matrix key from output: none file entry

2e2c9d9

mv common build steps to build_python_pypi.sh

e681811

fix build_python_pypi.sh

6ae8ccb

Merge branch 'branch-23.02' into gha

6c9d26e

ajschmidt8 removed the 5 - DO NOT MERGE Hold off on merging; see PR for details label Jan 1, 2023

enable recently_updated plugin

d9af730

and remove `external_contributors` since its not a valid entry

update nightly workflows [skip ci]

297accb

pentschev reviewed Jan 3, 2023

View reviewed changes

Merge branch 'branch-23.02' into gha

0b874d3

ajschmidt8 approved these changes Jan 3, 2023

View reviewed changes

Update .github/workflows/build.yaml

2a3eaff

ajschmidt8 reviewed Jan 3, 2023

View reviewed changes

.github/workflows/pr.yaml Show resolved Hide resolved

.github/workflows/pr.yaml Show resolved Hide resolved

ajschmidt8 and others added 2 commits January 3, 2023 17:59

Apply suggestions from code review

80bf6db

fix YAML spacing

5712a8b

rapids-bot bot merged commit d78c60a into rapidsai:branch-23.02 Jan 4, 2023

jakirkham mentioned this pull request Oct 4, 2023

Migrate to GitHub Actions rapidsai/jupyterlab-nvdashboard#170

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GitHub Actions Workflows #1062

Add GitHub Actions Workflows #1062

bdice commented Dec 5, 2022 •

edited

Loading

codecov-commenter commented Dec 7, 2022 •

edited

Loading

bdice commented Dec 7, 2022

ajschmidt8 commented Jan 1, 2023

pentschev commented Jan 3, 2023

pentschev left a comment

pentschev Jan 3, 2023

bdice Jan 3, 2023 •

edited

Loading

pentschev Jan 3, 2023

bdice Jan 3, 2023

pentschev Jan 3, 2023

bdice Jan 3, 2023 •

edited

Loading

ajschmidt8 commented Jan 3, 2023

ajschmidt8 Jan 3, 2023

ajschmidt8 commented Jan 4, 2023

	if: inputs.build_type == 'branch'
	if: inputs.build_type == 'nightly'

Add GitHub Actions Workflows #1062

Add GitHub Actions Workflows #1062

Conversation

bdice commented Dec 5, 2022 • edited Loading

Description

Task list

codecov-commenter commented Dec 7, 2022 • edited Loading

Codecov Report

bdice commented Dec 7, 2022

ajschmidt8 commented Jan 1, 2023

pentschev commented Jan 3, 2023

pentschev left a comment

Choose a reason for hiding this comment

pentschev Jan 3, 2023

Choose a reason for hiding this comment

bdice Jan 3, 2023 • edited Loading

Choose a reason for hiding this comment

pentschev Jan 3, 2023

Choose a reason for hiding this comment

bdice Jan 3, 2023

Choose a reason for hiding this comment

pentschev Jan 3, 2023

Choose a reason for hiding this comment

bdice Jan 3, 2023 • edited Loading

Choose a reason for hiding this comment

ajschmidt8 commented Jan 3, 2023

ajschmidt8 Jan 3, 2023

Choose a reason for hiding this comment

ajschmidt8 commented Jan 4, 2023

bdice commented Dec 5, 2022 •

edited

Loading

codecov-commenter commented Dec 7, 2022 •

edited

Loading

bdice Jan 3, 2023 •

edited

Loading

bdice Jan 3, 2023 •

edited

Loading