Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pin arrow 9 in testing dependencies to prevent conda solve issues #12377

Merged
merged 8 commits into from
Dec 14, 2022

Conversation

vyasr
Copy link
Contributor

@vyasr vyasr commented Dec 13, 2022

Description

This pinning is necessary because other testing dependencies require arrow (currently, transformers has a transitive dependency on it) and don't include a pinning as strict as ours, resulting in a newer arrow package (arrow 10) being installed. When we attempt to install our cudf/libcudf packages in CI, it should be able to downgrade to arrow 9, but the downgrade fails because changes in arrow's package structure in arrow 10 include the creation of a new libarrow package (arrow-cpp was renamed to libarrow) and libarrow has no equivalent in arrow 9. As a result, conda does not see a way to downgrade it, and it also does not see a way to satisfy its dependencies if the rest of the arrow packages are downgraded to arrow 9 (because libarrow is tied to arrow 10 packages), so the conda solve fails.

The current changeset will fix Github Actions CI. We may need additional fixes in our Jenkins CI.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@github-actions github-actions bot added the ci label Dec 13, 2022
@vyasr vyasr force-pushed the fix/pin_arrow_tests branch from 4b97c16 to 8dc0c28 Compare December 13, 2022 23:01
@github-actions github-actions bot removed the ci label Dec 13, 2022
@vyasr vyasr changed the title [DO NOT MERGE] Debug arrow 10 installation and openssl requirements Pin arrow 9 in testing dependencies to prevent conda solve issues Dec 13, 2022
@vyasr vyasr self-assigned this Dec 13, 2022
@vyasr vyasr added bug Something isn't working 2 - In Progress Currently a work in progress non-breaking Non-breaking change labels Dec 13, 2022
@codecov
Copy link

codecov bot commented Dec 14, 2022

Codecov Report

Base: 86.58% // Head: 86.57% // Decreases project coverage by -0.00% ⚠️

Coverage data is based on head (8b097b5) compared to base (b6dccb3).
Patch has no changes to coverable lines.

❗ Current head 8b097b5 differs from pull request most recent head 88600c4. Consider uploading reports for the commit 88600c4 to get more accurate results

Additional details and impacted files
@@               Coverage Diff                @@
##           branch-23.02   #12377      +/-   ##
================================================
- Coverage         86.58%   86.57%   -0.01%     
================================================
  Files               155      155              
  Lines             24368    24517     +149     
================================================
+ Hits              21098    21225     +127     
- Misses             3270     3292      +22     
Impacted Files Coverage Δ
python/cudf/cudf/core/buffer/spill_manager.py 74.37% <0.00%> (-5.63%) ⬇️
python/cudf/cudf/utils/dtypes.py 77.77% <0.00%> (-1.69%) ⬇️
python/cudf/cudf/options.py 86.11% <0.00%> (-1.59%) ⬇️
python/cudf/cudf/core/single_column_frame.py 94.30% <0.00%> (-1.27%) ⬇️
python/dask_cudf/dask_cudf/io/parquet.py 91.81% <0.00%> (-0.59%) ⬇️
python/cudf/cudf/core/multiindex.py 91.66% <0.00%> (-0.51%) ⬇️
python/cudf/cudf/core/algorithms.py 90.00% <0.00%> (-0.48%) ⬇️
python/cudf/cudf/core/dataframe.py 93.50% <0.00%> (-0.09%) ⬇️
python/cudf/cudf/core/join/_join_helpers.py 98.52% <0.00%> (-0.07%) ⬇️
python/dask_cudf/dask_cudf/tests/test_core.py 95.42% <0.00%> (-0.02%) ⬇️
... and 29 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@vyasr vyasr marked this pull request as ready for review December 14, 2022 00:16
@vyasr vyasr requested a review from a team as a code owner December 14, 2022 00:16
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. Note that this should be resolved by #12327.

@vyasr
Copy link
Contributor Author

vyasr commented Dec 14, 2022

@ajschmidt8 As expected, we'll need to do something similar to fix this for gpuCI (or we can simply disable gpuCI at this point!).

@github-actions github-actions bot added the ci label Dec 14, 2022
Copy link
Member

@ajschmidt8 ajschmidt8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I added a quick workaround that should hopefully work for Jenkins. But yes, we will soon disable Jenkins at which point those scripts will be removed entirely.

ajschmidt8 and others added 5 commits December 14, 2022 02:19
this commit (hopefully) ensures that `pyarrow` version `10` never gets installed in the first place
even though `pytorch` was excluded from the `aarch64` dependency list prior to this commit, it was still getting installed as a transitive dependency. further, the version being installed was `pytorch=1.13.0`, which seemed to pull in other dependencies that weren't compatible with our environment
@bdice bdice requested a review from a team as a code owner December 14, 2022 17:57
@github-actions github-actions bot added conda Python Affects Python cuDF API. labels Dec 14, 2022
@ajschmidt8
Copy link
Member

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 72361a4 into rapidsai:branch-23.02 Dec 14, 2022
@bdice bdice removed the 2 - In Progress Currently a work in progress label Dec 14, 2022
@vyasr vyasr deleted the fix/pin_arrow_tests branch December 14, 2022 21:29
@vyasr vyasr mentioned this pull request Dec 21, 2022
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants