use rapids-build-backend #1044

jameslamb · 2024-06-05T03:19:31Z

Contributes to rapidsai/build-planning#31
Contributes to rapidsai/dependency-file-generator#89
Contributes to rapidsai/build-planning#72

Proposes:

introducing rapids-build-backend as this project's build backend, to reduce the complexity of various CI/build scripts
making get-data-from-pyproject.toml code in conda recipe a bit stricter

Notes for Reviewers

This reverts some of the workarounds introduced for docs builds in #1041. See this thread for context: #1041 (comment)

~~To do that, I set 2 environment variables manually in in the readthedocs console (link):~~

~~RAPIDS_DISABLE_CUDA=true = used to prevent needing nvcc in the docs-building environment~~
~~RAPIDS_MATRIX_ENTRY='cuda=12.2' = used to ensure that pip install . tries to install a real package (e.g. libucx-cu12)~~

~~I think this is the first place we're using the rapids-build-backend environment variable support.~~

see #1044 (comment)

…build-backend

jameslamb · 2024-06-12T15:05:26Z

dependencies.yaml

@@ -136,8 +150,6 @@ dependencies:
          - matrix: {cuda: "11.*"}
            packages:
              - libucx-cu11==1.15.0
-          # NOTE: this fallback needs to be a real, suffixed version
-          #       so 'pip install .' (e.g. as used in docs builds) will work


I accidentally left behind this comment in #1041 ... it's not accurate on branch-0.39 or on this branch.

jameslamb · 2024-06-12T15:08:10Z

conda/environments/builddocs.yml

+  #       # install CUDA-12-specific dependencies
+  #       # (e.g. libucx-cu12)
+  #       RAPIDS_MATRIX_ENTRY="cuda=12.2"
+  #


Thanks to @pentschev for giving me access to set environment variables in the readthedocs console for this project 😁

bdice

A few suggestions, but overall this is pretty close.

bdice · 2024-06-12T15:31:07Z

conda/environments/builddocs.yml

+  #       # install CUDA-12-specific dependencies
+  #       # (e.g. libucx-cu12)
+  #       RAPIDS_MATRIX_ENTRY="cuda=12.2"
+  #
  - ../../


It looks like we use a conda env with a pip dependency on ../... I think it would be clearer to install a conda env with the relevant dependencies and then invoke pip install with the appropriate flags on the command line instead of requiring environment variables in the UI. Is that an option?

It's not an option. readthedocs doesn't allow you to customize the flags passed to pip install.

You can see the history that led to this pattern here: #1041 (comment)

Hmm. I saw there were options to run arbitrary commands like installing with poetry or uv, so I figured we could do the same with pip and custom flags. https://docs.readthedocs.io/en/stable/build-customization.html#install-dependencies-with-poetry

!!! in all of that conversation in #1041, I was so focused on "how do we change the install: step", I never considered the possibility "just ignore it completely and install the library through one of the other steps where you can run custom commands". The tunnel vision was real 😭

Thanks for that link, let me try this.

This worked! I can't believe I missed that going through options in #1041, thank you so much 😅

I removed the environment variables in the readthedocs console, then pushed the changes you now see in .readthedocs.yaml on this PR.

build: https://readthedocs.org/projects/ucx-py/builds/24674174/

docs: https://ucx-py--1044.org.readthedocs.build/en/1044/

I think this is unambiguously better than having the pip: section in a conda environment YAML file.

conda/recipes/ucx-py/meta.yaml

bdice · 2024-06-12T15:33:23Z

conda/recipes/ucx-py/meta.yaml

@@ -47,12 +50,12 @@ test:
    - ucp

 about:
-  home: {{ data.get("project", {}).get("urls", {}).get("Homepage", "") }}
-  license: {{ data.get("project", {}).get("license", {}).get("text", "") }}
+  home: {{ data["project"]["urls"]["Homepage"] }}


I thought the .get(...) was required because conda-build doesn't actually inject the Jinja variables on every pass and that led to problems in metadata. I'm not sure. We use the .get(...) logic in dask-cuda, too. https://github.com/rapidsai/dask-cuda/blob/branch-24.08/conda/recipes/dask-cuda/meta.yaml

It looks to me like this is working as expected.

wget \ https://downloads.rapids.ai/ci/ucx-py/pull-request/1046/43beaa1/ucx-py_conda_python_cuda12_py39_x86_64.tar.gz tar -xvzf ./ucx-py_conda_python_cuda12_py39_x86_64.tar.gz cat ./linux-64/repodata.json | jq .

For example, I see the right license: string there.

And looking in one of the packages, all this metadata looks correct too:

mkdir ./delete-me cp ./linux-64/ucx-py-0.39.00a1-py39_240612_g43beaa1_1.tar.bz2 ./delete-me cd ./delete-me cph extract ./ucx-py-0.39.00a1-py39_240612_g43beaa1_1.tar.bz2 cat ucx-py-0.39.00a1-py39_240612_g43beaa1_1/info/about.json | jq .

Unless you can remember the specific reason to prefer .get(), I think we should move away from it in all conda recipes, to turn "silently ignored missing metadata" cases into "got a loud error at build time" cases. I actually just wrote that up in rapidsai/build-planning#72.

Ok cool! Go for it.

@jakirkham may remember the specific reason why we used .get(...) in these recipes, I recall discussing it with him at some point.

I found a related thread: rapidsai/pynvjitlink#33 (comment)

I don't think it changes the fact that this is working and should be used for the specific places I've made this change in this PR, and doesn't need to block this PR. But might mean "never use .get()" is too strong.

I'll post over in the build-planning issue.

Maybe conda-build has changed. 🤷‍♂️ In any case, I am happy to drop .get() if this is working.

tests/test_version.py

Co-authored-by: Bradley Dice <[email protected]>

bdice

All seems fine once you're comfortable with the state of the conversations from my last review.

…to rapids-build-backend

pentschev

LGTM, I agree with @bdice 's suggestion on version test, other than that we're good. Thanks @jameslamb !

tests/test_version.py

jameslamb · 2024-06-12T18:33:01Z

/merge

@pentschev

Follow-up to #1044 Contributes to rapidsai/build-planning#31 Related to #1047 * updates `libucx` build requirements so they can be satisfied by only packages from pypi.org * removes `--extra-index-url https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/` in `pip install` calls, now that `rapids-build-backend` is on pypi.org (https://pypi.org/project/rapids-build-backend/) * sets `rapids-build-backend` config `disable-cuda=true` in `pyproject.toml` - *this means that now `pip install .` will not require `nvcc` or produce a wheel with a suffix like `-cu12`* - *modified CI script used to build wheels to override this, so the published wheels will still have `-cu${ver}` suffixes * updates docs on source installation to reflect these changes ## Notes for Reviewers These changes came out of an offline conversation with @pentschev and @vyasr Authors: - James Lamb (https://github.com/jameslamb) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Vyas Ramasubramani (https://github.com/vyasr) URL: #1048

use rapids-build-backend

d31ad60

jameslamb added non-breaking improvement labels Jun 5, 2024

jameslamb mentioned this pull request Jun 5, 2024

Update RAPIDS Python packages to use rapids-build-backend rapidsai/build-planning#31

Closed

jameslamb added 7 commits June 4, 2024 21:06

fix dependencies

bfdc823

version tests

745b40f

changes

6cf9e4e

fix

74dbf00

just run wheel builds

663320d

try moving libucx dependency

2f06f89

put libucx back, it is needed in setup.py

369d57e

This was referenced Jun 5, 2024

[RFC] handling CUDA-suffixed dependencies needed at setuptools.setup() time rapidsai/rapids-build-backend#39

Closed

use rapids-build-backend rapidsai/ucxx#234

Merged

jameslamb added 3 commits June 10, 2024 11:30

Merge branch 'branch-0.39' of github.com:rapidsai/ucx-py into rapids-…

0ed0e1d

…build-backend

test with rapidsai/rapids-build-backend#41

5fd0b04

move other dependencies down to rapids-build-backend's table

893ff43

jameslamb mentioned this pull request Jun 10, 2024

skip get_requires for setuptools projects rapidsai/rapids-build-backend#41

Merged

jameslamb added 11 commits June 10, 2024 11:50

wheel, not install

2e2889f

fix pip wheel invocation

6c9fb3f

get better logs

15fde53

fix pip wheel invocation

7fbb1c1

re-enable wheel tests

3e1e04f

move cudf dependency

473f306

empty commit to re-trigger CI

0dddea7

use rapids-build-backend packages

772ac92

re-enable conda builds

b964fee

fix data-gathering in conda recipe

d6241e5

fix variable name

39960dd

jameslamb mentioned this pull request Jun 12, 2024

Use stricter pattern for reading data from other files in conda recipes rapidsai/build-planning#72

Closed

jameslamb changed the title ~~WIP: use rapids-build-backend~~ use rapids-build-backend Jun 12, 2024

jameslamb commented Jun 12, 2024

View reviewed changes

jameslamb marked this pull request as ready for review June 12, 2024 15:13

jameslamb requested review from a team as code owners June 12, 2024 15:13

jameslamb requested a review from bdice June 12, 2024 15:13

Merge branch 'branch-0.39' into rapids-build-backend

5cd1908

bdice requested changes Jun 12, 2024

View reviewed changes

Apply suggestions from code review

8a443be

Co-authored-by: Bradley Dice <[email protected]>

bdice approved these changes Jun 12, 2024

View reviewed changes

jameslamb added 5 commits June 12, 2024 09:10

try library install in post_create_environment

3238af2

Merge branch 'rapids-build-backend' of github.com:jameslamb/ucx-py in…

6d3fde7

…to rapids-build-backend

fix keys

dbd5722

consider rapids nightly index

51c0153

fix formatting

ec63083

pentschev approved these changes Jun 12, 2024

View reviewed changes

tests/test_version.py Outdated Show resolved Hide resolved

use same __version__ test as other libraries, fix rtd config

5de66ec

jameslamb mentioned this pull request Jun 12, 2024

use libucx wheels #1041

Merged

rapids-bot bot merged commit 42c03ef into rapidsai:branch-0.39 Jun 12, 2024
39 checks passed

jameslamb deleted the rapids-build-backend branch June 12, 2024 18:33

jameslamb mentioned this pull request Jun 13, 2024

make pip source installs a bit easier #1048

Merged

pentschev mentioned this pull request Jun 14, 2024

[QST] How to install from source with system's ucx libs #1047

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use rapids-build-backend #1044

use rapids-build-backend #1044

jameslamb commented Jun 5, 2024 •

edited

Loading

jameslamb Jun 12, 2024

jameslamb Jun 12, 2024

bdice left a comment

bdice Jun 12, 2024

jameslamb Jun 12, 2024

bdice Jun 12, 2024

jameslamb Jun 12, 2024 •

edited

Loading

jameslamb Jun 12, 2024

bdice Jun 13, 2024

bdice Jun 12, 2024

jameslamb Jun 12, 2024

bdice Jun 12, 2024

bdice Jun 12, 2024

jameslamb Jun 12, 2024

bdice Jun 13, 2024

bdice left a comment

pentschev left a comment

jameslamb commented Jun 12, 2024

use rapids-build-backend #1044

use rapids-build-backend #1044

Conversation

jameslamb commented Jun 5, 2024 • edited Loading

Notes for Reviewers

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameslamb Jun 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

pentschev left a comment

Choose a reason for hiding this comment

jameslamb commented Jun 12, 2024

jameslamb commented Jun 5, 2024 •

edited

Loading

jameslamb Jun 12, 2024 •

edited

Loading