Add validation on wheel file characteristics in CI #110

jameslamb · 2024-10-22T15:23:59Z

Description

As we start to publish more RAPIDS wheels directly to https://pypi.org/, it'll be important to validate the following characteristics:

README is well-formatted and metadata is correct
- (because the README is rendered into the project homepage, e.g. https://pypi.org/project/numpy/)
individual file sizes are within PyPI's limits
- (100 MiB by default, larger for projects where we've been given exceptions, like File Limit Request: libcudf-cu12 - 600 MiB pypi/support#4893)

These and any other non-functional characteristics of wheels that can be validated easily, quickly, and programmatically should be validated in CI.

Benefits of this work

improves release confidence ... reduces the risk of "oops these wheels are too big for PyPI" surprises near the end of a release
reduces the effort required to identify which changes led to increases in wheel sizes
- similar to how setting a test coverage threshold works
- ref: Support dynamic linking between RAPIDS wheels #33 (comment)

Acceptance Criteria

CI for RAPIDS libraries fails on pull requests when wheel builds produce wheels that:
- do not pass twine check --strict
- are too large (compressed)

Approach

Such checks should run in wheel-building CI jobs, on PRs... they don't require GPUs, and should run so quickly that it isn't worth the spin-up time and cost of a dedicated wheel-validation CI job.

This could be accomplished with something like the following in wheel-build scripts, after the wheel file is finalized (e.g. has gone through auditwheel repair).

# check README formatting
twine check --strict dist/*

# assert size, and print a summary of contents
pydistcheck \
  --inspect \
  --select 'distro-too-large-compressed' \
  --max-allowed-size-compressed '100M' \
  ./delete-me/*

example of this using 24.12 libcudf-cu12 nightlies (click me)

On an x86_64 linux system:

mkdir -p ./delete-me
pip download \
    --extra-index-url https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/ \
    --no-deps \
    -d ./delete-me \
    'libcudf-cu12==24.12.*,>=0.0.0a0'

Checking the formatting.

twine check --strict ./delete-me/*

Checking ./delete-me/libcudf_cu12-24.12.0a223-py3-none-manylinux_2_28_x86_64.whl: PASSED

And checking the size (using 5M just to show how this would fail in CI if the wheels got too big):

pydistcheck \
  --inspect \
  --select 'distro-too-large-compressed' \
  --max-allowed-size-compressed '5M' \
  ./delete-me/*

==================== running pydistcheck ====================

checking './delete-me/libcudf_cu12-24.12.0a223-py3-none-manylinux_2_28_x86_64.whl'
----- package inspection summary -----
file size
  * compressed size: 0.4G
  * uncompressed size: 0.7G
  * compression space saving: 35.2%
contents
  * directories: 163
  * files: 1974 (2 compiled)
size by extension
  * .so - 0.6G (97.0%)
  * .h - 6.7M (1.0%)
  * no-extension - 5.3M (0.8%)
  * .cuh - 3.8M (0.6%)
  * .hpp - 2.2M (0.3%)
  * .a - 1.2M (0.2%)
  * .inl - 0.8M (0.1%)
  * .cmake - 0.1M (0.0%)
  * .md - 8.3K (0.0%)
  * .py - 3.6K (0.0%)
  * .pc - 0.2K (0.0%)
  * .txt - 34.0B (0.0%)
largest files
  * (0.6G) libcudf/lib64/libcudf.so
  * (3.8M) libcudf/bin/flatc
  * (1.1M) libcudf/lib64/libflatbuffers.a
  * (0.5M) libcudf/include/libcudf/rapids/libcudacxx/cuda/std/__atomic/functions/cuda_ptx_generated.h
  * (0.2M) libcudf_cu12-24.12.0a223.dist-info/RECORD
------------ check results -----------
1. [distro-too-large-compressed] Compressed size 0.4G is larger than the allowed size (5.0M).
errors found while checking: 1

==================== done running pydistcheck ===============

echo $?
# 1

This should be rolled out to all RAPIDS projects producing wheels, starting with those targeting publishing to https://pypi.org/.

Notes

Disclaimer

pydistcheck is something I (@jameslamb) created and am currently the sole contributor on, so I'm biased about it being the best tool for this purpose.

The narrow goal here, "assert files are not too big", could be accomplished other ways... like unzipping the wheel and using some mix of stat / du / find -size, or with a Python script.

I think the summaries pydistcheck generates (see "Approach" section) and all the others things we could ask it to check in the future (docs link) make it preferable to those approaches, but I shouldn't be the one to decide that.

Task Lists

The text was updated successfully, but these errors were encountered:

bdice · 2024-10-22T15:57:38Z

I took a brief look at pydistcheck and would support adopting it. It seems like a great tool for this job. It's also good to lean into your expertise in this area. 👍

vyasr · 2024-10-22T17:07:10Z

Yup I'm happy with trying out pydistcheck. Maybe we should do a group review of it in a future build-infra team meeting to make sure we're all aligned. Adding a size check to CI is definitely critical so that we know when someone tries to add a feature that suddenly bloats binary size.

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* ## Notes for Reviewers ### How I tested this Initially set the size threshold for `libcudf` to a value that I knew it'd violate (75MB compressed, when the wheels are 400+ MB compressed). Saw CI fail as expected, and print a summary with the expected contents. ```text checking 'final_dist/libcudf_cu11-24.12.0a333-py3-none-manylinux_2_28_aarch64.whl' ----- package inspection summary ----- file size * compressed size: 0.4G * uncompressed size: 0.6G * compression space saving: 34.6% contents * directories: 164 * files: 1974 (2 compiled) size by extension * .so - 0.6G (97.0%) * .h - 6.7M (1.0%) * no-extension - 4.8M (0.7%) * .cuh - 3.8M (0.6%) * .hpp - 2.2M (0.3%) * .a - 1.1M (0.2%) * .inl - 0.8M (0.1%) * .cmake - 0.1M (0.0%) * .md - 8.3K (0.0%) * .py - 4.0K (0.0%) * .pc - 0.2K (0.0%) * .txt - 34.0B (0.0%) largest files * (0.6G) libcudf/lib64/libcudf.so * (3.3M) libcudf/bin/flatc * (1.0M) libcudf/lib64/libflatbuffers.a * (0.5M) libcudf/include/libcudf/rapids/libcudacxx/cuda/std/__atomic/functions/cuda_ptx_generated.h * (0.2M) libcudf_cu11-24.12.0a333.dist-info/RECORD ------------ check results ----------- 1. [distro-too-large-compressed] Compressed size 0.4G is larger than the allowed size (75.0M). errors found while checking: 1 ``` ([build link](https://github.com/rapidsai/cudf/actions/runs/11748370606/job/32732391718?pr=17284#step:13:3062)) Updated that threshold in `python/libcudf/pyproject.toml`, and saw the build succeed (but the summary still printed). # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #17284

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1092

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #548

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1404

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1726

## Description Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/*

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #6136

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #797

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Mark Harris (https://github.com/harrism) - Bradley Dice (https://github.com/bdice) URL: #1482

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #464

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #319

…on Cython dependency (#2490) Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Also puts a ceiling on Cython to its latest stable release (`<=3.0.11`), to fix #2490 (comment). Work to relax that is tracked in (#2491). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #2490

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Ajay Thorve (https://github.com/AjayThorve) URL: #643

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Alex Barghi (https://github.com/alexbarghi-nv) URL: #67

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* ## Notes for Reviewers This intentionally omits the libraries that are about to be moved to other repos. * `cugraph-dgl` / `cugraph-pyg` (rapidsai/cugraph-gnn#67) * `nx-cugraph` (rapidsai/nx-cugraph#19) Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #4754

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#548

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Erik Welch (https://github.com/eriknw) - Bradley Dice (https://github.com/bdice) URL: #19

jameslamb · 2024-11-15T22:13:22Z

This is complete. Thanks everyone!

Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* This intentionally omits the libraries that are about to be moved to other repos. * `cugraph-dgl` / `cugraph-pyg` (rapidsai/cugraph-gnn#67) * `nx-cugraph` (rapidsai/nx-cugraph#19) Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#4754

jameslamb added the improvement Improves an existing functionality label Oct 22, 2024

jameslamb mentioned this issue Oct 22, 2024

Support dynamic linking between RAPIDS wheels #33

Open

jameslamb self-assigned this Nov 1, 2024

This was referenced Nov 8, 2024

enforce wheel size limits, README formatting in CI rapidsai/cudf#17284

Merged

add pydistcheck to ci-wheel image rapidsai/ci-imgs#205

Merged

jameslamb mentioned this issue Nov 14, 2024

Add wheel-size checks in CI rapidsai/pynvjitlink#116

Open

jameslamb closed this as completed Nov 15, 2024

jameslamb mentioned this issue Dec 11, 2024

Improve design and address comments in the new CI dmlc/xgboost#11079

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add validation on wheel file characteristics in CI #110

Add validation on wheel file characteristics in CI #110

jameslamb commented Oct 22, 2024 •

edited

Loading

Updates

bdice commented Oct 22, 2024

vyasr commented Oct 22, 2024

jameslamb commented Nov 15, 2024

Add validation on wheel file characteristics in CI #110

Add validation on wheel file characteristics in CI #110

Comments

jameslamb commented Oct 22, 2024 • edited Loading

Description

Benefits of this work

Acceptance Criteria

Approach

Notes

Disclaimer

Task Lists

Updates

bdice commented Oct 22, 2024

vyasr commented Oct 22, 2024

jameslamb commented Nov 15, 2024

jameslamb commented Oct 22, 2024 •

edited

Loading