Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Print sccache stats in all builds #111

Closed
18 tasks done
vyasr opened this issue Oct 22, 2024 · 2 comments
Closed
18 tasks done

Print sccache stats in all builds #111

vyasr opened this issue Oct 22, 2024 · 2 comments
Assignees
@jameslamb

Comments

@vyasr
Copy link
Contributor

vyasr commented Oct 22, 2024 •
edited by jameslamb
Loading

Adding an sccache -s call after every build is free and it is very useful to help diagnose issues when builds are suddenly and unexpectedly slow.

Notes

These can be done in any order.

Updates
Preview Give feedback

Do the GNN repos last, as some of the cugraph repos are currently being shuffled around. See e.g. rapidsai/cugraph-gnn#58 (comment).

GNN repos
Preview Give feedback
@vyasr vyasr mentioned this issue Oct 22, 2024
Merged
3 tasks
@jameslamb
Copy link
Member

Totally support this, for the reasons mentioned in rapidsai/cuspatial#1473 (comment).

I'm going to take this and make these changes at the same time as the changes for #108 .

@jameslamb jameslamb self-assigned this Oct 22, 2024
rapids-bot bot pushed a commit to rapidsai/cuspatial that referenced this issue Oct 22, 2024
@jameslamb
build wheels without build isolation (#1473)
86ef675 
Contributes to rapidsai/build-planning#108

Contributes to rapidsai/build-planning#111

Proposes building `libcuspatial` wheels with `--no-build-isolation`, to improve the rate of `sccache` cache hits and therefore reduce CI times.

Also proposes printing `sccache` stats to CI logs after wheel and conda builds.

## Notes for Reviewers

#

Authors:
  - James Lamb (https://github.com/jameslamb)
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Mike Sarahan (https://github.com/msarahan)
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Bradley Dice (https://github.com/bdice)

URL: #1473
This was referenced Oct 23, 2024
Merged
Merged
Merged
Merged
Merged
Merged
rapids-bot bot pushed a commit to rapidsai/cucim that referenced this issue Oct 28, 2024
@jameslamb
build wheels without build isolation, print sccache stats in builds (#…
89d7c2b 
…792)

Contributes to rapidsai/build-planning#108
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* removes the use of build isolation in wheel builds
* printing `sccache` stats to CI logs
* reducing `pip`'s verbosity in wheel building scripts
* using the RAPIDS conventions `py_build_{project}` and `py_rapids_build_{project}` in `dependencies.yaml`
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #792
rapids-bot bot pushed a commit to rapidsai/cudf that referenced this issue Oct 28, 2024
@jameslamb
build wheels without build isolation (#17088)
abecd0b 
Contributes to rapidsai/build-planning#108
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* building `libcudf` wheels with `--no-build-isolation` (for better `sccache` hit rate)
* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

#

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #17088
This was referenced Oct 28, 2024
Merged
Merged
Merged
Merged
Merged
Merged
Merged
rapids-bot bot pushed a commit to rapidsai/rmm that referenced this issue Oct 29, 2024
@jameslamb
print sccache stats in builds (#1712)
47dae24 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* reducing verbosity of `pip wheel` in wheel builds

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Jake Awe (https://github.com/AyodeAwe)

URL: #1712
rapids-bot bot pushed a commit to rapidsai/wholegraph that referenced this issue Oct 29, 2024
@jameslamb
print sccache stats in builds (#231)
6821f7a 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* reducing `pip wheel` verbosity
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `python` in calls to `rapids-upload-wheels-to-s3`

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Jake Awe (https://github.com/AyodeAwe)

URL: #231
rapids-bot bot pushed a commit to rapidsai/kvikio that referenced this issue Oct 29, 2024
@jameslamb
print sccache stats in builds (#524)
10bc842 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #524
rapids-bot bot pushed a commit to rapidsai/pynvjitlink that referenced this issue Oct 29, 2024
@jameslamb
use sccache, print sccache stats in builds (#112)
666375a 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0) and `rapidsai/pre-commit-hooks` (v0.4.0)
* always explicitly specifying `python` in calls to `rapids-upload-wheels-to-s3`

While doing that, I also noticed this project wasn't using `sccache`. This PR proposes using `sccache` to drive compilation, to speed up builds a bit.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #112
rapids-bot bot pushed a commit to rapidsai/cuvs that referenced this issue Oct 29, 2024
@jameslamb
print sccache stats in builds (#413)
d296d81 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* reducing `pip`'s verbosity in wheel building scripts
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`
* modifying `dependencies.yaml` to match RAPIDS-wide naming conventions

## Notes for Reviewers

This originally also ran wheel builds with `--no-build-isolation`, but I reverted that based on rapidsai/build-planning#108 (comment).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #413
rapids-bot bot pushed a commit to rapidsai/ucx-py that referenced this issue Oct 29, 2024
@jameslamb
remove unnecessary sccache configuration, other small CI changes (#1089)
317b220 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* removing unnecessary calls to `rapids-configure-sccache` (this project does not use `sccache`)

## Notes for Reviewers

I'd originally started this PR with the goal of printing `sccache` stats in builds here... but realized this project does not use `sccache`.

I chose not to pursue adding `sccache` here. The fact that this project doesn't use CMake means it'd take some effort to figure out how to inject `sccache` into the compilation (`CC` / `CXX` environment variables? something else?). Conda and wheel builds are only spending around 20 seconds actually compiling Cython code, so it doesn't seem worth the effort.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Mike Sarahan (https://github.com/msarahan)

URL: #1089
rapids-bot bot pushed a commit to rapidsai/cugraph-gnn that referenced this issue Oct 30, 2024
@jameslamb
add PR CI for cugraph-pyg and cugraph-dgl (#59)
31ee98f 
Another steps towards completing the work started in #53 

Fixes #15

Contributes to rapidsai/build-planning#111

Proposes changes to get CI running on pull requests for `cugraph-pyg` and `cugraph-dgl`

## Notes for Reviewers

Workflows for nightly builds and publishing nightly packages are intentionally not included here. See #58 (comment)

Notebook tests are intentionally not added here... they'll be added in the next PR.

Pulls in changes from these other upstream PRs that had not been ported over to this repo:

* rapidsai/cugraph#4690
* rapidsai/cugraph#4393

Authors:
  - James Lamb (https://github.com/jameslamb)
  - Alex Barghi (https://github.com/alexbarghi-nv)

Approvers:
  - Alex Barghi (https://github.com/alexbarghi-nv)
  - Bradley Dice (https://github.com/bdice)

URL: #59
rapids-bot bot pushed a commit to rapidsai/cugraph that referenced this issue Oct 30, 2024
@jameslamb
print sccache stats in builds (#4719)
050d524 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

## Notes for Reviewers

This originally also ran wheel builds with `--no-build-isolation`, but I reverted that based on rapidsai/build-planning#108 (comment).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #4719
rapids-bot bot pushed a commit to rapidsai/ucxx that referenced this issue Oct 31, 2024
@jameslamb
build wheels without build isolation (#301)
922b3a4 
Contributes to rapidsai/build-planning#108
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* building `libucxx` wheels with `--no-build-isolation` (for better `sccache` hit rate)
* printing `sccache` stats to CI logs
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`
* moving more one-wheel-specific logic into `build_wheel_{project}.sh` scripts, mimicking how `cudf` has structured its scripts

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #301
rapids-bot bot pushed a commit to rapidsai/cuml that referenced this issue Oct 31, 2024
@jameslamb
print sccache stats in builds (#6111)
27e47e5 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* reducing `pip`'s verbosity in wheel building scripts
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

## Notes for Reviewers

This originally also ran wheel builds with `--no-build-isolation`, but I reverted that based on rapidsai/build-planning#108 (comment).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #6111
rapids-bot bot pushed a commit to rapidsai/raft that referenced this issue Oct 31, 2024
@jameslamb
print sccache stats in builds (#2470)
41ddbfe 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* reducing `pip`'s verbosity in wheel building scripts
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

## Notes for Reviewers

This originally also ran wheel builds with `--no-build-isolation`, but I reverted that based on rapidsai/build-planning#108 (comment).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #2470
@jameslamb
Copy link
Member

This is complete, yay more metrics 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jameslamb jameslamb

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vyasr @jameslamb