Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] MNMG nightly tests remaining sporadic RF tests failures #3820

Closed
viclafargue opened this issue May 3, 2021 · 5 comments
Closed

[BUG] MNMG nightly tests remaining sporadic RF tests failures #3820

viclafargue opened this issue May 3, 2021 · 5 comments
Assignees
@lowener
Labels
bug Something isn't working Dask / cuml.dask Issue/PR related to Python level dask or cuml.dask features.

Comments

@viclafargue
Copy link
Contributor

MNMG nightly tests only has a few remaining test failures. Fixing these would make it possible to have a perfect MNMG testing for the first time!

3 GPUs:
cuml.test.dask.test_random_forest.test_rf_regression_dask_fil[5-float32]

8 GPUs:
cuml.test.dask.test_random_forest.test_rf_classification_dask_fil_predict_proba[5]
cuml.test.dask.test_random_forest.test_rf_broadcast[True-False-classification]

@hcho3 can you help on this? On my side, I'll investigate the test_rf_broadcast test and look for any regression related to CuPy update in it.
@viclafargue viclafargue added bug Something isn't working Dask / cuml.dask Issue/PR related to Python level dask or cuml.dask features. labels May 3, 2021
@JohnZed JohnZed changed the title [BUG] MNMG nightly tests remaining tests failures [BUG] MNMG nightly tests remaining sporadic RF tests failures May 4, 2021
@JohnZed
Copy link
Contributor

JohnZed commented May 4, 2021

@lowener to take a first look and sync with @hcho3 if more details needed.

@hcho3 hcho3 mentioned this issue May 5, 2021
Merged
@hcho3
Copy link
Contributor

hcho3 commented May 5, 2021

@viclafargue I filed #3830 to fix test_rf_regression_dask_fil. On the other hand, I was not able to reproduce the failure of test_rf_classification_dask_fil_predict_proba, even when I ran it 20 times in a row.

@lowener lowener mentioned this issue May 5, 2021
Merged
@lowener
Copy link
Contributor

lowener commented May 5, 2021 •
edited
Loading

I was able to reproduce the error of test_rf_classification_dask_fil_predict_proba so I made a PR, see #3831

@viclafargue viclafargue mentioned this issue May 6, 2021
Merged
@viclafargue
Copy link
Contributor Author

Thanks guys! On my side, I could discover the reason behind the test_rf_broadcast failure. It wasn't related to CuPy update. I created a fix for it, see #3833.

rapids-bot bot pushed a commit that referenced this issue May 10, 2021
@viclafargue
Fix RF broadcast feature (#3833)
1ea479b 
Answers #3820.

This PR fixes the broadcast feature of the Random Forest estimator. The weights used by the reduction step were generated incorrectly. Indeed, the right values are to be deducted, for each chunk to be predicted, by the number estimators trained by the specific worker holding that chunk. The values wrongly used previously were the number of estimators held by each worker in the order of their construction.

Authors:
  - Victor Lafargue (https://github.com/viclafargue)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #3833
@dantegd dantegd mentioned this issue May 12, 2021
Closed
rapids-bot bot pushed a commit that referenced this issue May 22, 2021
@lowener
Fix for MNMG test_rf_classification_dask_fil_predict_proba (#3831)
ea662e8 
See #3820

Authors:
  - Micka (https://github.com/lowener)

Approvers:
  - Victor Lafargue (https://github.com/viclafargue)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #3831
rapids-bot bot pushed a commit that referenced this issue May 22, 2021
@hcho3
Fix MNMG test test_rf_regression_dask_fil (#3830)
3e89f04 
See #3820

Authors:
  - Philip Hyunsu Cho (https://github.com/hcho3)

Approvers:
  - Victor Lafargue (https://github.com/viclafargue)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #3830
@viclafargue
Copy link
Contributor Author

Closed by #3830, #3831 and #3833.

vimarsh6739 pushed a commit to vimarsh6739/cuml that referenced this issue Oct 9, 2023
@viclafargue
Fix RF broadcast feature (rapidsai#3833)
ce96f3d 
Answers rapidsai#3820.

This PR fixes the broadcast feature of the Random Forest estimator. The weights used by the reduction step were generated incorrectly. Indeed, the right values are to be deducted, for each chunk to be predicted, by the number estimators trained by the specific worker holding that chunk. The values wrongly used previously were the number of estimators held by each worker in the order of their construction.

Authors:
  - Victor Lafargue (https://github.com/viclafargue)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: rapidsai#3833
vimarsh6739 pushed a commit to vimarsh6739/cuml that referenced this issue Oct 9, 2023
@lowener
Fix for MNMG test_rf_classification_dask_fil_predict_proba (rapidsai#…
7e6631c 
…3831)

See rapidsai#3820

Authors:
  - Micka (https://github.com/lowener)

Approvers:
  - Victor Lafargue (https://github.com/viclafargue)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: rapidsai#3831
vimarsh6739 pushed a commit to vimarsh6739/cuml that referenced this issue Oct 9, 2023
@hcho3
Fix MNMG test test_rf_regression_dask_fil (rapidsai#3830)
d2e7269 
See rapidsai#3820

Authors:
  - Philip Hyunsu Cho (https://github.com/hcho3)

Approvers:
  - Victor Lafargue (https://github.com/viclafargue)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: rapidsai#3830
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lowener lowener

Labels
bug Something isn't working Dask / cuml.dask Issue/PR related to Python level dask or cuml.dask features.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@JohnZed @hcho3 @lowener @viclafargue