Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add an Environment Variable for debugging the fast path in cudf.pandas #15837

Merged
merged 30 commits into from
Jun 9, 2024
Merged

Add an Environment Variable for debugging the fast path in cudf.pandas #15837

merged 30 commits into from
Jun 9, 2024

Conversation

Matt711
Copy link
Contributor

@Matt711 Matt711 commented May 23, 2024 •
edited
Loading

Description

Part of #14975 This PR adds a pandas debugging option to _fast_slow_function_call that runs the slow path after the fast and returns a warning if the results differ.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented May 23, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions bot added the Python Affects Python cuDF API. label May 23, 2024
Matt711
Matt711 commented May 23, 2024
View reviewed changes
Copy link
Contributor Author

@Matt711 Matt711 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a few questions on how to move forward with this PR.

python/cudf/cudf/pandas/fast_slow_proxy.py Outdated Show resolved Hide resolved
python/cudf/cudf_pandas_tests/test_fast_slow_proxy.py Outdated Show resolved Hide resolved
bdice
bdice reviewed May 23, 2024
View reviewed changes
python/cudf/cudf/options.py Outdated Show resolved Hide resolved
python/cudf/cudf/pandas/fast_slow_proxy.py Outdated Show resolved Hide resolved
python/cudf/cudf/pandas/fast_slow_proxy.py Outdated Show resolved Hide resolved
python/cudf/cudf/pandas/fast_slow_proxy.py Outdated Show resolved Hide resolved
python/cudf/cudf_pandas_tests/test_fast_slow_proxy.py Outdated Show resolved Hide resolved
@Matt711 Matt711 added feature request New feature or request non-breaking Non-breaking change labels May 23, 2024
@github-actions github-actions bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue Java Affects Java cuDF API. labels May 24, 2024
@Matt711 Matt711 closed this May 24, 2024
@Matt711 Matt711 force-pushed the feature/combine-fast-and-slow-paths branch from 2dee2bf to 8a40567 Compare May 24, 2024 17:00
@Matt711 Matt711 removed CMake CMake build issue Java Affects Java cuDF API. labels May 24, 2024
@Matt711 Matt711 reopened this May 24, 2024
@github-actions github-actions bot removed the libcudf Affects libcudf (C++/CUDA) code. label May 24, 2024
@Matt711 Matt711 marked this pull request as ready for review May 28, 2024 12:34
@Matt711 Matt711 requested a review from a team as a code owner May 28, 2024 12:34
@Matt711 Matt711 requested review from vyasr and lithomas1 and removed request for a team May 28, 2024 12:34
@Matt711 Matt711 changed the base branch from branch-24.06 to branch-24.08 May 28, 2024 12:37
@Matt711
Copy link
Contributor Author

Matt711 commented May 28, 2024

/ok to test

@Matt711
Copy link
Contributor Author

Matt711 commented May 29, 2024

/ok to test

@Matt711 Matt711 force-pushed the feature/combine-fast-and-slow-paths branch from 9ca34ff to af0d9ce Compare June 3, 2024 15:16
@Matt711
Copy link
Contributor Author

Matt711 commented Jun 3, 2024

/ok to test

@Matt711
Copy link
Contributor Author

Matt711 commented Jun 3, 2024

/ok to test

@Matt711 Matt711 mentioned this pull request Jun 3, 2024
Closed
3 tasks
@mroeschke
Copy link
Contributor

Just an idea that might avoid the need to monkeypatch.

What if if we use a UDF that should always be different between cudf and pandas like:

import time

def udf(*args, *kwargs):
     return time.time()

Then have s.apply(udf) as the base test case?

@bdice
Copy link
Contributor

bdice commented Jun 3, 2024 •
edited
Loading

@mroeschke I considered that kind of thing but I wasn't able to come up with a function (or UDF) that we support in cudf that would intentionally deviate from pandas behavior and be deterministic. The only documented difference is the row ordering of joins, and that's nondeterministic: https://docs.rapids.ai/api/cudf/nightly/cudf_pandas/faq/#are-there-any-known-limitations

@Matt711
Copy link
Contributor Author

Matt711 commented Jun 3, 2024 •
edited
Loading 

import time

def udf(*args, *kwargs):
     return time.time()

Then have s.apply(udf) as the base test case?

I think the results returned are the same. When I tried the udf you suggested before here, I got the same thing.

@mroeschke
Copy link
Contributor

Ah thanks both for checking 👍

@github-actions github-actions bot added the cudf.pandas Issues specific to cudf.pandas label Jun 4, 2024
@Matt711 Matt711 requested a review from bdice June 4, 2024 14:37
rapids-bot bot pushed a commit that referenced this pull request Jun 5, 2024
@Matt711
DOC: Add documentation for cudf.pandas in the Developer Guide (#15889)
20aa444 
This PR provides documentation for cudf.pandas in the Developer Guide. It will describe the fast-slow proxy wrapping scheme as well as document the `CUDF_PANDAS_DEBUGGING` environment variable created in PR #15837 for issue #14975.

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Lawrence Mitchell (https://github.com/wence-)

URL: #15889
@galipremsagar
Copy link
Contributor

/okay to test

@Matt711 Matt711 force-pushed the feature/combine-fast-and-slow-paths branch from 4637b04 to 4a61868 Compare June 7, 2024 15:35
@Matt711
Copy link
Contributor Author

Matt711 commented Jun 7, 2024

/okay to test

bdice
bdice approved these changes Jun 7, 2024
View reviewed changes
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the many iterations here. I would like a few clarifying comments about why the explicit monkeypatch undo is needed, otherwise I think this is good to go. Add those comments and you can merge once other reviewers are happy.

python/cudf/cudf_pandas_tests/test_cudf_pandas.py Show resolved Hide resolved
@galipremsagar
Copy link
Contributor

/okay to test

@Matt711 Matt711 dismissed stale reviews from vyasr and wence- June 8, 2024 00:48

All reviews addressed

@Matt711
Copy link
Contributor Author

Matt711 commented Jun 9, 2024

/merge

@rapids-bot rapids-bot bot merged commit bfad68c into rapidsai:branch-24.08 Jun 9, 2024
70 checks passed
@beckernick beckernick mentioned this pull request Jun 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mroeschke mroeschke mroeschke left review comments

@galipremsagar galipremsagar galipremsagar approved these changes

@bdice bdice bdice approved these changes

@vyasr vyasr Awaiting requested review from vyasr

@wence- wence- Awaiting requested review from wence-

Assignees

@Matt711 Matt711

Labels
cudf.pandas Issues specific to cudf.pandas feature request New feature or request non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
cuDF Python
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Matt711 @mroeschke @bdice @galipremsagar @wence- @vyasr