-
Notifications
You must be signed in to change notification settings - Fork 915
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Abstract polars function expression nodes to ensure they are serializable #17418
Merged
rapids-bot
merged 14 commits into
rapidsai:branch-25.02
from
pentschev:polars-expr-pickling
Nov 26, 2024
Merged
Changes from 4 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
6ea021c
Implement picklable `BooleanFunctionName` type
pentschev 9f2e5c8
Implement picklable `StringFunctionName` type
pentschev d142f50
Implement picklable `TemporalFunctionName` type
pentschev be7fa52
Rename method to `from_polars` and mark `@classmethod`
pentschev 3f69d98
Move types to internal classes `Name` and use `IntEnum`
pentschev 7d009f9
Use `is` for `IntEnum` equality checks
pentschev 5675ef9
Merge branch 'branch-25.02' into polars-expr-pickling
pentschev b7d2bf1
Merge remote-tracking branch 'upstream/branch-25.02' into polars-expr…
pentschev 9b54437
Handle string unpacking failure
pentschev 5a04207
Add basic serialization tests
pentschev 07ba6fe
Merge remote-tracking branch 'origin/polars-expr-pickling' into polar…
pentschev fcf820f
Replace pytest parametrization with a fixture
pentschev 4de3b79
Rewrite test to check for all polars attributes
pentschev 17a1a72
Fix regex to be matched in tests
pentschev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
from __future__ import annotations | ||
|
||
import pickle | ||
|
||
import pytest | ||
|
||
from polars.polars import _expr_nodes as pl_expr | ||
|
||
from cudf_polars.dsl.expressions.boolean import BooleanFunction | ||
from cudf_polars.dsl.expressions.datetime import TemporalFunction | ||
from cudf_polars.dsl.expressions.string import StringFunction | ||
|
||
|
||
@pytest.mark.parametrize( | ||
"function", [BooleanFunction, TemporalFunction, StringFunction] | ||
) | ||
def test_function_name_serialization_all_values(function): | ||
# Test serialization and deserialization for all values of function.Name | ||
for name in function.Name: | ||
serialized_name = pickle.dumps(name) | ||
deserialized_name = pickle.loads(serialized_name) | ||
assert deserialized_name is name | ||
|
||
|
||
@pytest.mark.parametrize( | ||
"function", [BooleanFunction, TemporalFunction, StringFunction] | ||
) | ||
def test_function_name_invalid(function): | ||
# Test invalid attribute name | ||
with pytest.raises( | ||
AttributeError, match="type object 'Name' has no attribute 'InvalidAttribute'" | ||
): | ||
assert function.Name.InvalidAttribute is function.Name.InvalidAttribute | ||
|
||
|
||
@pytest.mark.parametrize( | ||
"function", [BooleanFunction, TemporalFunction, StringFunction] | ||
) | ||
def test_from_polars_all_names(function): | ||
# Test that all valid names of polars expressions are correctly converted | ||
for name in function.Name: | ||
polars_function = getattr(pl_expr, function.__name__) | ||
polars_function_attr = getattr(polars_function, name.name) | ||
cudf_function = function.Name.from_polars(polars_function_attr) | ||
assert cudf_function == name | ||
wence- marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
@pytest.mark.parametrize( | ||
"function", [BooleanFunction, TemporalFunction, StringFunction] | ||
) | ||
def test_from_polars_invalid_attribute(function): | ||
# Test converting from invalid attribute name | ||
with pytest.raises(ValueError, match=f"{function.__name__} required"): | ||
function.Name.from_polars("InvalidAttribute") | ||
|
||
|
||
@pytest.mark.parametrize( | ||
"function", [BooleanFunction, TemporalFunction, StringFunction] | ||
) | ||
def test_from_polars_invalid_polars_attribute(function): | ||
# Test converting from polars function with invalid attribute name | ||
with pytest.raises( | ||
AttributeError, match="type object 'Name' has no attribute 'InvalidAttribute'" | ||
): | ||
function.Name.from_polars(f"{function.__name__}.InvalidAttribute") | ||
wence- marked this conversation as resolved.
Show resolved
Hide resolved
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's abstract this into a reusable fixture:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea, done in fcf820f . I've kept the module references instead of using strings and resolved them in the tests with
__name__
, let me know if you have a strong preference for strings instead.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, that's fine.