Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix sort_values when column is all empty strings #12988

Merged
merged 2 commits into from
Mar 22, 2023
Merged

Fix sort_values when column is all empty strings #12988

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9538a0b
Fix `sort_values` when column is all empty strings
eriknw Mar 21, 2023
0a18ddf
"this string intentionally left empty"
eriknw Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions python/dask_cudf/dask_cudf/sorting.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2020-2022, NVIDIA CORPORATION.
# Copyright (c) 2020-2023, NVIDIA CORPORATION.

from collections.abc import Iterator

Expand Down Expand Up @@ -218,9 +218,10 @@ def quantile_divisions(df, by, npartitions):
divisions[col].iloc[-1] += 1
divisions[col] = divisions[col].astype(dtype)
else:
divisions[col].iloc[-1] = chr(
ord(divisions[col].iloc[-1][0]) + 1
)
if last := divisions[col].iloc[-1]:
divisions[col].iloc[-1] = chr(ord(last[0]) + 1)
else:
divisions[col].iloc[-1] = chr(1) # b/c "" < chr(1)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, using chr(0) here doesn't work, because drop_duplicates below drops it by treating "" and chr(0) the same, even though "" < chr(0). Do we care that chr(1) is not printable?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually a bug in [i]loc.__setitem__:

import cudf
x = cudf.Series(["", chr(0)])
x.drop_duplicates() == x # True
y = cudf.Series(["a", "b"])
y.iloc[0] = ""
y.iloc[1] = chr(0)
y.iloc[0] == y.iloc[1] # True

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in the case that last is the empty string, it suffices to provide any non-empty string as the upper bound on division?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#12990.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it suffices to provide any non-empty string

Yup. How about "wence was here"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"nonempty"? I don't like to leave footprints :).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"this string intentionally left empty"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like David's suggestion (or we can wait for #12991 and use chr(0))

divisions = divisions.drop_duplicates().sort_index()
return divisions

Expand Down
12 changes: 11 additions & 1 deletion python/dask_cudf/dask_cudf/tests/test_sort.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2019-2022, NVIDIA CORPORATION.
# Copyright (c) 2019-2023, NVIDIA CORPORATION.

import cupy as cp
import numpy as np
Expand Down Expand Up @@ -104,3 +104,13 @@ def f(partition, by_columns, ascending, na_position, **kwargs):
)
expect = df.sort_values(by=by)
dd.assert_eq(got, expect, check_index=False)


@pytest.mark.parametrize("by", ["a", "b", ["a", "b"], ["b", "a"]])
def test_sort_values_empty_string(by):
df = cudf.DataFrame({"a": [3, 2, 1, 4], "b": [""] * 4})
ddf = dd.from_pandas(df, npartitions=2)
got = ddf.sort_values(by)
if "a" in by:
expect = df.sort_values(by)
assert dd.assert_eq(got, expect, check_index=False)