Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix warnings in test_csv.py. #10362

Merged
merged 3 commits into from
Mar 1, 2022
Merged

Fix warnings in test_csv.py. #10362

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
02d1de2
Fix warning related to integer overflow in pandas.
bdice Feb 25, 2022
e015b23
Refactor tests of overflowing hexadecimal values.
bdice Feb 28, 2022
d7045b1
Use string as dtype.
bdice Mar 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion python/cudf/cudf/tests/test_csv.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1315,7 +1315,7 @@ def test_csv_reader_aligned_byte_range(tmpdir):
[(None, None), ("int", "hex"), ("int32", "hex32"), ("int64", "hex64")],
)
def test_csv_reader_hexadecimals(pdf_dtype, gdf_dtype):
lines = ["0x0", "-0x1000", "0xfedcba", "0xABCDEF", "0xaBcDeF", "9512c20b"]
lines = ["0x0", "-0x1000", "0xfedcba", "0xABCDEF", "0xaBcDeF"]
values = [int(hex_int, 16) for hex_int in lines]

buffer = "\n".join(lines)
Expand All @@ -1334,6 +1334,35 @@ def test_csv_reader_hexadecimals(pdf_dtype, gdf_dtype):
assert_eq(pdf, gdf)


@pytest.mark.parametrize(
"np_dtype, gdf_dtype",
[("int", "hex"), ("int32", "hex32"), ("int64", "hex64")],
)
def test_csv_reader_hexadecimal_overflow(np_dtype, gdf_dtype):
# This tests values which cause an overflow warning that will become an
# error in pandas. NumPy wraps the overflow silently up to the bounds of a
# signed int64.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it always 64?

Copy link
Contributor Author

@bdice bdice Feb 28, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup. By default, numpy treats types larger than int64/uint64 with object as the dtype and uses a Python int to back it. There are larger types, like np.int128, but they're not used by default. The wider the type, the less-wide the support, I guess.

Copy link
Contributor Author

@bdice bdice Mar 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction: I was thinking of floating types, not integral types. There are 128-bit float types (depending on the platform/build of NumPy) like np.longdouble but 128-bit integers are not in the NumPy API. However, 128-bit integers seem to be used internally and references can be found in a few places in the source.

References:

>>> np.finfo(np.longdouble)
finfo(resolution=1e-18, min=-1.189731495357231765e+4932, max=1.189731495357231765e+4932, dtype=float128)

lines = [
"0x0",
"-0x1000",
"0xfedcba",
"0xABCDEF",
"0xaBcDeF",
"0x9512c20b",
"0x7fffffff",
"0x7fffffffffffffff",
"-0x8000000000000000",
]
values = [int(hex_int, 16) for hex_int in lines]
buffer = "\n".join(lines)

gdf = read_csv(StringIO(buffer), dtype=[gdf_dtype], names=["hex_int"])

expected = np.array(values, dtype=np_dtype)
actual = gdf["hex_int"].to_numpy()
np.testing.assert_array_equal(expected, actual)


@pytest.mark.parametrize("quoting", [0, 1, 2, 3])
def test_csv_reader_pd_consistent_quotes(quoting):
names = ["text"]
Expand Down