Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW] Fix repr and concat of StructColumn #10042

Merged
merged 8 commits into from
Jan 14, 2022
Merged

[REVIEW] Fix repr and concat of StructColumn #10042

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
3cdc00d
fix a concat issue that only shows up in repr
galipremsagar Jan 13, 2022
8e6f1c5
Update dataframe.py
galipremsagar Jan 13, 2022
8e66d09
redundant code
galipremsagar Jan 13, 2022
3d3fc55
add comment
galipremsagar Jan 14, 2022
d47b55e
add comment
galipremsagar Jan 14, 2022
376c606
Merge remote-tracking branch 'upstream/branch-22.02' into 8963
galipremsagar Jan 14, 2022
1500233
merge logic
galipremsagar Jan 14, 2022
7fbf59b
Add comment about type metadata.
bdice Jan 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions python/cudf/cudf/core/dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1548,10 +1548,18 @@ def _concat(
cudf.core.index.as_index(out.index._values)
)

# Reassign precision for any decimal cols
# Reassign precision for decimal cols & type schema for struct cols
for name, col in out._data.items():
if isinstance(col, cudf.core.column.Decimal64Column):
col = col._with_type_metadata(tables[0]._data[name].dtype)
if isinstance(
col,
(
cudf.core.column.Decimal64Column,
cudf.core.column.StructColumn,
Comment on lines +1556 to +1557
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these two types handled the same way? What do they have in common?

Are there other types that might need to be handled here -- and how would developers know to put them here? This may not be something that can be addressed in a code comment, but it seems important to consider. The StructColumn type was seemingly left out from a place it should have been included, until an issue was raised, cause identified, and bug fixed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a common pattern:

cudf/python/cudf/cudf/core/frame.py

Lines 2154 to 2157 in 813ac97

# TODO: _copy_type_metadata is a common pattern to apply after the
# roundtrip from libcudf. We should build this into a factory function
# to increase reusability.
result._copy_type_metadata(self)

To answer why only these two types are being handled separately is because _reassign_categories predates _with_type_metadata and will likely need a cleanup with some careful benchmarking to eliminate _reassign_categories which are out of the scope of this PR.

),
):
out._data[name] = col._with_type_metadata(
tables[0]._data[name].dtype
vyasr marked this conversation as resolved.
Show resolved Hide resolved
bdice marked this conversation as resolved.
Show resolved Hide resolved
)

# Reassign index and column names
if isinstance(objs[0].columns, pd.MultiIndex):
Expand Down
9 changes: 5 additions & 4 deletions python/cudf/cudf/core/series.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1448,10 +1448,11 @@ def _concat(cls, objs, axis=0, index=True):

col = concat_columns([o._column for o in objs])

if isinstance(col, cudf.core.column.Decimal64Column):
col = col._with_type_metadata(objs[0]._column.dtype)

if isinstance(col, cudf.core.column.StructColumn):
# Reassign precision for decimal cols & type schema for struct cols
if isinstance(
bdice marked this conversation as resolved.
Show resolved Hide resolved
col,
(cudf.core.column.Decimal64Column, cudf.core.column.StructColumn),
):
col = col._with_type_metadata(objs[0].dtype)

return cls(data=col, index=index, name=name)
Expand Down
30 changes: 30 additions & 0 deletions python/cudf/cudf/tests/test_repr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1475,3 +1475,33 @@ def test_empty_series_name():
gs = cudf.from_pandas(ps)

assert ps.__repr__() == gs.__repr__()


def test_repr_struct_after_concat():
df = cudf.DataFrame(
{
"a": cudf.Series(
[
{"sa": 2056831253},
{"sa": -1463792165},
{"sa": 1735783038},
{"sa": 103774433},
{"sa": -1413247520},
]
* 13
),
"b": cudf.Series(
[
{"sa": {"ssa": 1140062029}},
None,
{"sa": {"ssa": 1998862860}},
{"sa": None},
{"sa": {"ssa": -395088502}},
]
* 13
),
}
)
pdf = df.to_pandas()

assert df.__repr__() == pdf.__repr__()