Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: json_normalize raises boardcasting error with list-like metadata #47708

Closed
wants to merge 22 commits into from
+42 −5
Closed

BUG: json_normalize raises boardcasting error with list-like metadata #47708

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
039a659
fix repeat axis
GYHHAHA Jul 13, 2022
1645550
add tests
GYHHAHA Jul 13, 2022
6db3e6c
Update v1.5.0.rst
GYHHAHA Jul 13, 2022
49f2950
change tests dtype descriptions
GYHHAHA Jul 13, 2022
b73d0f1
Merge branch 'main' into patch-1
GYHHAHA Jul 19, 2022
8eb1421
drop np array
GYHHAHA Jul 27, 2022
de21683
Update _normalize.py
GYHHAHA Jul 28, 2022
5a3d8bd
Update test_normalize.py
GYHHAHA Jul 28, 2022
a4ac5ec
Merge branch 'pandas-dev:main' into patch-1
GYHHAHA Jul 28, 2022
cc00c3e
fix dtype
GYHHAHA Jul 28, 2022
7efe709
Update pandas/io/json/_normalize.py
GYHHAHA Jul 30, 2022
9ad3db4
type ignore
GYHHAHA Jul 30, 2022
23c090e
fix format
GYHHAHA Jul 30, 2022
b3d500f
remove object type
GYHHAHA Jul 30, 2022
2779bc2
remove astype
GYHHAHA Jul 30, 2022
6252821
fix format
GYHHAHA Jul 30, 2022
34a3ace
import is_list_like
GYHHAHA Jul 30, 2022
f2fea4e
Update _normalize.py
GYHHAHA Aug 6, 2022
06d9f34
Update test_normalize.py
GYHHAHA Aug 6, 2022
35a8494
fix format
GYHHAHA Aug 6, 2022
8135fea
Update _normalize.py
GYHHAHA Aug 6, 2022
fccfbba
Merge branch 'main' into patch-1
GYHHAHA Aug 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.5.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1047,6 +1047,7 @@ I/O
- Bug in :func:`read_parquet` with ``use_nullable_dtypes=True`` where ``float64`` dtype was returned instead of nullable ``Float64`` dtype (:issue:`45694`)
- Bug in :meth:`DataFrame.to_json` where ``PeriodDtype`` would not make the serialization roundtrip when read back with :meth:`read_json` (:issue:`44720`)
- Bug in :func:`read_xml` when reading XML files with Chinese character tags and would raise ``XMLSyntaxError`` (:issue:`47902`)
- Bug in :func:`json_normalize` raised boardcasting error with list-like metadata (:issue:`37782`, :issue:`47182`)

Period
^^^^^^
Expand Down
11 changes: 10 additions & 1 deletion pandas/io/json/_normalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
)
from pandas.util._decorators import deprecate

from pandas.core.dtypes.common import is_list_like

import pandas as pd
from pandas import DataFrame

Expand Down Expand Up @@ -531,7 +533,14 @@ def _recursive_extract(data, path, seen_meta, level=0):
raise ValueError(
f"Conflicting metadata name {k}, need distinguishing prefix "
)
result[k] = np.array(v, dtype=object).repeat(lengths)
if v and is_list_like(v[0]):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an alternative can we not just do something like:

arr = np.empty(1, dtype=object)
arr[0] = v
arr = arr.repeat(lengths)

Less than ideal but I think still gets us to the same place? The branching now is a little tough to follow

Copy link
Contributor Author

@GYHHAHA GYHHAHA Aug 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mroeschke suggested not to construct the np array for nested data, what I wrote before is

result[k] = np.array(v, dtype=object).repeat(lengths, axis=0).tolist()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @WillAyd's suggestion is good if it works. My main issue before was one path calling tolist() and the other returning an np.ndarray

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Late response. Unfortunately this won't work. When v is a list, arr will be [list(...)], which raises boardcasting error. Classified discussions may still be necessary. (Now both paths return list for consistency and auto-type-infering.) @mroeschke

out = []
for item, repeat in zip(v, lengths):
for _ in range(repeat):
out.append(item)
else:
out = np.array(v, dtype=object).repeat(lengths).tolist()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here dtype=object is necessary since np.array(["a", np.nan]) will result in the missing value is converted to string format "nan".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the tolist call needed as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary, but auto-type-infering won't be triggered if drop this. I think type infer is useful here. If we still want keep the object type, I can change. Suggestion?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping this line could avoid dtype=object and tolist such that this line doesn't trigger type inference and would be more performant. Otherwise since both if/else blocks convert to list are essentially similar.

Copy link
Contributor Author

@GYHHAHA GYHHAHA Aug 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about the following solution? We only point out the object dtype when this is a string np.array and do the tolist() when nested array detected. For the numeric dtype, we can still keep performant and handle both two corner cases.

if v and isinstance(v[0], str):
    out = np.array(v, dtype=object)
else:
    out = np.array(v)

out = out.repeat(lengths, axis=0)
if v and is_list_like(v[0]):
    out = out.tolist()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually looking back at the original line, it appears dtype=object was originally here and allowing dtype inference all the time might actually lead to more desired behavior (more specific types).

Would it make sense just to combine both branches then and always return a list (without using np.array)? Would be good to also to explore if there's any performance hit

result[k] = out
return result


Expand Down
35 changes: 31 additions & 4 deletions pandas/tests/io/json/test_normalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -546,7 +546,7 @@ def test_meta_non_iterable(self):

result = json_normalize(json.loads(data), record_path=["data"], meta=["id"])
expected = DataFrame(
{"one": [1], "two": [2], "id": np.array([99], dtype=object)}
{"one": [1], "two": [2], "id": np.array([99], dtype="int64")}
)
tm.assert_frame_equal(result, expected)

Expand Down Expand Up @@ -640,9 +640,7 @@ def test_missing_nested_meta(self):
)
ex_data = [[1, "foo", np.nan], [2, "foo", np.nan]]
columns = ["rec", "meta", "nested_meta.leaf"]
expected = DataFrame(ex_data, columns=columns).astype(
{"nested_meta.leaf": object}
)
expected = DataFrame(ex_data, columns=columns)
tm.assert_frame_equal(result, expected)

# If errors="raise" and nested metadata is null, we should raise with the
Expand Down Expand Up @@ -891,3 +889,32 @@ def test_series_non_zero_index(self):
}
)
tm.assert_frame_equal(result, expected)

def test_list_type_meta_data(self):
# GH 37782
data = {"values": [1, 2, 3], "metadata": {"listdata": [1, 2]}}
result = json_normalize(
data=data,
record_path=["values"],
meta=[["metadata", "listdata"]],
)
expected = DataFrame(
{
0: [1, 2, 3],
"metadata.listdata": [[1, 2], [1, 2], [1, 2]],
}
)
tm.assert_frame_equal(result, expected)

def test_empty_list_data(self):
# GH 47182
data = [
{"id": 1, "path": [{"a": 3, "b": 4}], "emptyList": []},
{"id": 2, "path": [{"a": 5, "b": 6}], "emptyList": []},
]
result = json_normalize(data, "path", ["id", "emptyList"])
expected = DataFrame(
[[3, 4, 1, []], [5, 6, 2, []]],
columns=["a", "b", "id", "emptyList"],
)
tm.assert_frame_equal(result, expected)