Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: json_normalize raises boardcasting error with list-like metadata #47708

Closed
wants to merge 22 commits into from
Closed

BUG: json_normalize raises boardcasting error with list-like metadata #47708

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
039a659
fix repeat axis
GYHHAHA Jul 13, 2022
1645550
add tests
GYHHAHA Jul 13, 2022
6db3e6c
Update v1.5.0.rst
GYHHAHA Jul 13, 2022
49f2950
change tests dtype descriptions
GYHHAHA Jul 13, 2022
b73d0f1
Merge branch 'main' into patch-1
GYHHAHA Jul 19, 2022
8eb1421
drop np array
GYHHAHA Jul 27, 2022
de21683
Update _normalize.py
GYHHAHA Jul 28, 2022
5a3d8bd
Update test_normalize.py
GYHHAHA Jul 28, 2022
a4ac5ec
Merge branch 'pandas-dev:main' into patch-1
GYHHAHA Jul 28, 2022
cc00c3e
fix dtype
GYHHAHA Jul 28, 2022
7efe709
Update pandas/io/json/_normalize.py
GYHHAHA Jul 30, 2022
9ad3db4
type ignore
GYHHAHA Jul 30, 2022
23c090e
fix format
GYHHAHA Jul 30, 2022
b3d500f
remove object type
GYHHAHA Jul 30, 2022
2779bc2
remove astype
GYHHAHA Jul 30, 2022
6252821
fix format
GYHHAHA Jul 30, 2022
34a3ace
import is_list_like
GYHHAHA Jul 30, 2022
f2fea4e
Update _normalize.py
GYHHAHA Aug 6, 2022
06d9f34
Update test_normalize.py
GYHHAHA Aug 6, 2022
35a8494
fix format
GYHHAHA Aug 6, 2022
8135fea
Update _normalize.py
GYHHAHA Aug 6, 2022
fccfbba
Merge branch 'main' into patch-1
GYHHAHA Aug 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.5.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -959,6 +959,7 @@ I/O
- Bug in :func:`read_sas` with RLE-compressed SAS7BDAT files that contain 0x00 control bytes (:issue:`47099`)
- Bug in :func:`read_parquet` with ``use_nullable_dtypes=True`` where ``float64`` dtype was returned instead of nullable ``Float64`` dtype (:issue:`45694`)
- Bug in :meth:`DataFrame.to_json` where ``PeriodDtype`` would not make the serialization roundtrip when read back with :meth:`read_json` (:issue:`44720`)
- Bug in :func:`json_normalize` raised boardcasting error with list-like metadata (:issue:`37782`, :issue:`47182`)

Period
^^^^^^
Expand Down
10 changes: 9 additions & 1 deletion pandas/io/json/_normalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
Scalar,
)
from pandas.util._decorators import deprecate
from pandas.core.dtypes.common import is_list_like

import pandas as pd
from pandas import DataFrame
Expand Down Expand Up @@ -531,7 +532,14 @@ def _recursive_extract(data, path, seen_meta, level=0):
raise ValueError(
f"Conflicting metadata name {k}, need distinguishing prefix "
)
result[k] = np.array(v, dtype=object).repeat(lengths)
if v and is_list_like(v[0]):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an alternative can we not just do something like:

arr = np.empty(1, dtype=object)
arr[0] = v
arr = arr.repeat(lengths)

Less than ideal but I think still gets us to the same place? The branching now is a little tough to follow

Copy link
Contributor Author

@GYHHAHA GYHHAHA Aug 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mroeschke suggested not to construct the np array for nested data, what I wrote before is

result[k] = np.array(v, dtype=object).repeat(lengths, axis=0).tolist()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @WillAyd's suggestion is good if it works. My main issue before was one path calling tolist() and the other returning an np.ndarray

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Late response. Unfortunately this won't work. When v is a list, arr will be [list(...)], which raises boardcasting error. Classified discussions may still be necessary. (Now both paths return list for consistency and auto-type-infering.) @mroeschke

out = []
for item, repeat in zip(v, lengths):
for _ in range(repeat):
out.append(item)
else:
out = np.array(v).repeat(lengths) # type: ignore[assignment]
result[k] = out
return result


Expand Down
37 changes: 31 additions & 6 deletions pandas/tests/io/json/test_normalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -545,9 +545,7 @@ def test_meta_non_iterable(self):
data = """[{"id": 99, "data": [{"one": 1, "two": 2}]}]"""

result = json_normalize(json.loads(data), record_path=["data"], meta=["id"])
expected = DataFrame(
{"one": [1], "two": [2], "id": np.array([99], dtype=object)}
)
expected = DataFrame({"one": [1], "two": [2], "id": np.array([99])})
tm.assert_frame_equal(result, expected)

def test_generator(self, state_data):
Expand Down Expand Up @@ -640,9 +638,7 @@ def test_missing_nested_meta(self):
)
ex_data = [[1, "foo", np.nan], [2, "foo", np.nan]]
columns = ["rec", "meta", "nested_meta.leaf"]
expected = DataFrame(ex_data, columns=columns).astype(
{"nested_meta.leaf": object}
)
expected = DataFrame(ex_data, columns=columns)
tm.assert_frame_equal(result, expected)

# If errors="raise" and nested metadata is null, we should raise with the
Expand Down Expand Up @@ -891,3 +887,32 @@ def test_series_non_zero_index(self):
}
)
tm.assert_frame_equal(result, expected)

def test_list_type_meta_data(self):
# GH 37782
data = {"values": [1, 2, 3], "metadata": {"listdata": [1, 2]}}
result = json_normalize(
data=data,
record_path=["values"],
meta=[["metadata", "listdata"]],
)
expected = DataFrame(
{
0: [1, 2, 3],
"metadata.listdata": [[1, 2], [1, 2], [1, 2]],
}
)
tm.assert_frame_equal(result, expected)

def test_empty_list_data(self):
# GH 47182
data = [
{"id": 1, "path": [{"a": 3, "b": 4}], "emptyList": []},
{"id": 2, "path": [{"a": 5, "b": 6}], "emptyList": []},
]
result = json_normalize(data, "path", ["id", "emptyList"])
expected = DataFrame(
[[3, 4, 1, []], [5, 6, 2, []]],
columns=["a", "b", "id", "emptyList"],
)
tm.assert_frame_equal(result, expected)