BUG: json_normalize raises boardcasting error with list-like metadata #47708
Conversation
fix list like Co-authored-by: Matthew Roeschke <[email protected]>
| @@ -531,7 +532,14 @@ def _recursive_extract(data, path, seen_meta, level=0): | |||
| raise ValueError( | |||
| f"Conflicting metadata name {k}, need distinguishing prefix " | |||
| ) | |||
| result[k] = np.array(v, dtype=object).repeat(lengths) | |||
| if v and is_list_like(v[0]): | |||
There was a problem hiding this comment.
As an alternative can we not just do something like:
arr = np.empty(1, dtype=object)
arr[0] = v
arr = arr.repeat(lengths)Less than ideal but I think still gets us to the same place? The branching now is a little tough to follow
There was a problem hiding this comment.
@mroeschke suggested not to construct the np array for nested data, what I wrote before is
result[k] = np.array(v, dtype=object).repeat(lengths, axis=0).tolist()
There was a problem hiding this comment.
I think @WillAyd's suggestion is good if it works. My main issue before was one path calling tolist() and the other returning an np.ndarray
There was a problem hiding this comment.
Late response. Unfortunately this won't work. When v is a list, arr will be [list(...)], which raises boardcasting error. Classified discussions may still be necessary. (Now both paths return list for consistency and auto-type-infering.) @mroeschke
| for _ in range(repeat): | ||
| out.append(item) | ||
| else: | ||
| out = np.array(v, dtype=object).repeat(lengths).tolist() |
There was a problem hiding this comment.
Here dtype=object is necessary since np.array(["a", np.nan]) will result in the missing value is converted to string format "nan".
There was a problem hiding this comment.
Is the tolist call needed as well?
There was a problem hiding this comment.
Not necessary, but auto-type-infering won't be triggered if drop this. I think type infer is useful here. If we still want keep the object type, I can change. Suggestion?
There was a problem hiding this comment.
I was hoping this line could avoid dtype=object and tolist such that this line doesn't trigger type inference and would be more performant. Otherwise since both if/else blocks convert to list are essentially similar.
There was a problem hiding this comment.
How about the following solution? We only point out the object dtype when this is a string np.array and do the tolist() when nested array detected. For the numeric dtype, we can still keep performant and handle both two corner cases.
if v and isinstance(v[0], str):
out = np.array(v, dtype=object)
else:
out = np.array(v)
out = out.repeat(lengths, axis=0)
if v and is_list_like(v[0]):
out = out.tolist()There was a problem hiding this comment.
Actually looking back at the original line, it appears dtype=object was originally here and allowing dtype inference all the time might actually lead to more desired behavior (more specific types).
Would it make sense just to combine both branches then and always return a list (without using np.array)? Would be good to also to explore if there's any performance hit
|
This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this. |
|
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen. |
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.