patch: pandas-like and pyarrow scalar reduction fix #716
Conversation
| lengths = [len(s) for s in series] | ||
| max_length = max(lengths) | ||
|
|
||
| idx = series[lengths.index(max_length)]._native_series.index |
There was a problem hiding this comment.
This is the first index that has max length, hence the left-most non scalar index will still guide the resulting index
| expected = {"min": [1, 1, 1], "max": [6, 6, 6], "a": [2, 2, 2], "b": [5, 5, 5]} | ||
| compare_dicts(result, expected) | ||
|
|
||
| df = nw.from_native(constructor(data)) |
There was a problem hiding this comment.
I am recreating the dataframe each time because modin was resulting is a weird weird bug, or at least I think so.
I will try to re-factor later on
narwhals/_arrow/utils.py
Outdated
| pa.array( | ||
| np.full(shape=max_length, fill_value=value, dtype=np_dtype), | ||
| type=pa_dtype, |
There was a problem hiding this comment.
Using pa.array directly because of pyarrow-numpy integration docs
| def test_scalar_reduction_select( | ||
| request: Any, constructor: Any, expr: list[Any], expected: dict[str, list[Any]] | ||
| ) -> None: | ||
| if "dask" in str(constructor) and int(request.node.callspec.id[-1]) != 1: |
There was a problem hiding this comment.
Don't panic (just yet): this friendly request.node.callspec.id will look like the following: <constructor_name>-<id>. The id is specified in @pytest.mark.parametrize and ranges from 0 to 4 (number of test cases).
As dask passes the second test which corresponds to request.node.callspec.id = "dask_lazy_constructor-1", I am extracting the id value
| if hasattr(value, "as_py"): # pragma: no cover | ||
| value = value.as_py() |
There was a problem hiding this comment.
does this need doing everywhere or just for self._backend_version < (13,)?
There was a problem hiding this comment.
Need to check, numpy was certainly failing to create an array with a value of arrow scalar
What type of PR is this? (check all applicable)
Related issues
DaskExprreduction to scalar can't beselect-ed #694, as mentioned in comment there are cases in which the issue in 694 doesn't match polars behavior for pandas and pyarrow as well.Compared to dask, this is easier to fix as we know series length due to eagerness
Checklist
If you have comments or can explain your changes, please do so below.