fix: scalar reductions on empty inputs

[camriddell]

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Related issue [Bug]: Summing empty Pandas DataFrame #1711
• Closes [Bug]: Summing empty Pandas DataFrame #1711

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

If you have comments or can explain your changes, please do so below

When performing a scalar reduction on an empty DataFrame, Polars, pandas, and PyArrow generated disparate result sets with default options. For example, when computing the sum of an empty column (via Narwhals)

• Polars returns a value of 0
• pandas raises a ValueError (on reconstruction of the Series, computationally it also produces 0 as the scalar)
• PyArrow returns null (default scalarreduction options requires at least 1 observed value to produce observed result)

pandas & PyArrow backends now produce output that is consistent with Polars.

Furthermore, there is a difference in the result set when using .select vs .with_columns where the former may return a value even if the the input was empty whereas the latter will return an empty DataFrame since its input was empty. Both Polars and PyArrow backends exhibited this behavior so pandas does the same as well now.

---

See the following example for how the behavior of .sum was changed.

import narwhals as nw
import pandas as pd
import polars as pl
import pyarrow as pa
from itertools import product

@nw.narwhalify
def nw_select(df):
    return (
        df
        .filter(nw.col("name") == "Boo")
        .select(nw.col("value").sum())
    )

@nw.narwhalify
def nw_with_columns(df):
    return (
        df
        .filter(nw.col("name") == "Boo")
        .with_columns(res=nw.col("value").sum())
    )

pl_df = pl.DataFrame(
    {
        "name": ["John", "Doe", "Jane"],
        "value": [10, 5, 20],
    },
)
pd_df = pl_df.to_pandas()
pa_df = pa.Table.from_pandas(pd_df)


for func, df in product([nw_select, nw_with_columns], [pl_df, pd_df, pa_df]):
    print(f' {func.__name__} → {(type(df).__module__.split(".")[0])} '.center(50, '\N{box drawings light horizontal}'))
    try:
        print(func(df))
    except ValueError as e:
        print(e)
    print()

Old behavior

• pandas errors when reconstructing a series from a scalar (index is empty, but passed data is not)
• pyarrow returns null where Polars returns a value

─────────────── nw_select → polars ───────────────
shape: (1, 1)
┌───────┐
│ value │
│ ---   │
│ i64   │
╞═══════╡
│ 0     │
└───────┘

─────────────── nw_select → pandas ───────────────
Length of values (1) does not match length of index (0)

────────────── nw_select → pyarrow ───────────────
pyarrow.Table
value: int64
----
value: [[null]]

──────────── nw_with_columns → polars ────────────
shape: (0, 3)
┌──────┬───────┬─────┐
│ name ┆ value ┆ res │
│ ---  ┆ ---   ┆ --- │
│ str  ┆ i64   ┆ i64 │
╞══════╪═══════╪═════╡
└──────┴───────┴─────┘

──────────── nw_with_columns → pandas ────────────
Length of values (1) does not match length of index (0)

─────────── nw_with_columns → pyarrow ────────────
pyarrow.Table
name: string
value: int64
res: null
----
name: []
value: []
res: [0 nulls]

New behavior

• select operations will produce a consistent scalar output even if the input was empty
  • all → True
  • any → False
  • sum → 0
  • max → Null or NaN if non-nullable
  • min → Null or NaN if non-nullable
  • mean → Null or NaN if non-nullable
• with_columns returns empty if the input was empty

─────────────── nw_select → polars ───────────────
shape: (1, 1)
┌───────┐
│ value │
│ ---   │
│ i64   │
╞═══════╡
│ 0     │
└───────┘

─────────────── nw_select → pandas ───────────────
   value
0      0

────────────── nw_select → pyarrow ───────────────
pyarrow.Table
value: int64
----
value: [[0]]

──────────── nw_with_columns → polars ────────────
shape: (0, 3)
┌──────┬───────┬─────┐
│ name ┆ value ┆ res │
│ ---  ┆ ---   ┆ --- │
│ str  ┆ i64   ┆ i64 │
╞══════╪═══════╪═════╡
└──────┴───────┴─────┘

──────────── nw_with_columns → pandas ────────────
Empty DataFrame
Columns: [name, value, res]
Index: []

─────────── nw_with_columns → pyarrow ────────────
pyarrow.Table
name: string
value: int64
res: null
----
name: []
value: []
res: [0 nulls]

[FBruzzesi]

Nice one! Thanks for the fix @camriddell 🚀

[MarcoGorelli]

thanks @camriddell and @FBruzzesi for review!