PERF: ArrowExtensionArray.to_numpy(dtype=object)

[lukemanley]

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v2.0.0.rst file if fixing a bug or adding a new feature.

Perf improvement for ArrowExtensionArray.to_numpy(dtype=object).

(dtype=object is not uncommon, see discussion in #22791)

Some examples where it has an impact:

import pandas as pd
import numpy as np

data = np.random.randn(1_000_000, 10)
df = pd.DataFrame(data, dtype="float64[pyarrow]")

%timeit df.sum(axis=1)

# 4.04 s ± 53.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <- main
# 796 ms ± 22.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <- PR

%timeit df.clip(0.0, 0.1)

# 4.82 s ± 159 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   <- main
# 1.51 s ± 10.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <- PR

[jbrockmendel]

dumb question: why isn't self._data.to_numpy() good enough here?

[lukemanley]

dumb question: why isn't self._data.to_numpy() good enough here?

I believe there is a desire to maintain the raw dtype (even if wrapped in an object array) and use pd.NA by default.

For instance, this converts int to float when nulls are present:

In [1]: import pandas as pd

In [2]: arr = pd.array([1, None], dtype="int64[pyarrow]")

In [3]: arr._data.to_numpy()
Out[3]: array([ 1., nan])

I'm not sure its required, but the current implementation aligns mostly with BaseMaskedArray:

In [4]: arr = pd.array([1, None], dtype="Int64")

In [5]: arr.to_numpy()
Out[5]: array([1, <NA>], dtype=object)

In [6]: arr = pd.array([1, None], dtype="int64[pyarrow]")

In [7]: arr.to_numpy()
Out[7]: array([1, <NA>], dtype=object)

[mroeschke]

Thanks @lukemanley

[phofl]

Not totally sure, but looks like this is failing the ci on main. Any idea why?

[mroeschke]

Ah darn. Should we revert this in the meantime?

[phofl]

Probably a good idea, if we don't know how to fix

[mroeschke]

Yeah locally reverting this change fixes the test failure