.interpolate() Method Incompatible with float[pyarrow] Dtype

[mattharrison]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

temp = pd.Series([32, 40, None, 42, 39, 32], dtype='float[pyarrow]')
temp.interpolate()

Issue Description

the .interpolate() method fails with a TypeError when applied to a Pandas Series with the dtype float[pyarrow].

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[18], line 2
      1 temp = pd.Series([32, 40, None, 42, 39, 32], dtype='float[pyarrow]')
----> 2 temp.interpolate()

File ~/.envs/menv/lib/python3.10/site-packages/pandas/core/generic.py:8206, in NDFrame.interpolate(self, method, axis, limit, inplace, limit_direction, limit_area, downcast, **kwargs)
   8204 else:
   8205     index = missing.get_interp_index(method, obj.index)
-> 8206     new_data = obj._mgr.interpolate(
   8207         method=method,
   8208         index=index,
   8209         limit=limit,
   8210         limit_direction=limit_direction,
   8211         limit_area=limit_area,
   8212         inplace=inplace,
   8213         downcast=downcast,
   8214         **kwargs,
   8215     )
   8217 result = self._constructor_from_mgr(new_data, axes=new_data.axes)
   8218 if should_transpose:

File ~/.envs/menv/lib/python3.10/site-packages/pandas/core/internals/base.py:265, in DataManager.interpolate(self, inplace, **kwargs)
    264 def interpolate(self, inplace: bool, **kwargs) -> Self:
--> 265     return self.apply_with_block(
    266         "interpolate", inplace=inplace, **kwargs, using_cow=using_copy_on_write()
    267     )

File ~/.envs/menv/lib/python3.10/site-packages/pandas/core/internals/managers.py:354, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
    352         applied = b.apply(f, **kwargs)
    353     else:
--> 354         applied = getattr(b, f)(**kwargs)
    355     result_blocks = extend_blocks(applied, result_blocks)
    357 out = type(self).from_blocks(result_blocks, self.axes)

File ~/.envs/menv/lib/python3.10/site-packages/pandas/core/internals/blocks.py:1507, in Block.interpolate(self, method, index, inplace, limit, limit_direction, limit_area, downcast, using_cow, **kwargs)
   1504 copy, refs = self._get_refs_and_copy(using_cow, inplace)
   1506 # Dispatch to the EA method.
-> 1507 new_values = self.array_values.interpolate(
   1508     method=method,
   1509     axis=self.ndim - 1,
   1510     index=index,
   1511     limit=limit,
   1512     limit_direction=limit_direction,
   1513     limit_area=limit_area,
   1514     copy=copy,
   1515     **kwargs,
   1516 )
   1517 data = extract_array(new_values, extract_numpy=True)
   1519 nb = self.make_block_same_class(data, refs=refs)

TypeError: ExtensionArray.interpolate() missing 1 required keyword-only argument: 'fill_value'

Expected Behavior

That this would work like it does in Pandas 1.

Installed Versions

INSTALLED VERSIONS

commit : e86ed37
python : 3.10.13.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:23 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : en_US.UTF-8
LANG : None
LOCALE : en_US.UTF-8

pandas : 2.1.1
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.6.1
pip : 23.2.1
Cython : 3.0.4
pytest : 7.2.0
hypothesis : 6.81.2
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.8.0
pandas_datareader : None
bs4 : 4.11.1
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : 2023.3.0
gcsfs : None
matplotlib : 3.6.2
numba : 0.56.4
numexpr : 2.8.4
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.0
sqlalchemy : 2.0.21
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[mroeschke]

The incorrect error message was fixed in 2.1.2 #55347

Generally this doesn't work (not supported) as an ExtensionArray needs to implement interpolate on the array now: #53659, so will mark as an enhancement request

[mattharrison]

In Pandas 2.0.2 I could cast the series to a (numpy backed) float and then do the interpolation. Now, in 2.2rc0 I can't cast to float.

>>> temp = pd.Series([32, 40, None, 42, 39, 32],
...    dtype='float[pyarrow]')
>>> temp.astype(float)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[55], line 3
      1 temp = pd.Series([32, 40, None, 42, 39, 32],
      2    dtype='float[pyarrow]')
----> 3 temp.astype(float)

File ~/.envs/pd22rc/lib/python3.11/site-packages/pandas/core/generic.py:6637, in NDFrame.astype(self, dtype, copy, errors)
   6631     results = [
   6632         ser.astype(dtype, copy=copy, errors=errors) for _, ser in self.items()
   6633     ]
   6635 else:
   6636     # else, only a single dtype is given
-> 6637     new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
   6638     res = self._constructor_from_mgr(new_data, axes=new_data.axes)
   6639     return res.__finalize__(self, method="astype")

File ~/.envs/pd22rc/lib/python3.11/site-packages/pandas/core/internals/managers.py:431, in BaseBlockManager.astype(self, dtype, copy, errors)
    428 elif using_copy_on_write():
    429     copy = False
--> 431 return self.apply(
    432     "astype",
    433     dtype=dtype,
    434     copy=copy,
    435     errors=errors,
    436     using_cow=using_copy_on_write(),
    437 )

File ~/.envs/pd22rc/lib/python3.11/site-packages/pandas/core/internals/managers.py:364, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
    362         applied = b.apply(f, **kwargs)
    363     else:
--> 364         applied = getattr(b, f)(**kwargs)
    365     result_blocks = extend_blocks(applied, result_blocks)
    367 out = type(self).from_blocks(result_blocks, self.axes)

File ~/.envs/pd22rc/lib/python3.11/site-packages/pandas/core/internals/blocks.py:754, in Block.astype(self, dtype, copy, errors, using_cow, squeeze)
    751         raise ValueError("Can not squeeze with more than one column.")
    752     values = values[0, :]  # type: ignore[call-overload]
--> 754 new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
    756 new_values = maybe_coerce_values(new_values)
    758 refs = None

File ~/.envs/pd22rc/lib/python3.11/site-packages/pandas/core/dtypes/astype.py:237, in astype_array_safe(values, dtype, copy, errors)
    234     dtype = dtype.numpy_dtype
    236 try:
--> 237     new_values = astype_array(values, dtype, copy=copy)
    238 except (ValueError, TypeError):
    239     # e.g. _astype_nansafe can fail on object-dtype of strings
    240     #  trying to convert to float
    241     if errors == "ignore":

File ~/.envs/pd22rc/lib/python3.11/site-packages/pandas/core/dtypes/astype.py:179, in astype_array(values, dtype, copy)
    175     return values
    177 if not isinstance(values, np.ndarray):
    178     # i.e. ExtensionArray
--> 179     values = values.astype(dtype, copy=copy)
    181 else:
    182     values = _astype_nansafe(values, dtype, copy=copy)

File ~/.envs/pd22rc/lib/python3.11/site-packages/pandas/core/arrays/base.py:721, in ExtensionArray.astype(self, dtype, copy)
    717     from pandas.core.arrays import TimedeltaArray
    719     return TimedeltaArray._from_sequence(self, dtype=dtype, copy=copy)
--> 721 return np.array(self, dtype=dtype, copy=copy)

File ~/.envs/pd22rc/lib/python3.11/site-packages/pandas/core/arrays/arrow/array.py:633, in ArrowExtensionArray.__array__(self, dtype)
    631 def __array__(self, dtype: NpDtype | None = None) -> np.ndarray:
    632     """Correctly construct numpy arrays when passed to `np.asarray()`."""
--> 633     return self.to_numpy(dtype=dtype)

File ~/.envs/pd22rc/lib/python3.11/site-packages/pandas/core/arrays/arrow/array.py:1363, in ArrowExtensionArray.to_numpy(self, dtype, copy, na_value)
   1361     result = np.empty(len(data), dtype=dtype)
   1362     mask = data.isna()
-> 1363     result[mask] = na_value
   1364     result[~mask] = data[~mask]._pa_array.to_numpy()
   1365 return result

TypeError: float() argument must be a string or a real number, not 'NAType'

[mroeschke]

cc @phofl looks like to_numpy_dtype_inference is returning na_value=pd.NA for this case

[phofl]

Actually that was a different change

Will open a PR soonish