[CI][Python] Tests with pandas nightlies have been failing for the last days witn NotImplementedError on S3 dtype

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

The following nightly job with the pandas nightlies have been failing for the last couple of weeks:

• test-conda-python-3.10-pandas-nightly

There seems to be an issue with S3:

 >       raise NotImplementedError(dtype)
E       NotImplementedError: |S3

opt/conda/envs/arrow/lib/python3.10/site-packages/pandas/core/indexes/base.py:634: NotImplementedError

Component(s)

Continuous Integration, Python

[raulcd]

cc @AlenkaF @jorisvandenbossche I am unsure if this was opened before but I haven't been able to find a related issue.

[AlenkaF]

This is a pandas regression known from version 2.0.0:

• pandas issue: REGR: Index.astype(<numpy string dtype>) started failing pandas-dev/pandas#50127
• Arrow PR that fixed it for older versions: https://github.com/apache/arrow/pull/15048/files#diff-a90dace803514318979968ba54f0c16c0439233e2261c13beb89d6e80610bb2b

arrow/python/pyarrow/tests/test_pandas.py

Lines 3098 to 3101 in de3130e

	if Version("2.0.0") <= Version(pd.__version__) < Version("2.2.0"):
	# TODO: regression in pandas, hopefully fixed in next version
	# https://issues.apache.org/jira/browse/ARROW-18394
	# https://github.com/pandas-dev/pandas/issues/50127

Will update the Version check (from 2.2.0 to 2.3.0 as the CI uses 2.3.0.dev)

[AlenkaF]

Numpy string dtype error is fixed with #39498, but there are still some issues:

• two TestConvertMetadata tests are failing with DeprecationWarning: make_block is deprecated and will be removed in a future version. Use public APIs instead.
• test_backwards_compatible_column_metadata_handling with AssertionError
• test_categories_with_string_pyarrow_dtype with AssertionError

I think there was an open issue for the deprecation warning, but am not sure. Will look for it now.

[AlenkaF]

Didn't find anything. Will look for the issue and a fix.

[jorisvandenbossche]

The deprecation warning can be ignored for now (that's still being discussed on the pandas side), although we could update the test to not fail for it so our CI can be green otherwise.

The other two items seem things we should investigate further.

[AlenkaF]

👍
I am currently investigating two remaining items and will change the test with deprecation warning to not fail.

[AlenkaF]

The error in the test_categories_with_string_pyarrow_dtype is due to a difference in the PyArrow array data type when being converted from pandas string[pyarrow]. For pandas version 2.1.4 I get:

>>> df1 = pd.DataFrame({"x": ["foo", "bar", "foo"]}, dtype="string[pyarrow]")
>>> df1 = df1.astype("category")

>>> df2 = pd.DataFrame({"x": ["foo", "bar", "foo"]})
>>> df2 = df2.astype("category")

>>> pa.array(df1["x"]).type
DictionaryType(dictionary<values=string, indices=int8, ordered=0>)
>>> pa.array(df2["x"]).type
DictionaryType(dictionary<values=string, indices=int8, ordered=0>)

and for the dev version I get:

>>> pa.array(df1["x"]).type
DictionaryType(dictionary<values=large_string, indices=int8, ordered=0>)
>>> pa.array(df2["x"]).type
DictionaryType(dictionary<values=string, indices=int8, ordered=0>)

meaning arrow string dtype gets converted to a large string with pandas dev hence raising an error. The PR that caused the change on the pandas side: pandas-dev/pandas#56220

Will update the test to reflect the change.

[jorisvandenbossche]

For the Parquet failure (old metadata), this seems a regression on the pandas side in how tz-aware data gets converted from arrow to pandas -> pandas-dev/pandas#56775

[raulcd]

@AlenkaF @jorisvandenbossche , do you think this is a blocker?

[jorisvandenbossche]

For CI purposes, I think it would be good to have the currently open PR in the release. But nothing in that PR is critical, it's only test changes.