CLN: Use type_mapper instead of manual conversion

[phofl]

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[jbrockmendel]

pandas/io/parquet.py:253: error: Incompatible types in assignment (expression has type "Type[ArrowDtype]", target has type overloaded function) [assignment]

[phofl]

thx fixed

[jorisvandenbossche]

I think you should be able to move the if pandas: .. elif pyarrow: .. higher up where to_pandas_kwargs["types_mapper"] = ... is defined. And then just have a single result = pa_table.to_pandas(**to_pandas_kwargs) here

[phofl]

Avoided this on purpose, because this would chang semantics. The other if is only reached if nullable_dtypes is set as well and I wanted to wait for the other discussion before touching anything,

[mroeschke]

Good call. I think there are a few more places where we can use type_mapper

./pandas/io/orc.py:                        pa_table.column_names, pa_table.itercolumns()
./pandas/io/feather_format.py:                        pa_table.column_names, pa_table.itercolumns()
./pandas/io/parsers/arrow_parser_wrapper.py:                    for col_name, pa_col in zip(table.column_names, table.itercolumns())
./pandas/io/parquet.py:                            pa_table.column_names, pa_table.itercolumns()
./pandas/io/json/_json.py:                                    pa_table.column_names, pa_table.itercolumns()
./pandas/tests/io/test_parquet.py:                    pa_table.column_names, pa_table.itercolumns()

[phofl]

Updated

[phofl]

Merging this into 2.0 would have the benefit that we get index support as soon as pyarrow 12 is out

[mroeschke]

Merging this into 2.0 would have the benefit that we get index support as soon as pyarrow 12 is out

SGTM

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 2.0.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 3e0c1dab6bcbe95ea1349f144e6f4bc606471ae0

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #51766: CLN: Use type_mapper instead of manual conversion'

4. Push to a named branch:

git push YOURFORK 2.0.x:auto-backport-of-pr-51766-on-2.0.x

5. Create a PR against branch 2.0.x, I would have named this PR:

"Backport PR #51766 on branch 2.0.x (CLN: Use type_mapper instead of manual conversion)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.