[FEA] Upgrade Arrow support to 4.0.0

[mughetto]

Hi,

we have developped internally a python package that uses pandas to read parquet files with the pyarrow engine. After some testing a few months ago it appeared that using pyarrow 2.0 was much faster than 1.0 for large files so we decided to enforce pyarrow>=2.0 in our requirements.txt

But now that we want to use this package in a RAPIDS environment (say 0.17 + our local pakcage install with pyarrow 2.0) we have noticed that import cudf was failing with this trace:

  File "<stdin>", line 1, in <module>
  File "/home/user/.conda/envs/rapids-0.17/lib/python3.8/site-packages/cudf/__init__.py", line 11, in <module>
    from cudf import core, datasets, testing
  File "/home/user/.conda/envs/rapids-0.17/lib/python3.8/site-packages/cudf/core/__init__.py", line 3, in <module>
    from cudf.core import buffer, column, common
  File "/home/user/.conda/envs/rapids-0.17/lib/python3.8/site-packages/cudf/core/column/__init__.py", line 3, in <module>
    from cudf.core.column.categorical import CategoricalColumn
  File "/home/user/.conda/envs/rapids-0.17/lib/python3.8/site-packages/cudf/core/column/categorical.py", line 8, in <module>
    from cudf import _lib as libcudf
  File "/home/user/.conda/envs/rapids-0.17/lib/python3.8/site-packages/cudf/_lib/__init__.py", line 4, in <module>
    from . import (
  File "cudf/_lib/gpuarrow.pyx", line 1, in init cudf._lib.gpuarrow
AttributeError: module 'pyarrow.lib' has no attribute '_CRecordBatchReader' 

Bumping back to pyarrow 1.0.1 solved the issue but represents a loss of performance for us on the pure pandas side.

Are we missing something obvious that would allow us to get pyarrow 2.0 to work with cudf? If not, are there any plan to make things compatible with more recent versions of pyarrow (3.0 as of yesterday)

Please let me know if you need details more about our environments.

Thanks a lot !

[lmeyerov]

Also relevant: arrow format version support vs. lib version dependency - see http://arrow.apache.org/docs/format/Versioning.html . I didn't find a clear breakdown on their sites for tracking these.

[kkraus14]

Hey @mughetto we're currently evaluating our options here. While there's people wanting newer versions of Arrow, others need to continue using Arrow 1.x, so ideally we'd support 1.0.1+, but that's not 100% straightforward to do. We're currently looking into this.

[lmeyerov]

To Keith's point: More important to us than the particular decision is clarity of rapids release timing in changing arrow formats, and ideally, a sense of what the expected impact area would be

[mughetto]

Also relevant: arrow format version support vs. lib version dependency - see http://arrow.apache.org/docs/format/Versioning.html . I didn't find a clear breakdown on their sites for tracking these.

Yeah the chain of dependencies and who fetches what where is a bit hard to track down at the moment :/

[mughetto]

@kkraus14 Ok thanks a lot for the quick answer!

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[kkraus14]

Pandas 1.2 support was added in #7375

[kkraus14]

This was attempted in #7495 but it was found there was blocking bugs in both Arrow 2.0.0 and 3.0.0 that prevented upgrading. These should be fixed in 4.0.0 where we'll try to upgrade.

[bdice]

@kkraus14 @galipremsagar Just came across this issue while using cuDF -- I have some other requirements that rely on newer Arrow versions. Could you provide any details about the blocking bugs you encountered in Arrow 2 / 3? I didn't see anything obvious in the comments or gpuCI build logs of #7495.

[kkraus14]

@kkraus14 @galipremsagar Just came across this issue while using cuDF -- I have some other requirements that rely on newer Arrow versions. Could you provide any details about the blocking bugs you encountered in Arrow 2 / 3? I didn't see anything obvious in the comments or gpuCI build logs of #7495.

It was discussed.on the Arrow mailing list, but on Arrow 3.0.0, you can't create Arrow Arrays or Arrow Tables from GPU backed Buffer objects. In Arrow 2.0.0 there was a bug that prevents round tripping lists of structs columns in the Parquet Reader/Writer.

We believe all of the current known issues on our side are fixed in the current tip of Arrow and the 4.0.0 release is scheduled for April which would give us plenty of time to upgrade in our 0.20 release.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.