[REVIEW] Upgrade arrow & pyarrow to 6.0.1

[galipremsagar]

Resolves: #9645
This PR upgrades arrow & pyarrow to 6.0.1 from 5.0.0.

[galipremsagar]

rerun tests

[beckernick]

Is there any risk to us using the constraint >= arrow/pyarrow 5 rather than requiring 6? This would likely make it much easier to interact with tools that are slower to upgrade.

[kkraus14]

Is there any risk to us using the constraint >= arrow/pyarrow 5 rather than requiring 6? This would likely make it much easier to interact with tools that are slower to upgrade.

We can't do that. We link against the libarrow shared library in libcudf, and libarrow 5.0.0 for example sets its SONAME as libarrow.so.500. This adds a NEEDED entry into libcudf of libarrow.so.500. If libarrow 6.0.0 is installed, the linker will fail to find libarrow.so.500 and libcudf will give a missing symbols crash.

[galipremsagar]

rerun tests

[galipremsagar]

rerun tests

[nirandaperera]

Arrow has new 6.0.1 bug fix release. Could you please use that instead of 6.0.0? I believe it would be a trivial change.

[robertmaynard]

CMake changes LGTM

[kkraus14]

Python / CMake / Conda changes all LGTM, thanks @galipremsagar!

[rgsl888prabhu]

Looks good, just need to update year in copyright

[galipremsagar]

Thanks for the reviews, everyone! @robertmaynard @rgsl888prabhu @bdice

Thanks @kkraus14 for the help with runtime gxx & cxx libraries.
Thanks @Ethyling for publishing arm packages and @ajschmidt8 for helping with conda issues in CI.
Thanks @jakirkham for the help in resolving pyorc packaging issues.

[galipremsagar]

rerun tests

[galipremsagar]

rerun tests

[jjacobelli]

rerun tests

[davidwendt]

@galipremsagar Not sure you are aware of the build error for QUANTILES_TEST

13:24:38 FAILED: gtests/QUANTILES_TEST 
13:24:38 : && /usr/local/gcc9/bin/g++ -O3 -DNDEBUG  tests/CMakeFiles/QUANTILES_TEST.dir/quantiles/percentile_approx_test.cu.o tests/CMakeFiles/QUANTILES_TEST.dir/quantiles/quantile_test.cpp.o tests/CMakeFiles/QUANTILES_TEST.dir/quantiles/quantiles_test.cpp.o -o gtests/QUANTILES_TEST -L/usr/local/cuda/targets/x86_64-linux/lib/stubs   -L/usr/local/cuda/targets/x86_64-linux/lib -Wl,-rpath,$PREFIX/lib:/opt/conda/envs/rapids/lib:$SRC_DIR/cpp/build:  libcudftestutil.a  /opt/conda/envs/rapids/lib/libgmock_main.so  /opt/conda/envs/rapids/lib/libgtest_main.so  libcudf.so  $PREFIX/lib/libarrow.so.600.1.0  $PREFIX/lib/libarrow_cuda.so.600.1.0  -ldl  $PREFIX/lib/libcudart.so  /usr/lib64/libcuda.so  /opt/conda/envs/rapids/lib/libgmock.so  /opt/conda/envs/rapids/lib/libgtest.so  -pthread  -lcudadevrt  -lcudart_static  -lrt  -lpthread  -ldl && :
13:24:38 tests/CMakeFiles/QUANTILES_TEST.dir/quantiles/percentile_approx_test.cu.o: In function `arrow_percentile_approx(cudf::column_view const&, int, std::vector<double, std::allocator<double> > const&)':
13:24:38 tmpxft_00002e7a_00000000-6_percentile_approx_test.compute_86.cudafe1.cpp:(.text+0x1284): undefined reference to `arrow::internal::TDigest::Quantile(double)'
13:24:38 tmpxft_00002e7a_00000000-6_percentile_approx_test.compute_86.cudafe1.cpp:(.text+0x1488): undefined reference to `arrow::internal::TDigest::MergeInput()'
13:24:38 collect2: error: ld returned 1 exit status

Looks like you will need to update this test code before the build will work?
I apologize if this is already known.

[galipremsagar]

@galipremsagar Not sure you are aware of the build error for QUANTILES_TEST

13:24:38 FAILED: gtests/QUANTILES_TEST 
13:24:38 : && /usr/local/gcc9/bin/g++ -O3 -DNDEBUG  tests/CMakeFiles/QUANTILES_TEST.dir/quantiles/percentile_approx_test.cu.o tests/CMakeFiles/QUANTILES_TEST.dir/quantiles/quantile_test.cpp.o tests/CMakeFiles/QUANTILES_TEST.dir/quantiles/quantiles_test.cpp.o -o gtests/QUANTILES_TEST -L/usr/local/cuda/targets/x86_64-linux/lib/stubs   -L/usr/local/cuda/targets/x86_64-linux/lib -Wl,-rpath,$PREFIX/lib:/opt/conda/envs/rapids/lib:$SRC_DIR/cpp/build:  libcudftestutil.a  /opt/conda/envs/rapids/lib/libgmock_main.so  /opt/conda/envs/rapids/lib/libgtest_main.so  libcudf.so  $PREFIX/lib/libarrow.so.600.1.0  $PREFIX/lib/libarrow_cuda.so.600.1.0  -ldl  $PREFIX/lib/libcudart.so  /usr/lib64/libcuda.so  /opt/conda/envs/rapids/lib/libgmock.so  /opt/conda/envs/rapids/lib/libgtest.so  -pthread  -lcudadevrt  -lcudart_static  -lrt  -lpthread  -ldl && :
13:24:38 tests/CMakeFiles/QUANTILES_TEST.dir/quantiles/percentile_approx_test.cu.o: In function `arrow_percentile_approx(cudf::column_view const&, int, std::vector<double, std::allocator<double> > const&)':
13:24:38 tmpxft_00002e7a_00000000-6_percentile_approx_test.compute_86.cudafe1.cpp:(.text+0x1284): undefined reference to `arrow::internal::TDigest::Quantile(double)'
13:24:38 tmpxft_00002e7a_00000000-6_percentile_approx_test.compute_86.cudafe1.cpp:(.text+0x1488): undefined reference to `arrow::internal::TDigest::MergeInput()'
13:24:38 collect2: error: ld returned 1 exit status

Looks like you will need to update this test code before the build will work? I apologize if this is already known.

Have seen this undefined symbol issue, We don't see this error locally. It appears to be happening in CI if we try to upgrade arrow to 6.0 from 5.0 in ci, if we force remove 5.0 arrow and install 6.0 we don't get this problem. So @Ethyling upgraded our build images too with arrow 6.0 and reran the tests.

[galipremsagar]

@gpucibot merge