Limit runtime dependency to libarrow>=16.0.0,<16.1.0a0

[pentschev]

Fix libarrow runtime dependency which is currently broken due to the release of libarrow=16.1.0:

$ python -c "import cudf"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/rapids/lib/python3.10/site-packages/cudf/__init__.py", line 9, in <module>
    _setup_numba()
  File "/opt/conda/envs/rapids/lib/python3.10/site-packages/cudf/utils/_numba.py", line 124, in _setup_numba
    _get_cc_60_ptx_file()
  File "/opt/conda/envs/rapids/lib/python3.10/site-packages/cudf/utils/_numba.py", line 16, in _get_cc_60_ptx_file
    from cudf._lib import strings_udf
  File "/opt/conda/envs/rapids/lib/python3.10/site-packages/cudf/_lib/__init__.py", line 4, in <module>
    from . import (
ImportError: libarrow.so.1600: cannot open shared object file: No such file or directory

[bdice]

Run-exports were recently fixed on conda-forge and should save us here. This was done less than 12 hours ago so a rebuild of nightlies across RAPIDS should fix it.

Fix for 16.1: conda-forge/arrow-cpp-feedstock#1409

Backport of this fix to 16.0: conda-forge/conda-forge-repodata-patches-feedstock@6155ba0

[pentschev]

That doesn’t seem to do the trick, build is picking up the latest release:

libarrow                  16.1.0           hbcc2d42_2_cpu    conda-forge
libarrow-acero            16.1.0           hac33072_2_cpu    conda-forge
libarrow-dataset          16.1.0           hac33072_2_cpu    conda-forge
libarrow-substrait        16.1.0           h7e0c224_2_cpu    conda-forge

still fails:

ImportError: libarrow.so.1600: cannot open shared object file: No such file or directory

Latest build is 2 from 10h ago, whereas 1 was 13h ago, as seen in https://anaconda.org/conda-forge/libarrow/files .

[jameslamb]

Summarizing some conversations from other places... in short, the root cause for the error reported in this PR's description is the mix of the following:

*libcudf conda package builds against libarrow supplied via conda-forge...it gets that libarrow's run_exports (conda docs)

• at build time, libcudf is linking against `libarrow=16.0.0.*
• with libarrow=16.0.0's current run_exports, libcudf picks up a runtime dependency like libarrow >=16.0.0,<17.0a0
• that is too broad... libarrow=16.1 is incompatible with libarrow=16.0 (Fix run exports to restrict the minor version conda-forge/arrow-cpp-feedstock#1409)

The ideal fix for this is to update the run exports for conda-forge's libarrow=16.0 packages. That's documented at conda-forge/arrow-cpp-feedstock#1418.

To hopefully unblock cudf and other projects using it in the interim, I just pushed a commit (ff82c0e) that does the following:

• ignore libarrow's run_exports when building libcudf
• manually specify the tighter runtime pins we want in libcudf's run dependencies

[jameslamb]

It looks to me like this fixes the libarrow issue... I see libarrow=16.0 being pulled in in both conda builds (conda-cpp-build job) and tests (conda-cpp-tests job).

But the Python tests appear to still be failing with the other, Dask-related issue @rjzamora is working on in #15788.

x......[gw2] node down: Not properly terminated
F
replacing crashed worker gw2
Fatal Python error: Segmentation fault

(wheel-tests job)

So @vyasr @bdice @pentschev @rjzamora I think we'll either need to admin merge this or decide on one PR (#15788 or this one) to fix both issues at the same time.

[jakirkham]

Is the libarrow pin sufficient? The rest are pinned exactly to it (for example)

Suggested change

	# TODO: start taking libarrow's run exports again wwhen they're correct for 16.0
	# ref: https://github.com/conda-forge/arrow-cpp-feedstock/issues/1418
	- libarrow>=16.0.0,<16.1.0a0
	- libarrow-acero>=16.0.0,<16.1.0a0
	- libarrow-dataset>=16.0.0,<16.1.0a0
	- libparquet>=16.0.0,<16.1.0a0
	# TODO: start taking libarrow's run exports again wwhen they're correct for 16.0
	# ref: https://github.com/conda-forge/arrow-cpp-feedstock/issues/1418
	- libarrow>=16.0.0,<16.1.0a0

[jameslamb]

You're probably right that it would be enough. I went for all of them just mimicking what I already saw in dependencies.yaml here for building the test environment.

Alright if we still merge this, and then I push a follow-up PR trying to pare this back down to just libarrow? That way we wouldn't have to wait for another CI cycle to get the fix onto branch-24.06.

[vyasr]

I don't think it's worth changing at this point. Once the upstream arrow packages are updated we'll just be dropping this altogether anyway.

[jameslamb]

Ok works for me, I won't touch this unless @jakirkham disagrees and thinks we should pursue it.