prefer wheel-provided libcudf.so in load_library(), use RTLD_LOCAL

[jameslamb]

Description

Contributes to rapidsai/build-planning#118

Modifies libcudf.load_library() in the following ways:

• prefer wheel-provided libcudf.so to system installation
• expose environment variable RAPIDS_LIBCUDF_PREFER_SYSTEM_LIBRARY for switching that preference
• load libcudf.so with RTLD_LOCAL, to prevent adding symbols to the global namespace (dlopen docs)

Notes for Reviewers

How I tested this

Locally (x86_64, CUDA 12, Python 3.12), built libcudf, pylibcudf, and cudf wheels from this branch, then libcuspatial and cuspatial from the corresponding cuspatial branch. Ran cuspatial's unit tests, and tried setting the environment variable and inspecting ld's logs to confirm that the environment variable changed the loading and search behavior.

e.g.

# clear ld cache to avoid cheating
rm -f /etc/ld.so.cache
ldconfig

# try using an env variable to say "prefer the system-installed version"
LD_DEBUG=libs \
LD_DEBUG_OUTPUT=/tmp/out.txt \
RAPIDS_LIBCUDF_PREFER_SYSTEM_LIBRARY=true \
python -c "import cuspatial; print(cuspatial.__version__)"

cat /tmp/out.txt.* > prefer-system.txt
# (then manually looked through those logs to confirm it searched places like /usr/lib64 and /lib64)

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[jameslamb]

Updated this to match the discussion on rapidsai/cuspatial#1483

• updated comment on the choice of RTLD_LOCAL (prefer wheel-provided libcuspatial.so in load_library(), use RTLD_LOCAL cuspatial#1483 (comment))
• stopped searching lib/ within the wheel, as all libs should be found in lib64/ (prefer wheel-provided libcuspatial.so in load_library(), use RTLD_LOCAL cuspatial#1483 (comment))

[vyasr]

LGTM, seems like the suggestions from rapidsai/cuspatial#1483 have all been incorporated as well.

[vyasr]

/merge