[BUG] cuDF fails to build if -DPython_LIBRARY is not set in the cmake command

[sevagh]

Related PR: #13188

It was noticed that a recent build of cudf failed:

CMake Error at /tmp/pip-build-env-2ez3h1il/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
      Could NOT find PythonLibs (missing: PYTHON_LIBRARIES) (found version
      "3.10.9")

The difference between this run and the last successful build is that scikit-build went from 0.17.1 to 0.17.2.

The consequence of this scikit-build change resulted in the generated cmake command not having the -DPython_LIBRARY option specified.

In the successful build we can see that option set:

Configuring Project
      Working directory:
        /project/python/cudf/_skbuild/linux-aarch64-3.10/cmake-build
      Command:
        /tmp/pip-build-env-fddpxjch/overlay/lib/python3.10/site-packages/cmake/data/bin/cmake /project/python/cudf \
       -G Ninja  ... -DPYTHON_LIBRARY:PATH=libpython3.10.a ...

Interestingly, I don't think we ever needed the file itself. Just the string needs to be defined.

The manylinux container (in which we do the build) has the static libpython libraries tarred in a special location (to save space in the docker container):

$ docker run --platform=linux/arm64 --rm -it rapidsai/manylinux_2_31:cuda-devel-11.8.0-ubuntu20.04
...
root@8c120d673c7f:/# ls /opt/_internal/static-libs-for-embedding-only.tar.xz
/opt/_internal/static-libs-for-embedding-only.tar.xz
root@8c120d673c7f:/test# mkdir -p test && cd test
root@8c120d673c7f:/test# tar -xf /opt/_internal/static-libs-for-embedding-only.tar.xz
root@8c120d673c7f:/test# find . -name '*libpython*'
./cpython-3.8.16/lib/libpython3.8.a
./cpython-3.10.9/lib/libpython3.10.a
./cpython-3.7.16/lib/libpython3.7m.a
./cpython-3.11.1/lib/libpython3.11.a
./cpython-3.6.15/lib/libpython3.6m.a
./cpython-3.9.16/lib/libpython3.9.a
root@8c120d673c7f:/test# ^C

We don't (and, I don't think ever needed to, not since the earliest proof-of-concept of cibuildwheel) untar those libraries. So it seems that despite not needing the actual library to exist, the cmake command fails simply because the string is missing.

[vyasr]

For context, we needed the static libraries at one point when we were using CMake's FindPython to get numpy includes. There's an issue with CMake<3.25 where finding the NumPy component requires finding the Development component, and the latter fails unless the static libs are visible. I switched to using the FindNumpy module in scikit-build to get around this. When we bump cudf's minimum required CMake version to 3.25 we'll switch back and shouldn't have that issue any longer.

As far as what's causing the above bug, I'll try to dig into this sometime and see why we need that library set.

[vyasr]

No longer relevant, our modern builds don't need this and all the above patches have been removed now that we're on a much newer version of CMake and we've switched to scikit-build-core.