Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

libcublas on aarch64 requires glibc 2.27 #13

Open
leofang opened this issue May 11, 2023 · 5 comments
Open

libcublas on aarch64 requires glibc 2.27 #13

leofang opened this issue May 11, 2023 · 5 comments

Comments

@leofang
Copy link
Member

leofang commented May 11, 2023 

(test) [conda@8e09d4a629ed ~]$ ldd -v /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12
/opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12)
	linux-vdso.so.1 =>  (0x0000007fa8019000)
	librt.so.1 => /lib64/librt.so.1 (0x0000007f8a7a6000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x0000007f8a771000)
	libdl.so.2 => /lib64/libdl.so.2 (0x0000007f8a750000)
	libm.so.6 => /lib64/libm.so.6 (0x0000007f8a69f000)
	libgcc_s.so.1 => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1 (0x0000007f8a66e000)
	libc.so.6 => /lib64/libc.so.6 (0x0000007f8a4e8000)
	/lib/ld-linux-aarch64.so.1 (0x0000007fa7feb000)

	Version information:
	/opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12:
		librt.so.1 (GLIBC_2.17) => /lib64/librt.so.1
		libm.so.6 (GLIBC_2.27) => not found
		libdl.so.2 (GLIBC_2.17) => /lib64/libdl.so.2
		libgcc_s.so.1 (GCC_4.2.0) => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1
		libgcc_s.so.1 (GCC_3.3) => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1
		libgcc_s.so.1 (GCC_4.5.0) => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1
		libgcc_s.so.1 (GCC_3.0) => /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1
		libpthread.so.0 (GLIBC_2.17) => /lib64/libpthread.so.0
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/librt.so.1:
		libpthread.so.0 (GLIBC_PRIVATE) => /lib64/libpthread.so.0
		libpthread.so.0 (GLIBC_2.17) => /lib64/libpthread.so.0
		libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/libpthread.so.0:
		ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
		ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1
		libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/libdl.so.2:
		ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1
		libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/libm.so.6:
		libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libgcc_s.so.1:
		libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
	/lib64/libc.so.6:
		ld-linux-aarch64.so.1 (GLIBC_2.17) => /lib/ld-linux-aarch64.so.1
		ld-linux-aarch64.so.1 (GLIBC_PRIVATE) => /lib/ld-linux-aarch64.so.1

Note the two lines

/opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/conda/envs/test/lib/python3.10/site-packages/cupy_backends/cuda/libs/../../../../../libcublasLt.so.12)

and

		libm.so.6 (GLIBC_2.27) => not found

Essentially, we are once again hitting the same issue as in cuSPARSELt v0.4.0 (conda-forge/cusparselt-feedstock#23 (comment)) and cuDNN v8.8.0 (conda-forge/cudnn-feedstock#58), but this went totally unnoticed until now when I work on conda-forge/cupy-feedstock#199, because we haven't added the dlopen test to all new CUDA packages.

cc: @jakirkham
@leofang leofang mentioned this issue May 11, 2023
Merged
1 task
@leofang
Copy link
Member Author

leofang commented May 11, 2023 

$ conda list
# packages in environment at /opt/conda/envs/test:
#
# Name                    Version                   Build  Channel
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                hf897c2e_4    conda-forge
ca-certificates           2023.5.7             hcefe29a_0    conda-forge
cuda-cudart               12.0.107             hac28a21_1    conda-forge
cuda-nvrtc                12.0.76              hac28a21_1    conda-forge
cuda-version              12.0                 hffde075_2    conda-forge
cupy                      12.0.0          py310h18013f7_2    <unknown>
fastrlock                 0.8             py310h16c9dca_3    conda-forge
ld_impl_linux-aarch64     2.40                 h2d8c526_0    conda-forge
libblas                   3.9.0           16_linuxaarch64_openblas    conda-forge
libcblas                  3.9.0           16_linuxaarch64_openblas    conda-forge
libcublas                 12.0.1.189           ha21bd82_2    conda-forge
libcufft                  11.0.0.21            ha21bd82_1    conda-forge
libcurand                 10.3.1.50            ha21bd82_0    conda-forge
libcusolver               11.4.2.57            ha21bd82_1    conda-forge
libffi                    3.4.2                h3557bc0_5    conda-forge
libgcc-ng                 12.2.0              h607ecd0_19    conda-forge
libgfortran-ng            12.2.0              he9431aa_19    conda-forge
libgfortran5              12.2.0              hf695500_19    conda-forge
libgomp                   12.2.0              h607ecd0_19    conda-forge
liblapack                 3.9.0           16_linuxaarch64_openblas    conda-forge
libnsl                    2.0.0                hf897c2e_0    conda-forge
libnvjitlink              12.0.76              ha21bd82_1    conda-forge
libopenblas               0.3.21          pthreads_h6cb6f83_3    conda-forge
libsqlite                 3.41.2               h194ca79_1    conda-forge
libstdcxx-ng              12.2.0              hc13a102_19    conda-forge
libuuid                   2.38.1               hb4cce97_0    conda-forge
libzlib                   1.2.13               h4e544f5_4    conda-forge
ncurses                   6.3                  headf329_1    conda-forge
numpy                     1.24.3          py310h7f380f4_0    conda-forge
openssl                   3.1.0                h31becfc_3    conda-forge
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
python                    3.10.11         ha43d526_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.2                  h8fc344f_1    conda-forge
setuptools                67.7.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               hd8af866_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h9cdd2b7_0    conda-forge

@jakirkham
Copy link
Member

Am curious why we didn't see an issue in the ARM build here? Would think this would have detected a new version of GLIBC is needed and not provided for (as seen elsewhere)

@bdice
Copy link
Contributor

bdice commented May 11, 2023

@jakirkham I think only the libraries in $PREFIX/lib/ are checked for symbol compatibility. I believe the files in $PREFIX/targets/sbsa-linux/lib/ are "just files" as far as conda-build cares.

@leofang
Copy link
Member Author

leofang commented May 16, 2023 •
edited
Loading

Am curious why we didn't see an issue in the ARM build here? Would think this would have detected a new version of GLIBC is needed and not provided for (as seen elsewhere)

I think what happens is that conda-build correctly found the dependency on libm.so

   INFO (libcublas,targets/sbsa-linux/lib/libcublasLt.so.12.0.1.189): Needed DSO aarch64-conda-linux-gnu/sysroot/lib64/libm.so.6 found in CDT/compiler package conda-forge::sysroot_linux-aarch64-2.17-h43d7e78_13

but it didn't check if the found DSO also satisfies the glibc requirement (i.e. containing all the versioned symbols needed by libcublasLt.so).

This was referenced Aug 30, 2023
Merged
Closed
Tobias-Fischer added a commit to Tobias-Fischer/dlib-feedstock that referenced this issue Jan 6, 2024
@Tobias-Fischer
Skip tests on aarch64+cuda because of conda-forge/libcublas-feedstock#13
064c897
@jakirkham
Copy link
Member

Cross-linking to CUDA Toolkit CentOS 7 deprecation issue ( conda-forge/cuda-feedstock#28 ), which this depends on

@h-vetinari h-vetinari mentioned this issue Oct 31, 2024
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jakirkham @bdice @leofang