tensorflow-gpu 2.17.0 Could not find GPU #410

alexfanqi · 2024-12-26T04:51:49Z

Solution to issue cannot be found in the documentation.

I checked the documentation.

Issue

I installed tensorflow-gpu 2.17.0 via conda. It cannot recognise my GPU and report the following warning,

Python 3.12.8 | packaged by conda-forge | (main, Dec  5 2024, 14:24:40) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2024-12-26 12:50:00.342945: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-12-26 12:50:00.387766: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-12-26 12:50:01.052616: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
2024-12-26 12:50:04.448423: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-12-26 12:50:04.453217: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Num GPUs Available:  0
>>>

Installed packages

cuda-cccl_linux-64           12.6.77  ha770c72_0  conda-forge
  cuda-crt-dev_linux-64        12.6.85  ha770c72_0  conda-forge
  cuda-crt-tools               12.6.85  ha770c72_0  conda-forge
  cuda-cudart                  12.6.77  h5888daf_0  conda-forge
  cuda-cudart-dev              12.6.77  h5888daf_0  conda-forge
  cuda-cudart-dev_linux-64     12.6.77  h3f2d84a_0  conda-forge
  cuda-cudart-static           12.6.77  h5888daf_0  conda-forge
  cuda-cudart-static_linux-64  12.6.77  h3f2d84a_0  conda-forge
  cuda-cudart_linux-64         12.6.77  h3f2d84a_0  conda-forge
  cuda-cupti                   12.6.80  hbd13f7d_0  conda-forge
  cuda-cupti-dev               12.6.80  h5888daf_0  conda-forge
  cuda-driver-dev_linux-64     12.6.77  h3f2d84a_0  conda-forge
  cuda-nvcc-dev_linux-64       12.6.85  he91c749_0  conda-forge
  cuda-nvcc-impl               12.6.85  h85509e4_0  conda-forge
  cuda-nvcc-tools              12.6.85  he02047a_0  conda-forge
  cuda-nvcc_linux-64           12.6.85  h04802cd_0  conda-forge
  cuda-nvrtc                   12.6.85  hbd13f7d_0  conda-forge
  cuda-nvtx                    12.6.77  hbd13f7d_0  conda-forge
  cuda-nvvm                    12.6.85  h69a702a_0  conda-forge
  cuda-nvvm-dev_linux-64       12.6.85  ha770c72_0  conda-forge
  cuda-nvvm-impl               12.6.85  he02047a_0  conda-forge
  cuda-nvvm-tools              12.6.85  he02047a_0  conda-forge
  cuda-version                 12.6     h7480c83_3  conda-forge
  libopenvino-tensorflow-frontend       2024.4.0  h9718a47_1                conda-forge
  libopenvino-tensorflow-lite-frontend  2024.4.0  h5888daf_1                conda-forge
  tensorflow                            2.17.0    cuda120py312h02ad488_203  conda-forge
  tensorflow-base                       2.17.0    cuda120py312hbec54f7_203  conda-forge
  tensorflow-estimator                  2.17.0    cuda120py312hfa0f5ef_203  conda-forge
  tensorflow-gpu                        2.17.0    cuda120py312hb76ca00_203  conda-forge

Environment info

libmamba version : 1.5.8
     micromamba version : 1.5.8
           curl version : libcurl/8.6.0 OpenSSL/3.2.2 zlib/1.3.1.zlib-ng brotli/1.1.0 libidn2/2.3.7 libpsl/0.21.5 libssh/0.10.6/openssl/zlib nghttp2/1.59.0 OpenLDAP/2.6.7
     libarchive version : libarchive 3.7.2 zlib/1.3.1.zlib-ng liblzma/5.4.6 bz2lib/1.0.8 liblz4/1.9.4 libzstd/1.5.6
       envs directories : /home/alexfanqi/micromamba/envs
          package cache : /home/alexfanqi/micromamba/pkgs
                          /home/alexfanqi/.mamba/pkgs
            environment : ml-py312 (active)
           env location : /home/alexfanqi/micromamba/envs/ml-py312
      user config files : /home/alexfanqi/.mambarc
 populated config files : /home/alexfanqi/.condarc
       virtual packages : __unix=0=0
                          __linux=6.12.4=0
                          __glibc=2.39=0
                          __archspec=1=x86_64-v3
                          __cuda=12.7=0
               channels : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/litex-hub/linux-64
                          https://conda.anaconda.org/litex-hub/noarch
                          https://conda.anaconda.org/main/linux-64
                          https://conda.anaconda.org/main/noarch
                          https://conda.anaconda.org/vlsida-eda/linux-64
                          https://conda.anaconda.org/vlsida-eda/noarch
                          https://conda.anaconda.org/anaconda/linux-64
                          https://conda.anaconda.org/anaconda/noarch
                          https://conda.anaconda.org/pyg/linux-64
                          https://conda.anaconda.org/pyg/noarch
       base environment : /home/alexfanqi/micromamba
               platform : linux-64

The text was updated successfully, but these errors were encountered:

alexfanqi · 2024-12-27T02:35:38Z

the error is still there after installing tensorrt_cu12==10.7.0 with pip and export LD_LIBRARY_PATH=ENV_PATH/lib/python3.12/site-packages/tensorrt_libs:${LD_LIBRARY_PATH}

njzjz · 2025-01-11T06:12:14Z

Hi @alexfanqi, please report the result when setting the environment variable TF_CPP_MAX_VLOG_LEVEL=3:

TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

alexfanqi · 2025-01-12T14:07:28Z

TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"
has the following output

2025-01-12 22:03:00.525134: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-01-12 22:03:00.528289: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2025-01-12 22:03:00.529039: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2025-01-12 22:03:00.530050: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2025-01-12 22:03:00.530072: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2025-01-12 22:03:00.530080: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2025-01-12 22:03:00.530091: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-12 22:03:00.530100: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-12 22:03:00.575313: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2025-01-12 22:03:00.692888: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2025-01-12 22:03:00.692925: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2025-01-12 22:03:00.692930: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2025-01-12 22:03:00.692933: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2025-01-12 22:03:00.692940: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-12 22:03:00.692944: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-12 22:03:00.693308: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-12 22:03:01.184378: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2025-01-12 22:03:01.184436: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2025-01-12 22:03:01.184453: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2025-01-12 22:03:01.184461: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2025-01-12 22:03:01.184473: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-12 22:03:01.184482: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-12 22:03:01.490179: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer.so.8.6.1'; dlerror: libnvinfer.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/ml-py312/lib/libfabric:
2025-01-12 22:03:01.490264: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer_plugin.so.8.6.1'; dlerror: libnvinfer_plugin.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/ml-py312/lib/libfabric:
2025-01-12 22:03:01.490270: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2025-01-12 22:03:03.772775: I external/local_xla/xla/parse_flags_from_env.cc:207] For env var TF_XLA_FLAGS found arguments:
2025-01-12 22:03:03.772805: I external/local_xla/xla/parse_flags_from_env.cc:209]   argv[0] = <argv[0]>
2025-01-12 22:03:03.772813: I external/local_xla/xla/parse_flags_from_env.cc:207] For env var TF_JITRT_FLAGS found arguments:
2025-01-12 22:03:03.772817: I external/local_xla/xla/parse_flags_from_env.cc:209]   argv[0] = <argv[0]>
2025-01-12 22:03:03.772822: I tensorflow/compiler/jit/xla_cpu_device.cc:46] Not creating XLA devices, tf_xla_enable_xla_devices not set and XLA device creation not requested
2025-01-12 22:03:03.772827: I tensorflow/compiler/jit/xla_gpu_device.cc:49] Not creating XLA devices, tf_xla_enable_xla_devices not set and XLA devices creation not required
2025-01-12 22:03:03.786459: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcuda.so.1
2025-01-12 22:03:03.893155: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:192] found DLL info with name: /lib64/libcuda.so.1
2025-01-12 22:03:03.893195: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:197] found DLL info with resolved path: /usr/lib64/libcuda.so.565.77
2025-01-12 22:03:03.893211: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:97] version string "565.77" made value 565.77.0
2025-01-12 22:03:03.893234: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:968] trying to read NUMA node for device ordinal: 0
2025-01-12 22:03:03.893275: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2025-01-12 22:03:03.896258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2219] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce MX550 computeCapability: 7.5
coreClock: 1.53GHz coreCount: 16 deviceMemorySize: 1.65GiB deviceMemoryBandwidth: 104.32GiB/s
2025-01-12 22:03:03.896285: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2025-01-12 22:03:03.896484: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcublas.so.12'; dlerror: libcublas.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/ml-py312/lib/libfabric:
2025-01-12 22:03:03.896616: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcublasLt.so.12'; dlerror: libcublasLt.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/ml-py312/lib/libfabric:
2025-01-12 22:03:03.896964: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcufft.so.11'; dlerror: libcufft.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/ml-py312/lib/libfabric:
2025-01-12 22:03:03.897304: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/ml-py312/lib/libfabric:
2025-01-12 22:03:03.897645: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcusparse.so.12'; dlerror: libcusparse.so.12: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/ml-py312/lib/libfabric:
2025-01-12 22:03:03.897727: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/ml-py312/lib/libfabric:
2025-01-12 22:03:03.897738: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]

And searching my conda env, I found

(ml-py312) alexfanqi@Bashrogh:~/.../ml-py312$ find . -name libcudart.so.12*
./targets/x86_64-linux/lib/libcudart.so.12.6.77
./targets/x86_64-linux/lib/libcudart.so.12
./lib/libcudart.so.12
./lib/libcudart.so.12.6.77

(ml-py312) alexfanqi@Bashrogh:~/.../ml-py312$ find . -name libcublas.so.12*
./targets/x86_64-linux/lib/libcublas.so.12.6.4.1
./targets/x86_64-linux/lib/libcublas.so.12
./lib/libcublas.so.12.6.4.1
./lib/libcublas.so.12

also jax and pytorch seem to be detecting my GPU correctly

hmaarrfk · 2025-01-12T17:05:38Z

Are you using Windows Subsystem for Linux?
WSL?

alexfanqi · 2025-01-13T12:51:32Z

Nah, this is on Fedora and micromamba. Maybe it is having similar issues with WSL?

hmaarrfk · 2025-01-13T13:01:16Z

Maybe it is having similar issues with WSL?

I'm not sure. many of us don't use windows, so WSL adds an other layer of complexity to troubleshoot. I wanted to make sure you were not seeing it.

On ubuntu it seems to find it fine.

But tbh, you report

                          __cuda=12.7=0

which I have never seen before (which is why i thought you were on WSL).

alexfanqi · 2025-01-13T13:47:03Z

ok, fiddling with LD_LIBRARY_PATH and XLA_FLAGS makes it detect my gpu successfully now

export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:${LD_LIBRARY_PATH}
export XLA_FLAGS=--xla_gpu_cuda_data_dir=$CONDA_PREFIX/targets/x86_64-linux

but apparently libcudnn version doesn't match with tensorflow. simple soft linking cudnn.so doesn't work.

2025-01-13 21:34:03.284797: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:458]
Loaded runtime CuDNN library: 9.3.0 but source was compiled with: 8.9.6. 
CuDNN library needs to have matching major version and equal or higher minor version.
If using a binary install, upgrade your CuDNN library. 
If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.

hmaarrfk · 2025-01-13T13:53:12Z

Can you please create a clean environment (without your own proprietary stuff) and see if the issue can be produced?

Please provide the full list of installed packages, not the subset that you feel is helpful.

The reason I ask is that build "X02" and above were built with CUDNN9 #402

alexfanqi · 2025-01-13T14:09:36Z

sure, I don't have proprietary stuff and I tried to create a clean environment with

micromamba env create -n test -c conda-forge
micromamba install tensorflow-gpu

And with the exports, I got the same mismatched cudnn version

(test) alexfanqi@Bashrogh:/tmp$ TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"
2025-01-13 22:04:21.652670: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-01-13 22:04:21.660104: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2025-01-13 22:04:21.660590: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2025-01-13 22:04:21.660615: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2025-01-13 22:04:21.660622: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2025-01-13 22:04:21.660627: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2025-01-13 22:04:21.660635: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-13 22:04:21.660641: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-13 22:04:21.663911: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2025-01-13 22:04:21.709640: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2025-01-13 22:04:21.709696: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2025-01-13 22:04:21.709703: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2025-01-13 22:04:21.709708: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2025-01-13 22:04:21.709717: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-13 22:04:21.709722: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-13 22:04:21.710228: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-13 22:04:22.207336: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2025-01-13 22:04:22.207387: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2025-01-13 22:04:22.207396: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2025-01-13 22:04:22.207407: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2025-01-13 22:04:22.207416: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-13 22:04:22.207422: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2025-01-13 22:04:22.440489: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer.so.8.6.1'; dlerror: libnvinfer.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/test/lib:/python3.12/site-packages/tensorrt_libs:
2025-01-13 22:04:22.440579: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer_plugin.so.8.6.1'; dlerror: libnvinfer_plugin.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/test/lib:/python3.12/site-packages/tensorrt_libs:
2025-01-13 22:04:22.440589: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2025-01-13 22:04:22.958465: I external/local_xla/xla/parse_flags_from_env.cc:207] For env var TF_XLA_FLAGS found arguments:
2025-01-13 22:04:22.958503: I external/local_xla/xla/parse_flags_from_env.cc:209]   argv[0] = <argv[0]>
2025-01-13 22:04:22.958512: I external/local_xla/xla/parse_flags_from_env.cc:207] For env var TF_JITRT_FLAGS found arguments:
2025-01-13 22:04:22.958517: I external/local_xla/xla/parse_flags_from_env.cc:209]   argv[0] = <argv[0]>
2025-01-13 22:04:22.958522: I tensorflow/compiler/jit/xla_cpu_device.cc:46] Not creating XLA devices, tf_xla_enable_xla_devices not set and XLA device creation not requested
2025-01-13 22:04:22.958527: I tensorflow/compiler/jit/xla_gpu_device.cc:49] Not creating XLA devices, tf_xla_enable_xla_devices not set and XLA devices creation not required
2025-01-13 22:04:22.959504: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcuda.so.1
2025-01-13 22:04:23.089323: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:192] found DLL info with name: /lib64/libcuda.so.1
2025-01-13 22:04:23.089382: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:197] found DLL info with resolved path: /usr/lib64/libcuda.so.565.77
2025-01-13 22:04:23.089396: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:97] version string "565.77" made value 565.77.0
2025-01-13 22:04:23.089419: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:968] trying to read NUMA node for device ordinal: 0
2025-01-13 22:04:23.089459: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2025-01-13 22:04:23.094291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2219] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce MX550 computeCapability: 7.5
coreClock: 1.53GHz coreCount: 16 deviceMemorySize: 1.65GiB deviceMemoryBandwidth: 104.32GiB/s
2025-01-13 22:04:23.094310: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2025-01-13 22:04:23.411902: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublas.so.12
2025-01-13 22:04:23.412016: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublasLt.so.12
2025-01-13 22:04:23.448138: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcufft.so.11
2025-01-13 22:04:23.572694: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusolver.so.11
2025-01-13 22:04:23.572845: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusparse.so.12
2025-01-13 22:04:23.573026: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/alexfanqi/micromamba/envs/test/lib:/python3.12/site-packages/tensorrt_libs:
2025-01-13 22:04:23.573041: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]

a full list of packages

(test) alexfanqi@Bashrogh:/tmp$ micromamba list
List of packages in environment: "/home/alexfanqi/micromamba/envs/test"

  Name                     Version       Build                     Channel    
────────────────────────────────────────────────────────────────────────────────
  _libgcc_mutex            0.1           conda_forge               conda-forge
  _openmp_mutex            4.5           2_gnu                     conda-forge
  absl-py                  2.1.0         pyhd8ed1ab_1              conda-forge
  astunparse               1.6.3         pyhd8ed1ab_3              conda-forge
  brotli-python            1.1.0         py312h2ec8cdc_2           conda-forge
  bzip2                    1.0.8         h4bc722e_7                conda-forge
  c-ares                   1.34.4        hb9d3cd8_0                conda-forge
  ca-certificates          2024.12.14    hbcca054_0                conda-forge
  cached-property          1.5.2         hd8ed1ab_1                conda-forge
  cached_property          1.5.2         pyha770c72_1              conda-forge
  certifi                  2024.12.14    pyhd8ed1ab_0              conda-forge
  cffi                     1.17.1        py312h06ac9bb_0           conda-forge
  charset-normalizer       3.4.1         pyhd8ed1ab_0              conda-forge
  cuda-crt-tools           12.6.85       ha770c72_0                conda-forge
  cuda-cudart              12.6.77       h5888daf_0                conda-forge
  cuda-cudart_linux-64     12.6.77       h3f2d84a_0                conda-forge
  cuda-cupti               12.6.80       hbd13f7d_0                conda-forge
  cuda-nvcc-tools          12.6.85       he02047a_0                conda-forge
  cuda-nvrtc               12.6.85       hbd13f7d_0                conda-forge
  cuda-nvtx                12.6.77       hbd13f7d_0                conda-forge
  cuda-nvvm-tools          12.6.85       he02047a_0                conda-forge
  cuda-version             12.6          h7480c83_3                conda-forge
  cudnn                    9.3.0.75      h62a6f1c_2                conda-forge
  flatbuffers              24.3.25       h59595ed_0                conda-forge
  gast                     0.6.0         pyhd8ed1ab_0              conda-forge
  giflib                   5.2.2         hd590300_0                conda-forge
  google-pasta             0.2.0         pyhd8ed1ab_2              conda-forge
  grpcio                   1.65.5        py312h374181b_0           conda-forge
  h2                       4.1.0         pyhd8ed1ab_1              conda-forge
  h5py                     3.12.1        nompi_py312hedeef09_103   conda-forge
  hdf5                     1.14.3        nompi_h2d575fe_108        conda-forge
  hpack                    4.0.0         pyhd8ed1ab_1              conda-forge
  hyperframe               6.0.1         pyhd8ed1ab_1              conda-forge
  icu                      75.1          he02047a_0                conda-forge
  idna                     3.10          pyhd8ed1ab_1              conda-forge
  importlib-metadata       8.5.0         pyha770c72_1              conda-forge
  keras                    3.8.0         pyh753f3f9_0              conda-forge
  keyutils                 1.6.1         h166bdaf_0                conda-forge
  krb5                     1.21.3        h659f571_0                conda-forge
  ld_impl_linux-64         2.43          h712a8e2_2                conda-forge
  libabseil                20240722.0    cxx17_hbbce691_4          conda-forge
  libaec                   1.1.3         h59595ed_0                conda-forge
  libblas                  3.9.0         26_linux64_openblas       conda-forge
  libcblas                 3.9.0         26_linux64_openblas       conda-forge
  libcublas                12.6.4.1      hbd13f7d_0                conda-forge
  libcufft                 11.3.0.4      hbd13f7d_0                conda-forge
  libcurand                10.3.7.77     hbd13f7d_0                conda-forge
  libcurl                  8.11.1        h332b0f4_0                conda-forge
  libcusolver              11.7.1.2      hbd13f7d_0                conda-forge
  libcusparse              12.5.4.2      hbd13f7d_0                conda-forge
  libedit                  3.1.20240808  pl5321h7949ede_0          conda-forge
  libev                    4.33          hd590300_2                conda-forge
  libexpat                 2.6.4         h5888daf_0                conda-forge
  libffi                   3.4.2         h7f98852_5                conda-forge
  libgcc                   14.2.0        h77fa898_1                conda-forge
  libgcc-ng                14.2.0        h69a702a_1                conda-forge
  libgfortran              14.2.0        h69a702a_1                conda-forge
  libgfortran5             14.2.0        hd5240d6_1                conda-forge
  libgomp                  14.2.0        h77fa898_1                conda-forge
  libgrpc                  1.65.5        hf5c653b_0                conda-forge
  libjpeg-turbo            3.0.0         hd590300_1                conda-forge
  liblapack                3.9.0         26_linux64_openblas       conda-forge
  liblzma                  5.6.3         hb9d3cd8_1                conda-forge
  libnghttp2               1.64.0        h161d5f1_0                conda-forge
  libnsl                   2.0.1         hd590300_0                conda-forge
  libnvjitlink             12.6.85       hbd13f7d_0                conda-forge
  libopenblas              0.3.28        pthreads_h94d23a6_1       conda-forge
  libpng                   1.6.45        h943b412_0                conda-forge
  libprotobuf              5.27.5        h5b01275_2                conda-forge
  libre2-11                2024.07.02    hbbce691_2                conda-forge
  libsqlite                3.47.2        hee588c1_0                conda-forge
  libssh2                  1.11.1        hf672d98_0                conda-forge
  libstdcxx                14.2.0        hc0a3c3a_1                conda-forge
  libstdcxx-ng             14.2.0        h4852527_1                conda-forge
  libuuid                  2.38.1        h0b41bf4_0                conda-forge
  libxcrypt                4.4.36        hd590300_1                conda-forge
  libzlib                  1.3.1         hb9d3cd8_2                conda-forge
  markdown                 3.6           pyhd8ed1ab_0              conda-forge
  markdown-it-py           3.0.0         pyhd8ed1ab_1              conda-forge
  markupsafe               3.0.2         py312h178313f_1           conda-forge
  mdurl                    0.1.2         pyhd8ed1ab_1              conda-forge
  ml_dtypes                0.4.0         py312hf9745cd_2           conda-forge
  namex                    0.0.8         pyhd8ed1ab_1              conda-forge
  nccl                     2.24.3.1      hb92ee24_0                conda-forge
  ncurses                  6.5           h2d0b736_2                conda-forge
  numpy                    1.26.4        py312heda63a1_0           conda-forge
  openssl                  3.4.0         h7b32b05_1                conda-forge
  opt_einsum               3.4.0         pyhd8ed1ab_1              conda-forge
  optree                   0.13.1        py312h68727a3_1           conda-forge
  packaging                24.2          pyhd8ed1ab_2              conda-forge
  pip                      24.3.1        pyh8b19718_2              conda-forge
  protobuf                 5.27.5        py312h2ec8cdc_0           conda-forge
  pycparser                2.22          pyh29332c3_1              conda-forge
  pygments                 2.19.1        pyhd8ed1ab_0              conda-forge
  pysocks                  1.7.1         pyha55dd90_7              conda-forge
  python                   3.12.8        h9e4cc4f_1_cpython        conda-forge
  python-flatbuffers       24.12.23      pyhe33e51e_0              conda-forge
  python_abi               3.12          5_cp312                   conda-forge
  re2                      2024.07.02    h9925aae_2                conda-forge
  readline                 8.2           h8228510_1                conda-forge
  requests                 2.32.3        pyhd8ed1ab_1              conda-forge
  rich                     13.9.4        pyhd8ed1ab_1              conda-forge
  setuptools               75.8.0        pyhff2d567_0              conda-forge
  six                      1.17.0        pyhd8ed1ab_0              conda-forge
  snappy                   1.2.1         h8bd8927_1                conda-forge
  tensorboard              2.17.1        pyhd8ed1ab_0              conda-forge
  tensorboard-data-server  0.7.0         py312hda17c39_2           conda-forge
  tensorflow               2.17.0        cuda120py312h02ad488_203  conda-forge
  tensorflow-base          2.17.0        cuda120py312hbec54f7_203  conda-forge
  tensorflow-estimator     2.17.0        cuda120py312hfa0f5ef_203  conda-forge
  tensorflow-gpu           2.17.0        cuda120py312hb76ca00_203  conda-forge
  termcolor                2.5.0         pyhd8ed1ab_1              conda-forge
  tk                       8.6.13        noxft_h4845f30_101        conda-forge
  typing-extensions        4.12.2        hd8ed1ab_1                conda-forge
  typing_extensions        4.12.2        pyha770c72_1              conda-forge
  tzdata                   2024b         hc8b5060_0                conda-forge
  urllib3                  2.3.0         pyhd8ed1ab_0              conda-forge
  werkzeug                 3.1.3         pyhd8ed1ab_1              conda-forge
  wheel                    0.45.1        pyhd8ed1ab_1              conda-forge
  wrapt                    1.17.1        py312h66e93f0_0           conda-forge
  zipp                     3.21.0        pyhd8ed1ab_1              conda-forge
  zstandard                0.23.0        py312hef9b889_1           conda-forge
  zstd                     1.5.6         ha6fb4c9_0                conda-forge

hmaarrfk · 2025-01-13T14:23:45Z

Can yo utry to run with the environment variable
#296 (comment)

XLA_FLAGS=--xla_gpu_cuda_data_dir=${CONDA_PREFIX}/targets/x86_64-linux TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

Low probability of working.... but got to try

alexfanqi · 2025-01-13T14:50:35Z

no luck :-\

hmaarrfk · 2025-01-13T15:15:11Z

silly thing, can you restart your computer? it helps when nvidia updates their driver live.

hmaarrfk · 2025-01-13T15:15:29Z

finally, does nvtop as installed from conda-forge report your GPU?

alexfanqi · 2025-01-14T05:07:36Z

still no luck.

btw, in case other people are searching for tensorrt, tensorrt can seemingly be loaded as well via

export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.12/site-packages/tensorrt_libs:${LD_LIBRARY_PATH}

Tensorflow 2.17.0 apparently requires tensorrt version 8.6.
install it with pip install --no-dependencies tensorrt_libs==8.6.1 (to avoid duplicating cublas cudnn libs from pip), potentially also ignore the tensorrt_binding error

alexfanqi · 2025-01-14T05:28:46Z

After downgrading to cudnn to 8.9.7, tensorflow loads my GPU successfully. So tensorflow 2.17.0 is not compiled with cudnn 9 although it is updated in the dependency list even after #402?

hmaarrfk · 2025-01-14T06:01:55Z

btw, in case other people are searching for tensorrt, tensorrt can seemingly be loaded as well via

lets not mix issues.

After downgrading to cudnn to 8.9.7, tensorflow loads my GPU successfully.

I think something is up with fedora's libraries names. It wouldn't be the first time.

Basically I have

GPU detected by tensorflow

python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"
2025-01-14 00:59:27.770389: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-14 00:59:27.809874: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-14 00:59:27.821288: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-14 00:59:28.054709: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1736834371.332007 2875512 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1736834371.404528 2875512 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1736834371.404832 2875512 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

and

conda list

``` $ conda list # packages in environment at /home/mark/miniforge3/envs/tf: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge absl-py 2.1.0 pyhd8ed1ab_1 conda-forge astunparse 1.6.3 pyhd8ed1ab_3 conda-forge brotli-python 1.1.0 py312h2ec8cdc_2 conda-forge bzip2 1.0.8 h4bc722e_7 conda-forge c-ares 1.34.4 hb9d3cd8_0 conda-forge ca-certificates 2024.12.14 hbcca054_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge certifi 2024.12.14 pyhd8ed1ab_0 conda-forge cffi 1.17.1 py312h06ac9bb_0 conda-forge charset-normalizer 3.4.1 pyhd8ed1ab_0 conda-forge cuda-crt-tools 12.6.85 ha770c72_0 conda-forge cuda-cudart 12.6.77 h5888daf_0 conda-forge cuda-cudart_linux-64 12.6.77 h3f2d84a_0 conda-forge cuda-cupti 12.6.80 hbd13f7d_0 conda-forge cuda-nvcc-tools 12.6.85 he02047a_0 conda-forge cuda-nvrtc 12.6.85 hbd13f7d_0 conda-forge cuda-nvtx 12.6.77 hbd13f7d_0 conda-forge cuda-nvvm-tools 12.6.85 he02047a_0 conda-forge cuda-version 12.6 h7480c83_3 conda-forge cudnn 9.3.0.75 h62a6f1c_2 conda-forge flatbuffers 24.3.25 h59595ed_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_3 conda-forge fontconfig 2.15.0 h7e30c49_1 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge freetype 2.12.1 h267a509_2 conda-forge gast 0.6.0 pyhd8ed1ab_0 conda-forge giflib 5.2.2 hd590300_0 conda-forge google-pasta 0.2.0 pyhd8ed1ab_2 conda-forge grpcio 1.65.5 py312h374181b_0 conda-forge h2 4.1.0 pyhd8ed1ab_1 conda-forge h5py 3.12.1 nompi_py312hd203070_103 conda-forge hdf5 1.14.4 nompi_h2d575fe_105 conda-forge hpack 4.0.0 pyhd8ed1ab_1 conda-forge hyperframe 6.0.1 pyhd8ed1ab_1 conda-forge icu 75.1 he02047a_0 conda-forge idna 3.10 pyhd8ed1ab_1 conda-forge importlib-metadata 8.5.0 pyha770c72_1 conda-forge keras 3.8.0 pyh753f3f9_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge krb5 1.21.3 h659f571_0 conda-forge ld_impl_linux-64 2.43 h712a8e2_2 conda-forge libabseil 20240722.0 cxx17_hbbce691_4 conda-forge libaec 1.1.3 h59595ed_0 conda-forge libblas 3.9.0 26_linux64_openblas conda-forge libcblas 3.9.0 26_linux64_openblas conda-forge libcublas 12.6.4.1 hbd13f7d_0 conda-forge libcufft 11.3.0.4 hbd13f7d_0 conda-forge libcurand 10.3.7.77 hbd13f7d_0 conda-forge libcurl 8.11.1 h332b0f4_0 conda-forge libcusolver 11.7.1.2 hbd13f7d_0 conda-forge libcusparse 12.5.4.2 hbd13f7d_0 conda-forge libedit 3.1.20240808 pl5321h7949ede_0 conda-forge libev 4.33 hd590300_2 conda-forge libexpat 2.6.4 h5888daf_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc 14.2.0 h77fa898_1 conda-forge libgcc-ng 14.2.0 h69a702a_1 conda-forge libgfortran 14.2.0 h69a702a_1 conda-forge libgfortran5 14.2.0 hd5240d6_1 conda-forge libgomp 14.2.0 h77fa898_1 conda-forge libgrpc 1.65.5 hf5c653b_0 conda-forge libjpeg-turbo 3.0.0 hd590300_1 conda-forge liblapack 3.9.0 26_linux64_openblas conda-forge liblzma 5.6.3 hb9d3cd8_1 conda-forge libnghttp2 1.64.0 h161d5f1_0 conda-forge libnsl 2.0.1 hd590300_0 conda-forge libnvjitlink 12.6.85 hbd13f7d_0 conda-forge libopenblas 0.3.28 pthreads_h94d23a6_1 conda-forge libpng 1.6.45 h943b412_0 conda-forge libprotobuf 5.27.5 h5b01275_2 conda-forge libre2-11 2024.07.02 hbbce691_2 conda-forge libsqlite 3.47.2 hee588c1_0 conda-forge libssh2 1.11.1 hf672d98_0 conda-forge libstdcxx 14.2.0 hc0a3c3a_1 conda-forge libstdcxx-ng 14.2.0 h4852527_1 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libxcb 1.17.0 h8a09558_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libzlib 1.3.1 hb9d3cd8_2 conda-forge markdown 3.6 pyhd8ed1ab_0 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_1 conda-forge markupsafe 3.0.2 py312h178313f_1 conda-forge mdurl 0.1.2 pyhd8ed1ab_1 conda-forge ml_dtypes 0.4.0 py312hf9745cd_2 conda-forge namex 0.0.8 pyhd8ed1ab_1 conda-forge nccl 2.24.3.1 hb92ee24_0 conda-forge ncurses 6.5 h2d0b736_2 conda-forge numpy 1.26.4 py312heda63a1_0 conda-forge openssl 3.4.0 h7b32b05_1 conda-forge opt_einsum 3.4.0 pyhd8ed1ab_1 conda-forge optree 0.13.1 py312h68727a3_1 conda-forge packaging 24.2 pyhd8ed1ab_2 conda-forge pip 24.3.1 pyh8b19718_2 conda-forge protobuf 5.27.5 py312h2ec8cdc_0 conda-forge pthread-stubs 0.4 hb9d3cd8_1002 conda-forge pycparser 2.22 pyh29332c3_1 conda-forge pygments 2.19.1 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha55dd90_7 conda-forge python 3.12.8 h04291be_102_cpython ramonaoptics python-flatbuffers 24.12.23 pyhe33e51e_0 conda-forge python_abi 3.12 5_cp312 conda-forge re2 2024.07.02 h9925aae_2 conda-forge requests 2.32.3 pyhd8ed1ab_1 conda-forge rich 13.9.4 pyhd8ed1ab_1 conda-forge setuptools 75.8.0 pyhff2d567_0 conda-forge six 1.17.0 pyhd8ed1ab_0 conda-forge snappy 1.2.1 h8bd8927_1 conda-forge tensorboard 2.17.1 pyhd8ed1ab_0 conda-forge tensorboard-data-server 0.7.0 py312hda17c39_2 conda-forge tensorflow 2.17.0 cuda120py312h02ad488_203 conda-forge tensorflow-base 2.17.0 cuda120py312hbec54f7_203 conda-forge tensorflow-estimator 2.17.0 cuda120py312hfa0f5ef_203 conda-forge termcolor 2.5.0 pyhd8ed1ab_1 conda-forge tk 8.6.13 xft_h3930020_2 mark.harfouche typing-extensions 4.12.2 hd8ed1ab_1 conda-forge typing_extensions 4.12.2 pyha770c72_1 conda-forge tzdata 2024b hc8b5060_0 conda-forge urllib3 2.3.0 pyhd8ed1ab_0 conda-forge werkzeug 3.1.3 pyhd8ed1ab_1 conda-forge wheel 0.45.1 pyhd8ed1ab_1 conda-forge wrapt 1.17.0 py312h66e93f0_0 conda-forge xorg-libx11 1.8.10 h4f16b4b_1 conda-forge xorg-libxau 1.0.12 hb9d3cd8_0 conda-forge xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge xorg-libxft 2.3.8 ha04879e_1 conda-forge xorg-libxrender 0.9.12 hb9d3cd8_0 conda-forge zipp 3.21.0 pyhd8ed1ab_1 conda-forge zstandard 0.23.0 py312hef9b889_1 conda-forge zstd 1.5.6 ha6fb4c9_0 conda-forge ```

so it has cudnn 9 and tensorflow 2.17.0 build 203

njzjz · 2025-01-14T06:10:18Z

Could you print

python -c "import tensorflow as tf;print(tf)"

Just want to confirm the TF you load is what you just installed.

alexfanqi · 2025-01-14T08:40:38Z

ah! yes, thank you for pointing that out. I didn't expect it is taking my local user version.

conda/conda#448

alexfanqi added the bug label Dec 26, 2024

alexfanqi changed the title ~~tensorflow-gpu 2.17.0 Could not find TensorRT~~ tensorflow-gpu 2.17.0 Could not find GPU Dec 27, 2024

hmaarrfk mentioned this issue Jan 12, 2025

How can we identify WSL (windows subsystem for linux) conda-forge/conda-forge.github.io#2419

Open

alexfanqi closed this as completed Jan 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tensorflow-gpu 2.17.0 Could not find GPU #410

tensorflow-gpu 2.17.0 Could not find GPU #410

alexfanqi commented Dec 26, 2024

alexfanqi commented Dec 27, 2024 •

edited

Loading

njzjz commented Jan 11, 2025

alexfanqi commented Jan 12, 2025 •

edited

Loading

hmaarrfk commented Jan 12, 2025

alexfanqi commented Jan 13, 2025 •

edited

Loading

hmaarrfk commented Jan 13, 2025

alexfanqi commented Jan 13, 2025 •

edited

Loading

hmaarrfk commented Jan 13, 2025

alexfanqi commented Jan 13, 2025 •

edited

Loading

hmaarrfk commented Jan 13, 2025

alexfanqi commented Jan 13, 2025

hmaarrfk commented Jan 13, 2025

hmaarrfk commented Jan 13, 2025

alexfanqi commented Jan 14, 2025

alexfanqi commented Jan 14, 2025 •

edited

Loading

hmaarrfk commented Jan 14, 2025

njzjz commented Jan 14, 2025

alexfanqi commented Jan 14, 2025 •

edited

Loading

tensorflow-gpu 2.17.0 Could not find GPU #410

tensorflow-gpu 2.17.0 Could not find GPU #410

Comments

alexfanqi commented Dec 26, 2024

Solution to issue cannot be found in the documentation.

Issue

Installed packages

Environment info

alexfanqi commented Dec 27, 2024 • edited Loading

njzjz commented Jan 11, 2025

alexfanqi commented Jan 12, 2025 • edited Loading

hmaarrfk commented Jan 12, 2025

alexfanqi commented Jan 13, 2025 • edited Loading

hmaarrfk commented Jan 13, 2025

alexfanqi commented Jan 13, 2025 • edited Loading

hmaarrfk commented Jan 13, 2025

alexfanqi commented Jan 13, 2025 • edited Loading

hmaarrfk commented Jan 13, 2025

alexfanqi commented Jan 13, 2025

hmaarrfk commented Jan 13, 2025

hmaarrfk commented Jan 13, 2025

alexfanqi commented Jan 14, 2025

alexfanqi commented Jan 14, 2025 • edited Loading

hmaarrfk commented Jan 14, 2025

njzjz commented Jan 14, 2025

alexfanqi commented Jan 14, 2025 • edited Loading

alexfanqi commented Dec 27, 2024 •

edited

Loading

alexfanqi commented Jan 12, 2025 •

edited

Loading

alexfanqi commented Jan 13, 2025 •

edited

Loading

alexfanqi commented Jan 13, 2025 •

edited

Loading

alexfanqi commented Jan 13, 2025 •

edited

Loading

alexfanqi commented Jan 14, 2025 •

edited

Loading

alexfanqi commented Jan 14, 2025 •

edited

Loading