CUDA does not load on Windows

[koush]

Describe the issue

I want to preface that ONNX runtime + CUDA provider works fine on Ubuntu 22.04. However, the same machine booted into windows results in the following error (while PyTorch works):

2024-09-05 08:30:27.7810630 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\koush\Desktop\work\.venv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

2024-09-05 08:30:27.7882169 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:965 onnxruntime::python::CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Require cuDNN 9.* and CUDA 12.*, and the latest MSVC runtime. Please install all dependencies as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.

I have CUDA, CUDNN, and zlib installed and in path. MSVC runtime is also installed.

This was a clean windows install.

I think there's some dependency missing from the documentation.

To reproduce

Follow the onnxruntime-gpu instructions for windows. Try to load the CUDA provider.

Urgency

No response

Platform

Windows

OS Version

11

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.19.2 or 1.18.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 12.4, CUDA 12.6, CUDA 11.8

[tianleiwu]

What's your PyTorch version? Try install latest PyTorch for cuda 12 (older version of PyTorch uses cudnn 8, which will not work properly with latest onnxruntime-gpu).

If you install other onnxruntime packages (like onnxruntime or onnxruntime-directml), you might uninstall them first.

Need add cuda and cudnn bin directories to PATH environment variable. Example is like

set PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin;C:\nvidia\cudnn-windows-x86_64-9.3.0.75_cuda12-archive\bin;%PATH%

Recommend the following combination:

• CUDA 12.4, 12.5.1 or 12.6 (Note that 12.6 is good for inference, but cannot build onnxruntime-gpu from source for now).
• cuDNN 9.3 for cuda 12: unzip https://developer.download.nvidia.com/compute/cudnn/redist/cudnn/windows-x86_64/cudnn-windows-x86_64-9.3.0.75_cuda12-archive.zip to a local directory.
• onnxruntime-gpu 1.19.2: pip install onnxruntime-gpu==1.19.2
• PyTorch 2.4 for cuda 12.4: pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
• Latest VC Runtime

If you cannot find the root cause, try Dependency Walker to open the DLL (You can use pip show onnxruntime-gpu to find the site package localtion, the DLL is under site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll). For example, the following shows that cuDNN dll has a question mark, which means it is not found based on current path setting:

[koush]

I was testing with latest torch.

I'll check this combination sholrtly, but why is PyTorch necessary in the combination you listed Is that required my onmx runtime?

[tianleiwu]

why is PyTorch necessary in the combination you listed Is that required my onmx runtime?

PyTorch is not required to run onnxruntime.
However, if you use both of them in same process, make sure they are compatible.

[koush]

why is PyTorch necessary in the combination you listed Is that required my onmx runtime?

PyTorch is not required to run onnxruntime. However, if you use both of them in same process, make sure they are compatible.

Understood. I was not using them in the same process. I set up a second virtual env for verification of the CUDA/CUDNN installs.

[koush]

I installed latest nvidia drivers, CUDA 12.6, and CUDNN 9.4 (I also had CUDNN 8.x installed at some point too). I noticed that the PATH set by NVIDIA's installer was using Program Files\NVIDIA\CUDNN\bin when it needed Progra Files\NVIDIA\CUDNN\bin\12.6 which is where the DLLS are located.

[tianleiwu]

That's good finding.

When I installed cudnn 9.4 with setup file, the DLLs are in
C:\Program Files\NVIDIA\CUDNN\v9.4\bin\11.8
C:\Program Files\NVIDIA\CUDNN\v9.4\bin\12.6
And PATH has "C:\Program Files\NVIDIA\CUDNN\v9.4\bin", so I need manually change the PATH setting according to cuda 12 version "C:\Program Files\NVIDIA\CUDNN\v9.4\bin\12.6".

I usually use the tarball (zip file) to install cuDNN, which does not have such issue.