Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set CUDA12 as default in GPU packages #21438

Merged
merged 17 commits into from
Jul 25, 2024
Merged

Set CUDA12 as default in GPU packages #21438

merged 17 commits into from
Jul 25, 2024

Conversation

yf711
Copy link
Contributor

@yf711 yf711 commented Jul 22, 2024 •
edited
Loading

Description

  • Swap cuda version 11.8/12.2 in GPU CIs
  • Set CUDA12 as default version in yamls of publishing nuget/python/java GPU packages
  • Suppress warnings as errors of flash_api.cc during ort win-build

Motivation and Context

@yf711 yf711 requested a review from a team as a code owner July 22, 2024 07:22
@yf711 yf711 requested review from snnn and jchen351 July 22, 2024 07:31
@jywu-msft
Copy link
Member

pls fix python lint errors

@jywu-msft
Copy link
Member

do we know why there are build warning/errors on CUDA build on Windows? @tianleiwu @jchen351 , was it encountered previously in CUDA12 pipeline?

215>D:\a_work\1\s\onnxruntime\contrib_ops\cuda\bert\flash_attention\flash_api.cc(97,27): error C2220: the following warning is treated as an error [D:\a_work\1\b\RelWithDebInfo\onnxruntime_providers_cuda.vcxproj]
##[warning]onnxruntime\contrib_ops\cuda\bert\flash_attention\flash_api.cc(97,27): Warning C4616: #pragma warning: warning number '2220' not a valid compiler warning
215>D:\a_work\1\s\onnxruntime\contrib_ops\cuda\bert\flash_attention\flash_api.cc(97,27): warning C4616: #pragma warning: warning number '2220' not a valid compiler warning [D:\a_work\1\b\RelWithDebInfo\onnxruntime_providers_cuda.vcxproj]
##[warning]onnxruntime\contrib_ops\cuda\bert\flash_attention\flash_api.cc(99,14): Warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
215>D:\a_work\1\s\onnxruntime\contrib_ops\cuda\bert\flash_attention\flash_api.cc(99,14): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a_work\1\b\RelWithDebInfo\onnxruntime_providers_cuda.vcxproj]
##[warning]onnxruntime\contrib_ops\cuda\bert\flash_attention\flash_api.cc(100,14): Warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
215>

@snnn
Copy link
Member

snnn commented Jul 24, 2024

Our nightly pipelines do not treat warning as errors. So the problems have been there for a while.

@tianleiwu
Copy link
Contributor

tianleiwu commented Jul 24, 2024 •
edited
Loading

do we know why there are build warning/errors on CUDA build on Windows? @tianleiwu @jchen351 , was it encountered previously in CUDA12 pipeline?

I did not see such warnings/error in my local build. I use command like

call build.bat --cmake_generator "Visual Studio 17 2022" --config Release --build_dir build\cuda12 --build_wheel ^
          --parallel --build_shared_lib --use_cuda --cuda_version "12.5" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5" ^
	  --cudnn_home "C:\nvidia\CuDNN\9.1.1.17_cuda12" ^
	  --use_tensorrt --tensorrt_home "C:\nvidia\TensorRT\10.0.1.6.cuda-12.4" --use_tensorrt_builtin_parser ^
          --parallel 4 --nvcc_threads 1 ^
	  --skip_tests ^
          --use_binskim_compliant_compile_flags ^
          --build_csharp --enable_pybind --use_dml ^
	  --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON CMAKE_CUDA_ARCHITECTURES=89

maybe some setting is different in CI pipeline.

@aciddelgado to fix the warnings in flash_api.cc.

@snnn
Copy link
Member

snnn commented Jul 24, 2024

Tianlei, the pipeline uses:

python.exe tools\ci_build\build.py --config RelWithDebInfo --build_dir xxxx --skip_submodule_sync --build_csharp --update --parallel --use_binskim_compliant_compile_flags --cmake_generator "Visual Studio 17 2022" --build_shared_lib --enable_onnx_tests --enable_pybind --build_java --build_nodejs --use_cuda --cuda_home=D:\a_work_temp\v12.2 --enable_cuda_profiling --enable_transformers_tool_test --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=86 --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON --cmake_extra_defines onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON

@snnn
Copy link
Member

snnn commented Jul 24, 2024 •
edited
Loading

The TRT CI pipeline still has CUDA 11 in PATH

D:\a\_work\1\b>set PATH=D:\a\_work\_temp\TensorRT-10.2.0.19.Windows10.x86_64.cuda-11.8\lib;C:\Users\cloudtest\AppData\Roaming\Python\Python38\Scripts;C:\ToolCache\Python\3.8.10\x64\Scripts;C:\ToolCache\Python\3.8.10\x64;C:\vss-agent\3.241.0\externals\git\cmd;C:\vss-agent\3.241.0\externals\git\mingw64\bin;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\VC\VCPackages;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Team Tools\DiagnosticsHub\Collector;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\CodeCoverage.Console;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\Tools;C:\Users\cloudtest\AppData\Roaming\Python\Python38\Scripts;C:\ToolCache\Python\3.8.10\x64\Scripts;C:\ToolCache\Python\3.8.10\x64;C:\vss-agent\3.241.0\externals\git\cmd;C:\vss-agent\3.241.0\externals\git\mingw64\bin;C:\ToolCache\Java_Temurin-Hotspot_jdk\11.0.24-8\x64\bin;C:\npm\prefix;C:\ProgramData\Chocolatey\bin;C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\dotnet;C:\Program Files\PowerShell\7;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn;C:\Program Files\Microsoft SQL Server\150\Tools\Binn;C;C:\Program Files\nodejs;C:\Program Files\CMake\bin;C:\Program Files\Git\cmd;C:\Users\cloudtest\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\VC\Linux\bin\ConnectionManagerExe;D:\a\_work\_temp\v11.8\bin;D:\a\_work\_temp\v11.8\extras\CUPTI\lib64;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\extras\CUPTI\lib64

@yf711 yf711 merged commit ebcb707 into main Jul 25, 2024
99 checks passed
@yf711 yf711 deleted the yifanl/default_cuda_12 branch July 25, 2024 17:17
@tianleiwu
Copy link
Contributor

Could we upgrade cuDNN 8 to cuDNN 9 in all pipelines?

PyTorch 2.4 uses cudnn 9 for both cuda 11.8 and cuda 12.4. I think we can also move to cudnn 9 completely (including for CUDA 11).

yf711 added a commit that referenced this pull request Jul 29, 2024
@yf711
Set CUDA12 as default in GPU packages (#21438)
3e58282 
### Description
* Swap cuda version 11.8/12.2 in GPU CIs
* Set CUDA12 as default version in yamls of publishing nuget/python/java
GPU packages
* Suppress warnings as errors of flash_api.cc during ort win-build
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tianleiwu tianleiwu tianleiwu left review comments

@snnn snnn snnn left review comments

@jchen351 jchen351 jchen351 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@yf711 @jywu-msft @snnn @tianleiwu @jchen351