Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding DML to python cuda package #22606

Merged
merged 62 commits into from
Dec 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
1fe6bae
Adding new Python package testing pipeline for CUda Alt
jchen351 Oct 24, 2024
e973742
Remove Linux_Test_GPU_x86_64_stage from stage final
jchen351 Oct 24, 2024
2a5be8c
Adding dependencies
jchen351 Oct 24, 2024
7690e26
Distinguish between DML and the generic 'GPU' term. This is needed fo…
pranavsharma Oct 25, 2024
a0918e0
Update dml pkg with python
jchen351 Oct 25, 2024
10dcad9
Update dml pkg with python
jchen351 Oct 25, 2024
3b9630e
Update dml pkg with python
jchen351 Oct 25, 2024
381f55f
Address code review comment
pranavsharma Oct 25, 2024
b671f41
Merge remote-tracking branch 'origin/package_dml' into Cjian/pydml
jchen351 Oct 25, 2024
8196b65
remove --enable_wcos and --cmake_extra_defines "CMAKE_SYSTEM_VERSION=…
jchen351 Oct 25, 2024
6dd4694
Merge branch 'main' into Cjian/pytest
jchen351 Oct 25, 2024
5ba0c12
Split DML test out of cuda
jchen351 Oct 26, 2024
847a7ff
Merge branch 'Cjian/pytest' into Cjian/pydml
jchen351 Oct 26, 2024
36c5bde
Split DML test out of cuda
jchen351 Oct 26, 2024
7113a15
Split DML test out of cuda
jchen351 Oct 26, 2024
bdce800
Split DML test out of cuda
jchen351 Oct 26, 2024
3755562
Split DML test out of cuda
jchen351 Oct 26, 2024
7a2cd9c
Split DML test out of cuda
jchen351 Oct 26, 2024
5188f4c
Split DML test out of cuda
jchen351 Oct 26, 2024
e334bfe
Merge remote-tracking branch 'origin/Cjian/pytest' into Cjian/pytest
jchen351 Oct 26, 2024
ff73270
lintrunner
jchen351 Oct 26, 2024
4c29c54
Merge branch 'Cjian/pytest' into Cjian/pydml
jchen351 Oct 26, 2024
a9b5a90
update ruff
jchen351 Oct 28, 2024
775ff5b
Merge branch 'Cjian/pytest' into Cjian/pydml
jchen351 Oct 28, 2024
9557f40
update cuda
jchen351 Oct 28, 2024
42c8702
exclude test_convtranspose_autopad_same_cuda
jchen351 Oct 28, 2024
2007bb3
update os.environ["ORT_ONNX_BACKEND_EXCLUDE_PROVIDERS"]
jchen351 Oct 29, 2024
a4dddc5
# exclude DML EP when CUDA test is running.
jchen351 Oct 29, 2024
b11b993
Distinguish between DML and the generic 'GPU' term. This is needed fo…
pranavsharma Oct 30, 2024
b3fa3b2
Merge remote-tracking branch 'origin/dml_device' into Cjian/pydml
jchen351 Oct 30, 2024
b66393c
Move py_packaging_test_step into a template
jchen351 Oct 30, 2024
9d61bf4
parameters
jchen351 Oct 30, 2024
e4c3269
$(Agent.TempDirectory)
jchen351 Oct 30, 2024
a1d0985
Merge branch 'main' into Cjian/pydml
jchen351 Nov 1, 2024
2470bd0
Lintrunner -a
jchen351 Nov 1, 2024
4993088
Update DNNL CI python to 310
jchen351 Nov 1, 2024
12b5ff6
Merge branch 'Cjian/dnnl' into Cjian/pydml
jchen351 Nov 1, 2024
f963b6e
Replace reference to python 3.8 with python 3.19
jchen351 Nov 1, 2024
e0b895b
Merge branch 'Cjian/dnnl' into Cjian/pydml
jchen351 Nov 1, 2024
ba7dd01
Replace reference to python 3.8 with python 3.10
jchen351 Nov 1, 2024
c16a1d4
Merge branch 'Cjian/dnnl' into Cjian/pydml
jchen351 Nov 1, 2024
b8b98ea
Enable Win CUDA python test
jchen351 Nov 4, 2024
bcd173b
Enable Win CUDA python test
jchen351 Nov 4, 2024
720fce9
Disable 3 failed test due to upgrading to python 3.10
jchen351 Nov 4, 2024
94e152c
Merge branch 'Cjian/dnnl' into Cjian/pydml
jchen351 Nov 4, 2024
3a66769
Using Iterable instead of list
jchen351 Nov 4, 2024
5bb6617
Undo Iterable instead of list
jchen351 Nov 4, 2024
a48cccd
linrunner -a
jchen351 Nov 4, 2024
0efb9fc
Merge branch 'Cjian/dnnl' into Cjian/pydml
jchen351 Nov 4, 2024
015b32b
exclude failed cuda 12 tests
jchen351 Nov 5, 2024
ef629ad
Adding verbose to run onnx_backend_test_series.py
jchen351 Nov 5, 2024
bc4e825
Adding verbose to run onnx_backend_test_series.py
jchen351 Nov 5, 2024
cb3b48b
Merge branch 'Cjian/enable_cuda_py_test' into Cjian/pydml
jchen351 Nov 5, 2024
f9f0d46
Merge branch 'main' into Cjian/pydml
jchen351 Nov 5, 2024
d462901
rolling back dml texts exclusions,
jchen351 Nov 5, 2024
099d7dd
remove CudaVersion: ${{ parameters.cuda_version }}
jchen351 Nov 5, 2024
057ae40
Merge branch 'Cjian/enable_cuda_py_test' into Cjian/pydml
jchen351 Nov 5, 2024
4cd9ecd
Merge branch 'main' into Cjian/pydml
jchen351 Nov 6, 2024
01a4359
Merge branch 'main' into Cjian/pydml
jchen351 Nov 13, 2024
82d3f6e
Merge remote-tracking branch 'origin/main' into Cjian/pydml
jchen351 Nov 16, 2024
bc8e9d7
Exclude failed cuda test that running during the DML testing
jchen351 Nov 18, 2024
ce4666d
Reduce parallel counts
jchen351 Nov 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 27 additions & 14 deletions onnxruntime/test/python/onnx_backend_test_series.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ def load_jsonc(basename: str):
return json.loads("\n".join(lines))


def create_backend_test(test_name=None):
def create_backend_test(devices: list[str], test_name=None):
snnn marked this conversation as resolved.
Show resolved Hide resolved
"""Creates an OrtBackendTest and adds its TestCase's to global scope so unittest will find them."""

overrides = load_jsonc("onnx_backend_test_series_overrides.jsonc")
Expand All @@ -126,30 +126,29 @@ def create_backend_test(test_name=None):
else:
filters = load_jsonc("onnx_backend_test_series_filters.jsonc")
current_failing_tests = apply_filters(filters, "current_failing_tests")

if platform.architecture()[0] == "32bit":
current_failing_tests += apply_filters(filters, "current_failing_tests_x86")

if backend.supports_device("DNNL"):
if backend.supports_device("DNNL") or "DNNL" in devices:
current_failing_tests += apply_filters(filters, "current_failing_tests_DNNL")

if backend.supports_device("NNAPI"):
if backend.supports_device("NNAPI") or "NNAPI" in devices:
current_failing_tests += apply_filters(filters, "current_failing_tests_NNAPI")

if backend.supports_device("OPENVINO_GPU"):
if backend.supports_device("OPENVINO_GPU") or "OPENVINO_GPU" in devices:
current_failing_tests += apply_filters(filters, "current_failing_tests_OPENVINO_GPU")

if backend.supports_device("OPENVINO_CPU"):
if backend.supports_device("OPENVINO_CPU") or "OPENVINO_CPU" in devices:
current_failing_tests += apply_filters(filters, "current_failing_tests_OPENVINO_CPU_FP32")
current_failing_tests += apply_filters(filters, "current_failing_tests_OPENVINO_CPU_FP16")

if backend.supports_device("OPENVINO_NPU"):
if backend.supports_device("OPENVINO_NPU") or "OPENVINO_NPU" in devices:
current_failing_tests += apply_filters(filters, "current_failing_tests_OPENVINO_NPU")

if backend.supports_device("OPENVINO"):
if backend.supports_device("OPENVINO") or "OPENVINO" in devices:
current_failing_tests += apply_filters(filters, "current_failing_tests_OPENVINO_opset18")

if backend.supports_device("MIGRAPHX"):
if backend.supports_device("MIGRAPHX") or "MIGRAPHX" in devices:
current_failing_tests += apply_filters(filters, "current_failing_tests_MIGRAPHX")

if backend.supports_device("WEBGPU"):
Expand All @@ -158,8 +157,16 @@ def create_backend_test(test_name=None):
# Skip these tests for a "pure" DML onnxruntime python wheel. We keep these tests enabled for instances where both DML and CUDA
# EPs are available (Windows GPU CI pipeline has this config) - these test will pass because CUDA has higher precedence than DML
# and the nodes are assigned to only the CUDA EP (which supports these tests)
if backend.supports_device("DML") and not backend.supports_device("GPU"):
if (backend.supports_device("DML") and not backend.supports_device("GPU")) or "DML" in devices:
current_failing_tests += apply_filters(filters, "current_failing_tests_pure_DML")
# exclude CUDA EP when DML test is running.
os.environ["ORT_ONNX_BACKEND_EXCLUDE_PROVIDERS"] = "TensorrtExecutionProvider,CUDAExecutionProvider"
elif backend.supports_device("DML") and "DML" not in devices:
# exclude DML EP when CUDA test is running.
os.environ["ORT_ONNX_BACKEND_EXCLUDE_PROVIDERS"] = "TensorrtExecutionProvider,DmlExecutionProvider"
else:
# exclude TRT EP temporarily and only test CUDA EP to retain previous behavior
os.environ["ORT_ONNX_BACKEND_EXCLUDE_PROVIDERS"] = "TensorrtExecutionProvider"

filters = (
current_failing_tests
Expand All @@ -172,9 +179,6 @@ def create_backend_test(test_name=None):
backend_test.exclude("(" + "|".join(filters) + ")")
print("excluded tests:", filters)

# exclude TRT EP temporarily and only test CUDA EP to retain previous behavior
os.environ["ORT_ONNX_BACKEND_EXCLUDE_PROVIDERS"] = "TensorrtExecutionProvider"

# import all test cases at global scope to make
# them visible to python.unittest.
globals().update(backend_test.enable_report().test_cases)
Expand All @@ -199,6 +203,15 @@ def parse_args():
help="Only run tests that match this value. Matching is regex based, and '.*' is automatically appended",
)

parser.add_argument(
"--devices",
type=str,
choices=["CPU", "CUDA", "MIGRAPHX", "DNNL", "DML", "OPENVINO_GPU", "OPENVINO_CPU", "OPENVINO_NPU", "OPENVINO"],
nargs="+", # allows multiple values
default=["CPU"], # default to ["CPU"] if no input is given
help="Select one or more devices CPU, CUDA, MIGRAPHX, DNNL, DML, OPENVINO_GPU, OPENVINO_CPU, OPENVINO_NPU, OPENVINO",
)

# parse just our args. python unittest has its own args and arg parsing, and that runs inside unittest.main()
parsed, unknown = parser.parse_known_args()
sys.argv = sys.argv[:1] + unknown
Expand All @@ -209,5 +222,5 @@ def parse_args():
if __name__ == "__main__":
args = parse_args()

create_backend_test(args.test_name)
create_backend_test(args.devices, args.test_name)
unittest.main()
Original file line number Diff line number Diff line change
Expand Up @@ -750,6 +750,13 @@
"^test_reduce_log_sum_empty_set_cpu",
"^test_reduce_log_sum_exp_empty_set_cpu",
"^test_reduce_prod_empty_set_cpu",
// Bug: DML EP some how executes these CUDA tests and failed
// TODO: Remove these tests when DML EP is fixed
"^test_convtranspose_autopad_same_cuda",
"^test_asin_example_cuda",
"^test_dynamicquantizelinear_cuda",
"^test_dynamicquantizelinear_expanded_cuda",
"^test_reduce_min_empty_set_cuda",
//Bug: DML EP does not execute operators with an empty input tensor
//TODO: Resolve as a graph implementation that returns a constant inf tensor with appropriate strides
"^test_reduce_min_empty_set_cpu"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
parameters:
- name: EP_NAME
type: string
default: CPU

- name: PYTHON_VERSION
type: string

steps:
- powershell: |
python -m pip uninstall -y onnxruntime onnxruntime-gpu -qq
Get-ChildItem -Path $(Build.ArtifactStagingDirectory)/*cp${{ replace(parameters.PYTHON_VERSION,'.','') }}*.whl | foreach {pip --disable-pip-version-check install --upgrade $_.fullname tabulate}
mkdir -p $(Agent.TempDirectory)\ort_test_data
Copy-Item -Path $(Build.sourcesDirectory)/onnxruntime/test/python/onnx_backend_test_series.py -Destination $(Agent.TempDirectory)\ort_test_data
Copy-Item -Recurse -Path $(Build.sourcesDirectory)/onnxruntime/test/testdata -Destination $(Agent.TempDirectory)\ort_test_data
cd $(Agent.TempDirectory)\ort_test_data
python onnx_backend_test_series.py --devices ${{ parameters.EP_NAME }} -v
cd $(Agent.TempDirectory)
Remove-Item -Path $(Agent.TempDirectory)\ort_test_data -Recurse -Force
workingDirectory: '$(Build.sourcesDirectory)'
displayName: 'Run Python Tests with ${{ parameters.EP_NAME }} EP'
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ stages:
PYTHON_VERSION: ${{ python_version }}
EP_NAME: gpu
CudaVersion: ${{ parameters.cuda_version }}
EP_BUILD_FLAGS: --enable_lto --cuda_home=$(Agent.TempDirectory)\v${{ parameters.cuda_version }} --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
EP_BUILD_FLAGS: --use_dml --enable_lto --cuda_home=$(Agent.TempDirectory)\v${{ parameters.cuda_version }} --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80"
use_tensorrt: True

- ${{ if eq(parameters.enable_linux_cuda, true) }}:
Expand Down
27 changes: 14 additions & 13 deletions tools/ci_build/github/azure-pipelines/stages/py-win-gpu-stage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ parameters:
- Release
- RelWithDebInfo
- MinSizeRel

- name: use_tensorrt
type: boolean
default: false
Expand Down Expand Up @@ -134,7 +134,7 @@ stages:
--cmake_generator "$(VSGenerator)"
--enable_pybind
--enable_onnx_tests
--parallel --use_binskim_compliant_compile_flags --update --build
--parallel 4 --use_binskim_compliant_compile_flags --update --build
$(TelemetryOption) ${{ parameters.BUILD_PY_PARAMETERS }} ${{ parameters.EP_BUILD_FLAGS }} ${{ variables.trt_build_flag }}
workingDirectory: '$(Build.BinariesDirectory)'

Expand Down Expand Up @@ -206,19 +206,20 @@ stages:
DownloadTRT: ${{ parameters.use_tensorrt }}

- task: PowerShell@2
displayName: 'Install ONNX'
displayName: 'Install Third Party Dependencies'
inputs:
filePath: '$(Build.SourcesDirectory)/tools/ci_build/github/windows/install_third_party_deps.ps1'
workingDirectory: '$(Build.BinariesDirectory)'
arguments: -cpu_arch x64 -install_prefix $(Build.BinariesDirectory)\${{ parameters.cmake_build_type }}\installed -build_config ${{ parameters.cmake_build_type }}

- powershell: |
python -m pip uninstall -y onnxruntime onnxruntime-gpu -qq
Get-ChildItem -Path $(Build.ArtifactStagingDirectory)/*cp${{ replace(parameters.PYTHON_VERSION,'.','') }}*.whl | foreach {pip --disable-pip-version-check install --upgrade $_.fullname tabulate}
mkdir -p $(Agent.TempDirectory)\ort_test_data
Copy-Item -Path $(Build.sourcesDirectory)/onnxruntime/test/python/onnx_backend_test_series.py -Destination $(Agent.TempDirectory)\ort_test_data
Copy-Item -Recurse -Path $(Build.sourcesDirectory)/onnxruntime/test/testdata -Destination $(Agent.TempDirectory)\ort_test_data
cd $(Agent.TempDirectory)\ort_test_data
python onnx_backend_test_series.py
workingDirectory: '$(Build.sourcesDirectory)'
displayName: 'Run Python Tests'
- template: jobs/steps/py_packaging_test_step.yml
parameters:
EP_NAME: DML
PYTHON_VERSION: ${{ parameters.PYTHON_VERSION }}

- template: jobs/steps/py_packaging_test_step.yml
parameters:
EP_NAME: CUDA
PYTHON_VERSION: ${{ parameters.PYTHON_VERSION }}


Loading