Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows CI build steps template #17263

Merged
merged 15 commits into from
Aug 24, 2023
Merged

Windows CI build steps template #17263

merged 15 commits into from
Aug 24, 2023

Conversation

mszhanyi
Copy link
Contributor

@mszhanyi mszhanyi commented Aug 23, 2023 •
edited
Loading

Description

  1. New windows ci build steps template.
  2. Remove useless variables.

Motivation and Context

  1. Make it easier to apply build cache to all windows CIs.
  2. Other team's devs only need to take care of build options

###Comparision
Before:

onnxruntime/tools/ci_build/github/azure-pipelines/win-gpu-tensorrt-ci-pipeline.yml

Lines 19 to 82 in 9f21f69

- task: UsePythonVersion@0
inputs:
versionSpec: '3.8'
addToPath: true
architecture: $(buildArch)
- task: BatchScript@1
displayName: 'setup env'
inputs:
filename: '$(Build.SourcesDirectory)\tools\ci_build\github\windows\$(EnvSetupScript)'
modifyEnvironment: true
workingFolder: '$(Build.BinariesDirectory)'
- script: |
python -m pip install -q setuptools wheel numpy
workingDirectory: '$(Build.BinariesDirectory)'
displayName: 'Install python modules'
- template: templates/download-deps.yml
- task: PythonScript@0
displayName: 'Update deps.txt'
inputs:
scriptPath: $(Build.SourcesDirectory)/tools/ci_build/replace_urls_in_deps.py
arguments: --new_dir $(Build.BinariesDirectory)/deps
workingDirectory: $(Build.BinariesDirectory)
- powershell: |
azcopy.exe cp --recursive "https://lotusscus.blob.core.windows.net/models/cuda_sdk/v11.8" $(Agent.TempDirectory)
displayName: 'Download CUDA SDK'
- task: BatchScript@1
displayName: 'setup env'
inputs:
filename: '$(Build.SourcesDirectory)\tools\ci_build\github\windows\$(EnvSetupScript)'
modifyEnvironment: true
workingFolder: '$(Build.BinariesDirectory)'
- task: PowerShell@2
displayName: 'Install ONNX'
inputs:
filePath: '$(Build.SourcesDirectory)/tools/ci_build/github/windows/install_third_party_deps.ps1'
workingDirectory: '$(Build.BinariesDirectory)'
arguments: -cpu_arch x64 -install_prefix $(Build.BinariesDirectory)\$(BuildConfig)\installed -build_config $(BuildConfig)
- task: PythonScript@0
displayName: 'Generate cmake config'
inputs:
scriptPath: '$(Build.SourcesDirectory)\tools\ci_build\build.py'
arguments: '--config $(BuildConfig) --build_dir $(Build.BinariesDirectory) --skip_submodule_sync --build_shared_lib --update --cmake_generator "Visual Studio 17 2022" --build_wheel --enable_onnx_tests --use_tensorrt --tensorrt_home="C:\local\TensorRT-8.6.1.6.Windows10.x86_64.cuda-11.8" --cuda_version=11.8 --cuda_home="$(Agent.TempDirectory)\v11.8" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=75'
workingDirectory: '$(Build.BinariesDirectory)'
- task: VSBuild@1
displayName: 'Build'
inputs:
solution: '$(Build.BinariesDirectory)\$(BuildConfig)\onnxruntime.sln'
platform: 'x64'
configuration: $(BuildConfig)
msbuildArgs: $(MsbuildArguments)
msbuildArchitecture: $(buildArch)
maximumCpuCount: true
logProjectEvents: false
workingFolder: '$(Build.BinariesDirectory)\$(BuildConfig)'
createLogFile: true

After:

onnxruntime/tools/ci_build/github/azure-pipelines/win-gpu-tensorrt-ci-pipeline.yml

Lines 35 to 54 in b4c1f22

- template: templates/jobs/win-ci-prebuild-steps.yml
parameters:
EnvSetupScript: $(EnvSetupScript)
DownloadCUDA: true
BuildArch: $(buildArch)
BuildConfig: $(BuildConfig)
MachinePool: 'onnxruntime-Win2022-GPU-T4'
WithCache: true
Today: $(Today)
- template: templates/jobs/win-ci-build-steps.yml
parameters:
WithCache: True
Today: $(TODAY)
AdditionalKey: "gpu-tensorrt | $(BuildConfig)"
BuildPyArguments: '--config $(BuildConfig) --build_dir $(Build.BinariesDirectory) --skip_submodule_sync --build_shared_lib --update --cmake_generator "Visual Studio 17 2022" --build_wheel --enable_onnx_tests --use_tensorrt --tensorrt_home="C:\local\TensorRT-8.6.1.6.Windows10.x86_64.cuda-11.8" --cuda_version=11.8 --cuda_home="$(Agent.TempDirectory)\v11.8" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=75'
MsbuildArguments: $(MsbuildArguments)
BuildArch: $(buildArch)
Platform: 'x64'
BuildConfig: $(BuildConfig)

@mszhanyi mszhanyi requested a review from a team as a code owner August 23, 2023 08:14
@mszhanyi mszhanyi marked this pull request as draft August 23, 2023 10:10
@mszhanyi mszhanyi closed this Aug 24, 2023
@mszhanyi mszhanyi reopened this Aug 24, 2023
@mszhanyi mszhanyi marked this pull request as ready for review August 24, 2023 07:19
@mszhanyi mszhanyi merged commit 756eda2 into main Aug 24, 2023
@mszhanyi mszhanyi deleted the zhanyi/wincachetask branch August 24, 2023 21:58
@mszhanyi mszhanyi mentioned this pull request Aug 25, 2023
Merged
mszhanyi added a commit that referenced this pull request Aug 25, 2023
@mszhanyi
Readd Tests in Window GPU Reduced Ops workflow (#17294)
9cd33e0 
### Description
Add single test step in Window GPU Reduced Ops workflow


### Motivation and Context
The old workflow's building and testing were running in one command.
In PR #17263, the test step was removed by mistake.
So, readd it.
How to consolidate the test step is in consideration.
@MaxMood96 MaxMood96 mentioned this pull request Sep 3, 2023
Closed
kleiti pushed a commit to kleiti/onnxruntime that referenced this pull request Mar 22, 2024
@mszhanyi @kleiti
Windows CI build steps template (microsoft#17263)
029a097 
### Description
1. New windows ci build steps template.
2. Remove useless variables.

### Motivation and Context
1. Make it easier to apply build cache to all windows CIs.
2. Other team's devs only need to take care of build options


###Comparision
Before: 

https://github.com/microsoft/onnxruntime/blob/9f21f694cf939a960f5b516ba5ff9675fc3ec6d1/tools/ci_build/github/azure-pipelines/win-gpu-tensorrt-ci-pipeline.yml#L19-L82

After:
https://github.com/microsoft/onnxruntime/blob/b4c1f2261bbc282139e46c0d68520f65940883f3/tools/ci_build/github/azure-pipelines/win-gpu-tensorrt-ci-pipeline.yml#L35-L54
kleiti pushed a commit to kleiti/onnxruntime that referenced this pull request Mar 22, 2024
@mszhanyi @kleiti
Readd Tests in Window GPU Reduced Ops workflow (microsoft#17294)
6f77ad1 
### Description
Add single test step in Window GPU Reduced Ops workflow


### Motivation and Context
The old workflow's building and testing were running in one command.
In PR microsoft#17263, the test step was removed by mistake.
So, readd it.
How to consolidate the test step is in consideration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@snnn snnn snnn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mszhanyi @snnn