OpenVINO-EP v4.0 Release PR with OpenVINO 2022.1 #11025

MaajidKhan · 2022-03-28T17:46:11Z

Description:
OpenVINO-EP v4.0 Release PR with OpenVINO 2022.1

Motivation and Context

OpenVINO-EP now supports the latest OpenVINO 2022.1 version
OpenVINO-EP is now updated to use the new OpenVINO 2.0 API's starting from this release
Backward compatibility support for older OpenVINO versions (OV 2021.3, OV 2021.4) is available
New code design changes introduced to have cleaner code structure for different OpenVINO versions
Opset 13 compliance w.r.t OpenVINO (More operators added)
New features added (GPU opencl throttling, device type checks)

->Added ov-ep 2022.1 flow ->Validated CPU Unit tests with OV Master using onnxruntime_test_all unit tests. Signed-off-by: MaajidKhan <[email protected]>

Refer: https://jira.devtools.intel.com/browse/CVS-60310 Signed-off-by: MaajidKhan <[email protected]>

->Enable Resize op for iGPU ->Enable Add op for iGPU Signed-off-by: MaajidKhan <[email protected]>

->Removing some conditions from GetCapability() which are now not required. (Removed conditions for OV version support less than 2021.2) Signed-off-by: MaajidKhan <[email protected]>

Signed-off-by: MaajidKhan <[email protected]>

*Added GPU Throttling feature for iGPU's. when user enables it as a runtime option, it helps in reducing overall CPU usage of the application *Added changes to exercise this option using onnxruntime_perf_test application. Signed-off-by: MaajidKhan <[email protected]>

Signed-off-by: MaajidKhan <[email protected]>

->Handling corner cases for device_type checks Signed-off-by: MaajidKhan <[email protected]>

->Enabled Few ops ->Added Debug info for case 3b in getcapability() Signed-off-by: MaajidKhan <[email protected]>

->Added ov-ep 2022.1 flow ->Validated CPU Unit tests with OV Master using onnxruntime_test_all unit tests. Signed-off-by: MaajidKhan <[email protected]>

Refer: https://jira.devtools.intel.com/browse/CVS-60310 Signed-off-by: MaajidKhan <[email protected]>

->Enable Resize op for iGPU ->Enable Add op for iGPU Signed-off-by: MaajidKhan <[email protected]>

->Removing some conditions from GetCapability() which are now not required. (Removed conditions for OV version support less than 2021.2) Signed-off-by: MaajidKhan <[email protected]>

Signed-off-by: MaajidKhan <[email protected]>

*Added GPU Throttling feature for iGPU's. when user enables it as a runtime option, it helps in reducing overall CPU usage of the application *Added changes to exercise this option using onnxruntime_perf_test application. Signed-off-by: MaajidKhan <[email protected]>

Signed-off-by: MaajidKhan <[email protected]>

oliviajain · 2022-04-01T00:07:02Z

onnxruntime/core/providers/openvino/openvino_execution_provider.cc

+  //using ie_core capability GetAvailableDevices to fetch list of devices plugged in
+  bool device_found = false;
+  bool device_id_found = false;
+  auto available_devices = openvino_ep::BackendManager::GetGlobalContext().ie_core.GetAvailableDevices();


Is there a way to silence the output from this call? If the inference takes long, the user only sees below for some time:
[E:] [BSL] found 0 ioexpander device

unfortunately, there is not much we can do to disable this as its coming from the OpenVINO API's. I will create a ticket with OpenVINO team to see if we can get this print removed from the next OpenVINO release (2022.2)

->Disabled this model for openvino. The test is failing in Internal_CI pipelines. Signed-off-by: MaajidKhan <[email protected]>

Signed-off-by: MaajidKhan <[email protected]>

jywu-msft · 2022-04-05T15:33:23Z

/azp run Linux OpenVINO CI Pipeline

azure-pipelines · 2022-04-05T15:33:34Z

Azure Pipelines successfully started running 1 pipeline(s).

jywu-msft · 2022-04-05T16:18:25Z

/azp run MacOS NoContribops CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows WebAssembly CI Pipeline, orttraining-amd-gpu-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, Linux CPU CI Pipeline

jywu-msft · 2022-04-05T16:18:42Z

/azp run Linux CPU Minimal Build E2E CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-python-checks-ci-pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux CPU x64 NoContribops CI Pipeline, Linux CPU x64 NoContribops CI Pipeline, Linux GPU CI Pipeline

azure-pipelines · 2022-04-05T16:18:59Z

Azure Pipelines successfully started running 8 pipeline(s).

azure-pipelines · 2022-04-05T16:19:17Z

Azure Pipelines successfully started running 7 pipeline(s).

jywu-msft · 2022-04-05T16:28:31Z

@MaajidKhan FYI, the python flake8 check failed:

[flake8 PEP8 ERROR] ./tools/nuget/generate_nuspec_for_native_nuget.py:475:121: E501 line too long (122 > 120 characters)

Signed-off-by: MaajidKhan <[email protected]>

MaajidKhan · 2022-04-05T16:39:43Z

@MaajidKhan FYI, the python flake8 check failed:

[flake8 PEP8 ERROR] ./tools/nuget/generate_nuspec_for_native_nuget.py:475:121: E501 line too long (122 > 120 characters)

@jywu-msft Done. commit added.

oliviajain · 2022-04-05T16:41:01Z

/azp run onnxruntime-python-checks-ci-pipeline

azure-pipelines · 2022-04-05T16:41:11Z

Azure Pipelines successfully started running 1 pipeline(s).

oliviajain · 2022-04-05T16:47:12Z

/azp run Linux OpenVINO CI Pipeline

oliviajain · 2022-04-05T16:47:22Z

/azp run MacOS NoContribops CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows WebAssembly CI Pipeline, orttraining-amd-gpu-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, Linux CPU CI Pipeline

azure-pipelines · 2022-04-05T16:47:22Z

Azure Pipelines successfully started running 1 pipeline(s).

oliviajain · 2022-04-05T16:47:33Z

/azp run Linux CPU Minimal Build E2E CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-python-checks-ci-pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux CPU x64 NoContribops CI Pipeline, Linux CPU x64 NoContribops CI Pipeline, Linux GPU CI Pipeline

azure-pipelines · 2022-04-05T16:48:05Z

Azure Pipelines successfully started running 8 pipeline(s).

azure-pipelines · 2022-04-05T16:48:10Z

Azure Pipelines successfully started running 7 pipeline(s).

oliviajain · 2022-04-05T17:11:13Z

/azp run onnxruntime-binary-size-checks-ci-pipeline

azure-pipelines · 2022-04-05T17:11:24Z

Azure Pipelines successfully started running 1 pipeline(s).

oliviajain · 2022-04-06T20:02:53Z

Python tests were disabled for OpenVINO CPU because openvino doesn't support max min sum operators with 1 input. The fix will come in the next OV release. Let's add this regression to the OV release notes.

* Enabling ov-ep for 2022.1 Release ->Added ov-ep 2022.1 flow ->Validated CPU Unit tests with OV Master using onnxruntime_test_all unit tests. Signed-off-by: MaajidKhan <[email protected]> * Fix for output mismatch b/w OpenVINO and ONNX Refer: https://jira.devtools.intel.com/browse/CVS-60310 Signed-off-by: MaajidKhan <[email protected]> * Enabling Adobe ops ->Enable Resize op for iGPU ->Enable Add op for iGPU Signed-off-by: MaajidKhan <[email protected]> * Removing irrelevant conditions ->Removing some conditions from GetCapability() which are now not required. (Removed conditions for OV version support less than 2021.2) Signed-off-by: MaajidKhan <[email protected]> * Enable upsample op Signed-off-by: MaajidKhan <[email protected]> * Enable Adobe proxy-e model Signed-off-by: MaajidKhan <[email protected]> * Removing any extra conditions for Opset13 ops * Opset13 changes Signed-off-by: MaajidKhan <[email protected]> * Exception handling for devices * Added comments * Implement GPU Throttling feature *Added GPU Throttling feature for iGPU's. when user enables it as a runtime option, it helps in reducing overall CPU usage of the application *Added changes to exercise this option using onnxruntime_perf_test application. Signed-off-by: MaajidKhan <[email protected]> * Renaming the runtime config option Signed-off-by: MaajidKhan <[email protected]> * Added the user to video and users group * Handling_GPU.0_GPU.1 * Handling special conditions ->Handling corner cases for device_type checks Signed-off-by: MaajidKhan <[email protected]> * Modification to include new api 2.0 changes in the code * Added opset13 changes ->Enabled Few ops ->Added Debug info for case 3b in getcapability() Signed-off-by: MaajidKhan <[email protected]> * Enabling ov-ep for 2022.1 Release ->Added ov-ep 2022.1 flow ->Validated CPU Unit tests with OV Master using onnxruntime_test_all unit tests. Signed-off-by: MaajidKhan <[email protected]> * Fix for output mismatch b/w OpenVINO and ONNX Refer: https://jira.devtools.intel.com/browse/CVS-60310 Signed-off-by: MaajidKhan <[email protected]> * Enabling Adobe ops ->Enable Resize op for iGPU ->Enable Add op for iGPU Signed-off-by: MaajidKhan <[email protected]> * Removing irrelevant conditions ->Removing some conditions from GetCapability() which are now not required. (Removed conditions for OV version support less than 2021.2) Signed-off-by: MaajidKhan <[email protected]> * Enable upsample op Signed-off-by: MaajidKhan <[email protected]> * Enable Adobe proxy-e model Signed-off-by: MaajidKhan <[email protected]> * Removing any extra conditions for Opset13 ops * Opset13 changes Signed-off-by: MaajidKhan <[email protected]> * Exception handling for devices * Added comments * Implement GPU Throttling feature *Added GPU Throttling feature for iGPU's. when user enables it as a runtime option, it helps in reducing overall CPU usage of the application *Added changes to exercise this option using onnxruntime_perf_test application. Signed-off-by: MaajidKhan <[email protected]> * Renaming the runtime config option Signed-off-by: MaajidKhan <[email protected]> * Added the user to video and users group * Handling_GPU.0_GPU.1 * Handling special conditions ->Handling corner cases for device_type checks Signed-off-by: MaajidKhan <[email protected]> * Added opset13 changes ->Enabled Few ops ->Added Debug info for case 3b in getcapability() Signed-off-by: MaajidKhan <[email protected]> * Log comments updated * Changes to enable 2.0 api * Enabling ov-ep for 2022.1 Release ->Added ov-ep 2022.1 flow ->Validated CPU Unit tests with OV Master using onnxruntime_test_all unit tests. Signed-off-by: MaajidKhan <[email protected]> * Fix for output mismatch b/w OpenVINO and ONNX Refer: https://jira.devtools.intel.com/browse/CVS-60310 Signed-off-by: MaajidKhan <[email protected]> * Enabling Adobe ops ->Enable Resize op for iGPU ->Enable Add op for iGPU Signed-off-by: MaajidKhan <[email protected]> * Removing irrelevant conditions ->Removing some conditions from GetCapability() which are now not required. (Removed conditions for OV version support less than 2021.2) Signed-off-by: MaajidKhan <[email protected]> * Enable upsample op Signed-off-by: MaajidKhan <[email protected]> * Enable Adobe proxy-e model Signed-off-by: MaajidKhan <[email protected]> * Removing any extra conditions for Opset13 ops * Opset13 changes Signed-off-by: MaajidKhan <[email protected]> * Exception handling for devices * Added comments * Implement GPU Throttling feature *Added GPU Throttling feature for iGPU's. when user enables it as a runtime option, it helps in reducing overall CPU usage of the application *Added changes to exercise this option using onnxruntime_perf_test application. Signed-off-by: MaajidKhan <[email protected]> * Renaming the runtime config option Signed-off-by: MaajidKhan <[email protected]> * Added the user to video and users group * Handling_GPU.0_GPU.1 * Handling special conditions ->Handling corner cases for device_type checks Signed-off-by: MaajidKhan <[email protected]> * Added opset13 changes ->Enabled Few ops ->Added Debug info for case 3b in getcapability() Signed-off-by: MaajidKhan <[email protected]> * Fix build issue Signed-off-by: MaajidKhan <[email protected]> * Fixes issues *Fixes compiler warnings c4458 on windows. *Fixes the bug in device_type check logic *Adds print info for enable_opencl_throttling option in onnxruntime_perf_test Signed-off-by: MaajidKhan <[email protected]> * commit to make openvino_2021.4 compatible * Fixed IO Buffer Optimization * Fix output names issue * Fix 2021.3 branch * Bug Fix for Multiple inputs/outputs - Assigns the right output_name and input_name for the graph when returned by CompiledModel::inputs() OV function. - Also takex care of output mismatch issue b/w openvino output and onnx output Signed-off-by: MaajidKhan <[email protected]> * Add comments for the changes made Signed-off-by: MaajidKhan <[email protected]> * IO Buffer Changes * Commit for Disabling GPU Throttling for 2021.4 * Updated branch * Fix windows build ->Fixed windows build in debug mode ->Disabled scatternd3_tensor_int64 Signed-off-by: MaajidKhan <[email protected]> * Fixed CPP Unit tests for CPU -Fixed shrink, MVN, ReduceL2, Maxpool, upsample, scatter, slice, reshape, unsqueeze. Signed-off-by: MaajidKhan <[email protected]> * Fixed first set of GPU Tests Signed-off-by: MaajidKhan <[email protected]> * Fixed additional failing tests on GPU ->Added conditions to disable certain ops under certain conditions ->Disabled certain tests ->Added some op supports for no_dimension supported Signed-off-by: MaajidKhan <[email protected]> * Added Expand op support for CPU Signed-off-by: MaajidKhan <[email protected]> * Added condition for squeeze op ->Shape can't have empty axes attribute Signed-off-by: MaajidKhan <[email protected]> * Add support for LessOrEqual op function Signed-off-by: MaajidKhan <[email protected]> * OV Interface wait for replaced by indefinite wait call * use names from ONNX model to access OV tensors This chnage is to use the input/output names retrieved from original onnx model to access OV tensors and to check if there's any input or output names mismatch b/w ONNX naming and OV naming. Signed-off-by: MaajidKhan <[email protected]> * Fixes Myriad unit tests and other issues ->Fixes Myriad CPP unit tests ->Fixes output mismatch issue with models with sub graph partitioning Signed-off-by: MaajidKhan <[email protected]> * Fix segfault issue ->Fixed case 3b condition in get_capability() which was causing the segfault issue Signed-off-by: MaajidKhan <[email protected]> * Fixed build isuse with ov 2021.4 with I/O buffer Signed-off-by: MaajidKhan <[email protected]> * Disables performance counters for I/O Buffer Signed-off-by: MaajidKhan <[email protected]> * Fixed inputs/outputs mismatch for HDDL with 2022.1 Signed-off-by: Mohammad Amir Aqeel <[email protected]> * Fix to enable GPU FP16 * Enabled mlperf_ssd_mobilenet_300 model fully on CPU Signed-off-by: MaajidKhan <[email protected]> * Added ov version specific dll packaging for nuget * Fixed conditions for few ops Signed-off-by: MaajidKhan <[email protected]> * Dockerfile updates * Updated License Info -Updated the copyrights License Info -modified FP16 transformations with OV 2022.1 Signed-off-by: MaajidKhan <[email protected]> * Disabling mlperf_ssd_mobilenet_300 model ->Disabled this model for openvino. The test is failing in Internal_CI pipelines. Signed-off-by: MaajidKhan <[email protected]> * Disabling failing python CPU Tests Signed-off-by: MaajidKhan <[email protected]> * Fixed flake8 python errors Signed-off-by: MaajidKhan <[email protected]> Co-authored-by: hdgx <[email protected]> Co-authored-by: mayavijx <[email protected]> Co-authored-by: sfatimar <[email protected]> Co-authored-by: mohsinmx <[email protected]> Co-authored-by: Mohammad Amir Aqeel <[email protected]>

MaajidKhan and others added 30 commits February 15, 2022 21:38

Enabling ov-ep for 2022.1 Release

133fb01

->Added ov-ep 2022.1 flow ->Validated CPU Unit tests with OV Master using onnxruntime_test_all unit tests. Signed-off-by: MaajidKhan <[email protected]>

Fix for output mismatch b/w OpenVINO and ONNX

948941f

Refer: https://jira.devtools.intel.com/browse/CVS-60310 Signed-off-by: MaajidKhan <[email protected]>

Enabling Adobe ops

140c599

->Enable Resize op for iGPU ->Enable Add op for iGPU Signed-off-by: MaajidKhan <[email protected]>

Removing irrelevant conditions

d5908a6

->Removing some conditions from GetCapability() which are now not required. (Removed conditions for OV version support less than 2021.2) Signed-off-by: MaajidKhan <[email protected]>

Enable upsample op

72cc91b

Signed-off-by: MaajidKhan <[email protected]>

Enable Adobe proxy-e model

4b55164

Signed-off-by: MaajidKhan <[email protected]>

Removing any extra conditions for Opset13 ops

155765b

Opset13 changes

8455317

Signed-off-by: MaajidKhan <[email protected]>

Exception handling for devices

7ad2a02

Added comments

47116d2

Renaming the runtime config option

dc24bad

Signed-off-by: MaajidKhan <[email protected]>

Added the user to video and users group

117e3e9

Handling_GPU.0_GPU.1

9cdbdf3

Handling special conditions

2c94fc3

->Handling corner cases for device_type checks Signed-off-by: MaajidKhan <[email protected]>

Modification to include new api 2.0 changes in the code

e4a455b

Added opset13 changes

a14b8c0

->Enabled Few ops ->Added Debug info for case 3b in getcapability() Signed-off-by: MaajidKhan <[email protected]>

Enabling ov-ep for 2022.1 Release

e264ca9

->Added ov-ep 2022.1 flow ->Validated CPU Unit tests with OV Master using onnxruntime_test_all unit tests. Signed-off-by: MaajidKhan <[email protected]>

Fix for output mismatch b/w OpenVINO and ONNX

049d8c9

Refer: https://jira.devtools.intel.com/browse/CVS-60310 Signed-off-by: MaajidKhan <[email protected]>

Enabling Adobe ops

e0cc70f

->Enable Resize op for iGPU ->Enable Add op for iGPU Signed-off-by: MaajidKhan <[email protected]>

Removing irrelevant conditions

d0141be

->Removing some conditions from GetCapability() which are now not required. (Removed conditions for OV version support less than 2021.2) Signed-off-by: MaajidKhan <[email protected]>

Enable upsample op

ec96986

Signed-off-by: MaajidKhan <[email protected]>

Enable Adobe proxy-e model

b9e87d6

Signed-off-by: MaajidKhan <[email protected]>

Removing any extra conditions for Opset13 ops

ca70463

Opset13 changes

fde21af

Signed-off-by: MaajidKhan <[email protected]>

Exception handling for devices

c9f7ea6

Added comments

d95f8df

Renaming the runtime config option

682c466

Signed-off-by: MaajidKhan <[email protected]>

Added the user to video and users group

d5c3a16

oliviajain reviewed Apr 1, 2022

View reviewed changes

MaajidKhan added 3 commits April 5, 2022 08:43

Merge branch 'master' into enable_2022.1_mar28

1fc1620

Disabling mlperf_ssd_mobilenet_300 model

3b35966

->Disabled this model for openvino. The test is failing in Internal_CI pipelines. Signed-off-by: MaajidKhan <[email protected]>

Disabling failing python CPU Tests

2561683

Signed-off-by: MaajidKhan <[email protected]>

Fixed flake8 python errors

e3e5958

Signed-off-by: MaajidKhan <[email protected]>

MaajidKhan mentioned this pull request Apr 6, 2022

OpenVINO-EP v4.0 Release documentation update #11125

Merged

tomdol mentioned this pull request Apr 6, 2022

ONNX node names as friendly names openvinotoolkit/openvino#10532

Merged

oliviajain self-requested a review April 6, 2022 20:30

oliviajain approved these changes Apr 6, 2022

View reviewed changes

oliviajain merged commit 81fa28b into microsoft:master Apr 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenVINO-EP v4.0 Release PR with OpenVINO 2022.1 #11025

OpenVINO-EP v4.0 Release PR with OpenVINO 2022.1 #11025

MaajidKhan commented Mar 28, 2022

oliviajain Apr 1, 2022 •

edited

Loading

MaajidKhan Apr 1, 2022

jywu-msft commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

jywu-msft commented Apr 5, 2022

jywu-msft commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

jywu-msft commented Apr 5, 2022

MaajidKhan commented Apr 5, 2022

oliviajain commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

oliviajain commented Apr 5, 2022

oliviajain commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

oliviajain commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

oliviajain commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

oliviajain commented Apr 6, 2022 •

edited

Loading

OpenVINO-EP v4.0 Release PR with OpenVINO 2022.1 #11025

OpenVINO-EP v4.0 Release PR with OpenVINO 2022.1 #11025

Conversation

MaajidKhan commented Mar 28, 2022

oliviajain Apr 1, 2022 • edited Loading

Choose a reason for hiding this comment

MaajidKhan Apr 1, 2022

Choose a reason for hiding this comment

jywu-msft commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

jywu-msft commented Apr 5, 2022

jywu-msft commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

jywu-msft commented Apr 5, 2022

MaajidKhan commented Apr 5, 2022

oliviajain commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

oliviajain commented Apr 5, 2022

oliviajain commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

oliviajain commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

oliviajain commented Apr 5, 2022

azure-pipelines bot commented Apr 5, 2022

oliviajain commented Apr 6, 2022 • edited Loading

oliviajain Apr 1, 2022 •

edited

Loading

oliviajain commented Apr 6, 2022 •

edited

Loading