Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[js/web] rewrite backend resolve to allow multiple EPs #19735

Merged
merged 6 commits into from
Mar 15, 2024
Merged

[js/web] rewrite backend resolve to allow multiple EPs #19735

merged 6 commits into from
Mar 15, 2024

Conversation

fs-eire
Copy link
Contributor

@fs-eire fs-eire commented Mar 1, 2024

Description

This PR rewrite the backend resolve logic to support specifying multiple EPs.

Backend

The first version of ONNX Runtime Web actually carried some existing code from ONNX.js, which includes the "backend" concept. The original "backend" in ONNX.js is designed in a way assuming there is only one backend from user's backend hint list will be used. For example, in ONNX.js, if user specify a backend hint as ['webgl', 'wasm'], ONNX.js will first try to use WebGL backend - if it loads successfully (the browser supports webgl), then "webgl" backend will be used and "wasm" will be ignored; otherwise, "webgl" will be ignored and try to load "wasm" backend.

In short: only one backend will be used when initializing a session.

Execution Provider

Execution Provider, or EP, in ONNX Runtime is a different concept. One of the differences is that users are allow to specify multiple EPs, and if one does not support a particular kernel, it can fallback to other EP. This is a very common case when using a GPU EP in ONNX Runtime.

Current Status: Backend v.s. EP

Because of the history reasons mentioned above, the current status is quite confusing. There are real backends, which means it's different implementation in code; and there are backend hints, which are used as string names for backend hint; and there are EPs of the ONNX Runtime concepts.

currently there are only 2 backends in our code base: The "onnxjs backend", and the "wasm backend". The "onnxjs backend" currently only powers backend hint "webgl", which go into the old onnx.js code path. All other backend hints including "wasm", "cpu"(alias to wasm), "webgpu" and "webnn" are all powered by "wasm backend".

And because ORT Web treat "backend" as an internal concept and want to align with ONNX Runtime, so those names of backend hints are becoming EP names.

The following table shows today's status:

Execution Provider Name (public) / Backend Hint (internal) Backend EP in ORT
"wasm"/"cpu" WasmBackend CPU EP
"webgl" OnnxjsBackend * technically not an EP
"webgpu" WasmBackend JSEP
"webnn" WasmBackend WebNN EP

Problem

While the API allows to specify multiple EPs, the backend resolving only allows one backend. This causes issues when user specify multiple EP names in session options, the backend resolve behavior and EP registration behavior is inconsistent. Specifically, in this issue: #15796 (comment):

EP list ['webgpu', 'wasm'] on a browser without WebGPU support resolves to 'wasm' backend, but the full EP list is passed in session options, so JSEP is still enabled, causing the runtime error.

Solution

Since we still need WebGL backend, we cannot totally remove the backend register/resolve system. In this PR I made the following changes:

  • initialize every backend from the EP list, instead of only do that for the first successful one.
  • for the first resolved backend, filter all EP using the exact same backend. Remove all EPs not using this backend from session options
  • for every explicitly specified EP, if it's removed, show a warning message in console

@gyagp gyagp mentioned this pull request Mar 4, 2024
Closed
guschmue
guschmue previously approved these changes Mar 11, 2024
View reviewed changes
@mszhanyi mszhanyi mentioned this pull request Mar 13, 2024
Closed
@fs-eire fs-eire merged commit 79e50ae into main Mar 15, 2024
93 of 95 checks passed
@fs-eire fs-eire deleted the fs-eire/allow-validation-test-on-eps branch March 15, 2024 18:47
fs-eire added a commit that referenced this pull request Mar 20, 2024
@fs-eire
[js/web] rewrite backend resolve to allow multiple EPs (#19735)
648cc42 
### Description

This PR rewrite the backend resolve logic to support specifying multiple
EPs.

#### Backend

The first version of ONNX Runtime Web actually carried some existing
code from [ONNX.js](https://github.com/microsoft/onnxjs), which includes
the "backend" concept. The original "backend" in ONNX.js is designed in
a way assuming there is only one backend from user's backend hint list
will be used. For example, in ONNX.js, if user specify a backend hint as
`['webgl', 'wasm']`, ONNX.js will first try to use WebGL backend - if it
loads successfully (the browser supports webgl), then "webgl" backend
will be used and "wasm" will be ignored; otherwise, "webgl" will be
ignored and try to load "wasm" backend.

In short: only one backend will be used when initializing a session.

#### Execution Provider

Execution Provider, or EP, in ONNX Runtime is a different concept. One
of the differences is that users are allow to specify multiple EPs, and
if one does not support a particular kernel, it can fallback to other
EP. This is a very common case when using a GPU EP in ONNX Runtime.

#### Current Status: Backend v.s. EP

Because of the history reasons mentioned above, the current status is
quite confusing. There are **real backend**s, which means it's different
implementation in code; and there are **backend hint**s, which are used
as string names for backend hint; and there are **EP**s of the ONNX
Runtime concepts.

currently there are only 2 **backend**s in our code base: The "onnxjs
backend", and the "wasm backend". The "onnxjs backend" currently only
powers backend hint "webgl", which go into the old onnx.js code path.
All other backend hints including "wasm", "cpu"(alias to wasm), "webgpu"
and "webnn" are all powered by "wasm backend".

And because ORT Web treat "backend" as an internal concept and want to
align with ONNX Runtime, so those names of backend hints are becoming EP
names.

The following table shows today's status:

| Execution Provider Name (public) / Backend Hint (internal) | Backend |
EP in ORT
| -------- | ------- | ------- |
| "wasm"/"cpu" | WasmBackend | CPU EP
| "webgl" | OnnxjsBackend | \* technically not an EP
| "webgpu" | WasmBackend | JSEP
| "webnn" | WasmBackend | WebNN EP

#### Problem

While the API allows to specify multiple EPs, the backend resolving only
allows one backend. This causes issues when user specify multiple EP
names in session options, the backend resolve behavior and EP
registration behavior is inconsistent. Specifically, in this issue:
#15796 (comment):

EP list `['webgpu', 'wasm']` on a browser without WebGPU support
resolves to 'wasm' backend, but the full EP list is passed in session
options, so JSEP is still enabled, causing the runtime error.


#### Solution

Since we still need WebGL backend, we cannot totally remove the backend
register/resolve system. In this PR I made the following changes:
- initialize every backend from the EP list, instead of only do that for
the first successful one.
- for the first resolved backend, filter all EP using the exact same
backend. Remove all EPs not using this backend from session options
- for every explicitly specified EP, if it's removed, show a warning
message in console
YUNQIUGUO pushed a commit that referenced this pull request Mar 29, 2024
@fs-eire @qjia7
@axinging @satyajandhyala @Honry @hujiajie @guschmue @martholomew @dependabot @segevfiner @ibelem
1.17.3 cherry-picks for ORT Web changes (#19926)
45ff957 
### Description
This PR is a preview of cherry-picks for ort-web to `rel-1.17.3` based
on `rel-1.17.2`.

<details>

<summary>Changes of ort-web to cherry-pick</summary>

The following commits are from main branch.

`o` stands for pick, and `x` stands for skip.
```
o   2e0a388 [js/webgpu] Add HardSigmoid support (#19215)
o   d226e40 [js/webgpu] set query type in onRunStart (#19202)
o   61610ff [js/webgpu] Add FusedConv clip test case (#18900)
o   a33b5bd [JS/WebGPU] Added Uniforms to SkipLayerNorm. (#18788)
o   591f90c [js/webgpu] Fix issue of timestamp query (#19258)
o   7252c6e [WebNN EP] Support WebNN async API with Asyncify (#19145)
o   5b06505 [js/webgpu] Fix Tanh explosion (#19201)
o   656ca66 [js/webgpu] Support uniforms for conv, conv transpose, conv grouped (#18753)
o   a3f0e24 [js/webgpu] Support f16 uniform (#19098)
o   9e69606 fix f16 for attention, enable slice and flatten for more types (#19262)
o   624b4e2 [js/webgpu] Remove enableShapesUniforms (#19279)
o   90883a3 [js/webgpu] Add hardSigmoid activation for fusedConv (#19233)
o   85cef0a [js/webgpu] Support capture and replay for jsep (#18989)
o   d73131c [js/webgpu] Use DataType as uniform cpu type (#19281)
o   dd1f6cc [js/webgpu] resolve codescan alert (#19343)
o   3a2ab19 [js/webgpu] Refactor createTensorShapeVariables (#18883)
o   efc17e7 [js/webgpu] Fix the undefined push error (#19366)
 x  50806a7 [js/web] support external data in npm test (#19377)
o   ccbe264 [js/webgpu] Add LeakyRelu activation for fusedConv (#19369)
o   5ff27ef [js/webgpu] support customop FastGelu (#19392)
 x  03be65e [js/web] fix types exports in package.json (#19458)
o   06269a3 [js/webgpu] allow uint8 tensors for webgpu (#19545)
o   dfeda90 [JS/WebGPU] Add MatMulNBits (#19446)
o   1b48054 [js/webgpu] Create Split indices helpers by rank, not by shape (#19554)
o   3fe2c13 [js] small fix to workaround formatter (#19400)
 x  70567a4 [js/web] use ApiTensor insteadof onnxjs Tensor in TensorResultValidator (#19358)
o   6e04e36 [js/common] upgrade tsc in common from 4.9.5 to 5.2.2 (#19317)
o   58f4921 [js] changes to allow Float16Array if any polyfill is available (#19305)
o   57d6819 [js/web] Fix fused-conv is not included in npm test (#19581)
o   ebd220b Misspelling in README.md (#19433)
o   38c3432 Bump ip from 1.1.8 to 1.1.9 in /js/react_native (#19582)
o   fe82fcc [js/webgpu] Fix Conv2DTransposeMatMul f16 compilation failure (#19596)
o   76a2a48 Bump ip from 1.1.8 to 1.1.9 in /js/react_native/e2e (#19583)
o   29b1106 [node] Switch to setImmediate to avoid starving the Node.js event loop (#19610)
o   ae3d73c [JS/WebGPU] Fix Split and Where to handle corner cases. (#19613)
o   aec2389 [js/webgpu] allows a ProgramInfo's RunData to use zero sized output (#19614)
o   bb43a0f [js/webgpu] minor fixes to make tinyllama work (#19564)
o   0edb035 [js/web] fix suite test list for zero sized tensor (#19638)
o   3cb81cd [js/common] move 'env.wasm.trace' to 'env.trace' (#19617)
o   e30618d [js/webgpu] use Headless for webgpu test by default (#19702)
o   f06164e [js/web] transfer input buffer back to caller thread (#19677)
 x  a788514 [js/web] dump debug logs for karma for diagnose purpose (#19785)
o   24b72d2 [JS/WebGPU] Preserve zero size input tensor dims. (#19737)
o   4538d31 [js/webgpu] expose a few properties in WebGPU API (#19857)
o   53de2d8 [js/webgpu] Enable GroupedConvVectorize path (#19791)
o   ed250b8 [JS/WebGPU] Optimize MatMulNBits (#19852)
 x  e771a76 [js/test] align web test runner flags with ort.env (#19790)
o   79e50ae [js/web] rewrite backend resolve to allow multiple EPs (#19735)
o   acb0df2 Fix #19931 broken Get Started link of "ONNX Runtime JavaScript API" page (#19932)
o   b29849a [js/common] fix typedoc warnings (#19933)
o   afdab62 Bump follow-redirects from 1.15.4 to 1.15.6 in /js/web (#19949)
o   28ad6c3 Bump follow-redirects from 1.15.4 to 1.15.6 in /js/node (#19951)
o   7e0d424 accumulate in fp32 for Reduce* (#19868)
o   4c6a6a3 [js/webgpu] Fix NAN caused by un-initialized buffer in instance-norm (#19387)
o   01c7aaf [js/webgpu] allow setting env.webgpu.adapter (#19940)
o   c45cff6 [js/webgpu] fix maxpool / fp16 (#19981)
```

</details>

<details>
<summary>Cherry-pick commandlines</summary>

```sh
git cherry-pick 2e0a388
git cherry-pick d226e40
git cherry-pick 61610ff
git cherry-pick a33b5bd
git cherry-pick 591f90c
git cherry-pick 7252c6e
git cherry-pick 5b06505
git cherry-pick 656ca66
git cherry-pick a3f0e24
git cherry-pick 9e69606
git cherry-pick 624b4e2
git cherry-pick 90883a3
git cherry-pick 85cef0a  #<<<<< Note: conflicts
git cherry-pick d73131c
git cherry-pick dd1f6cc
git cherry-pick 3a2ab19
git cherry-pick efc17e7
git cherry-pick ccbe264
git cherry-pick 5ff27ef
git cherry-pick 06269a3
git cherry-pick dfeda90
git cherry-pick 1b48054
git cherry-pick 3fe2c13
git cherry-pick 6e04e36
git cherry-pick 58f4921
git cherry-pick 57d6819
git cherry-pick ebd220b
git cherry-pick 38c3432
git cherry-pick fe82fcc
git cherry-pick 76a2a48
git cherry-pick 29b1106
git cherry-pick ae3d73c
git cherry-pick aec2389
git cherry-pick bb43a0f
git cherry-pick 0edb035
git cherry-pick 3cb81cd
git cherry-pick e30618d
git cherry-pick f06164e
git cherry-pick 24b72d2
git cherry-pick 4538d31
git cherry-pick 53de2d8
git cherry-pick ed250b8
git cherry-pick 79e50ae
git cherry-pick acb0df2
git cherry-pick b29849a
git cherry-pick afdab62
git cherry-pick 28ad6c3
git cherry-pick 7e0d424
git cherry-pick 4c6a6a3
git cherry-pick 01c7aaf
git cherry-pick c45cff6
```
</details>

<details>
<summary>Cherry-pick conflicts</summary>

- 85cef0a #18989
this change is for enabling graph capture feature for JSEP, and it is
done after ROCM EP enabled graph capture feature. However, the ROCM EP
graph capture feature is not cherry-picked in rel-1.17.2.
</details>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Jiajia Qin <[email protected]>
Co-authored-by: Xu Xing <[email protected]>
Co-authored-by: satyajandhyala <[email protected]>
Co-authored-by: Yang Gu <[email protected]>
Co-authored-by: Wanming Lin <[email protected]>
Co-authored-by: Jiajie Hu <[email protected]>
Co-authored-by: Guenther Schmuelling <[email protected]>
Co-authored-by: Matttttt <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Segev Finer <[email protected]>
Co-authored-by: Belem Zhang <[email protected]>
siweic0 pushed a commit to siweic0/onnxruntime-web that referenced this pull request May 9, 2024
@fs-eire
[js/web] rewrite backend resolve to allow multiple EPs (microsoft#19735)
d62fe8b 
### Description

This PR rewrite the backend resolve logic to support specifying multiple
EPs.

#### Backend

The first version of ONNX Runtime Web actually carried some existing
code from [ONNX.js](https://github.com/microsoft/onnxjs), which includes
the "backend" concept. The original "backend" in ONNX.js is designed in
a way assuming there is only one backend from user's backend hint list
will be used. For example, in ONNX.js, if user specify a backend hint as
`['webgl', 'wasm']`, ONNX.js will first try to use WebGL backend - if it
loads successfully (the browser supports webgl), then "webgl" backend
will be used and "wasm" will be ignored; otherwise, "webgl" will be
ignored and try to load "wasm" backend.

In short: only one backend will be used when initializing a session.

#### Execution Provider

Execution Provider, or EP, in ONNX Runtime is a different concept. One
of the differences is that users are allow to specify multiple EPs, and
if one does not support a particular kernel, it can fallback to other
EP. This is a very common case when using a GPU EP in ONNX Runtime.

#### Current Status: Backend v.s. EP

Because of the history reasons mentioned above, the current status is
quite confusing. There are **real backend**s, which means it's different
implementation in code; and there are **backend hint**s, which are used
as string names for backend hint; and there are **EP**s of the ONNX
Runtime concepts.

currently there are only 2 **backend**s in our code base: The "onnxjs
backend", and the "wasm backend". The "onnxjs backend" currently only
powers backend hint "webgl", which go into the old onnx.js code path.
All other backend hints including "wasm", "cpu"(alias to wasm), "webgpu"
and "webnn" are all powered by "wasm backend".

And because ORT Web treat "backend" as an internal concept and want to
align with ONNX Runtime, so those names of backend hints are becoming EP
names.

The following table shows today's status:

| Execution Provider Name (public) / Backend Hint (internal) | Backend |
EP in ORT
| -------- | ------- | ------- |
| "wasm"/"cpu" | WasmBackend | CPU EP
| "webgl" | OnnxjsBackend | \* technically not an EP
| "webgpu" | WasmBackend | JSEP
| "webnn" | WasmBackend | WebNN EP

#### Problem

While the API allows to specify multiple EPs, the backend resolving only
allows one backend. This causes issues when user specify multiple EP
names in session options, the backend resolve behavior and EP
registration behavior is inconsistent. Specifically, in this issue:
microsoft#15796 (comment):

EP list `['webgpu', 'wasm']` on a browser without WebGPU support
resolves to 'wasm' backend, but the full EP list is passed in session
options, so JSEP is still enabled, causing the runtime error.


#### Solution

Since we still need WebGL backend, we cannot totally remove the backend
register/resolve system. In this PR I made the following changes:
- initialize every backend from the EP list, instead of only do that for
the first successful one.
- for the first resolved backend, filter all EP using the exact same
backend. Remove all EPs not using this backend from session options
- for every explicitly specified EP, if it's removed, show a warning
message in console
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guschmue guschmue guschmue approved these changes

Assignees
No one assigned
Labels
release:1.17.3
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@fs-eire @guschmue