Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WebNN EP] Automatically move input CPU tensors to ml-tensor #23073

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

[WebNN EP] Automatically move input CPU tensors to ml-tensor #23073

wants to merge 7 commits into from
+196 −43

Conversation

egalli
Copy link
Contributor

@egalli egalli commented Dec 11, 2024

Description

If it would improve performance, this patch moves the CPU to ml-tensor before sending the to the ONNXRuntime WebNN EP.

Motivation and Context

We are currently performing 2 extra copies on input tensors located in the CPU when using the WebNN EP (JS -(copy)-> wasm heap -(copy)-> JS -> WebNN API). This patch removes these extra copies.

@egalli
[WebNN EP] Automatically move input CPU tensors to ml-tensor
18224f3 
### Description
If it would improve performance, this patch moves the CPU to ml-tensor before sending the to the ONNXRuntime WebNN EP.

### Motivation and Context
We are currently performing 2 extra copies on input tensors located in the CPU when using the WebNN EP (JS -(copy)-> wasm heap -(copy)-> JS -> WebNN API). This patch removes these extra copies.
fdwr
fdwr reviewed Dec 11, 2024
View reviewed changes
Copy link
Contributor

@fdwr fdwr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yulong or Guenther can review this one effectively (as it impacts TypeScript interfaces). I'll start the CI's though.

@fdwr
Copy link
Contributor

fdwr commented Dec 11, 2024

/azp run ONNX Runtime Web CI Pipeline,Windows GPU CI Pipeline,Linux Android Emulator QNN CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Dec 11, 2024

/azp run Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,Windows ARM64 QNN CI Pipeline,Windows CPU CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Dec 11, 2024

/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Dec 11, 2024

/azp run Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline,Big Models

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 4 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

Honry
Honry reviewed Dec 11, 2024
View reviewed changes
Copy link
Contributor

@Honry Honry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

js/web/lib/wasm/jsep/backend-webnn.ts Outdated Show resolved Hide resolved
js/web/lib/wasm/jsep/backend-webnn.ts Outdated Show resolved Hide resolved
Honry
Honry approved these changes Dec 12, 2024
View reviewed changes
Copy link
Contributor

@Honry Honry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % a nit.

js/web/lib/wasm/jsep/backend-webnn.ts Outdated Show resolved Hide resolved
@Honry
Copy link
Contributor

Honry commented Dec 12, 2024

@fs-eire, @guschmue, pls. take another look, thanks!

@guschmue guschmue added the ep:WebNN WebNN execution provider label Dec 16, 2024
@guschmue
Copy link
Contributor

/azp run ONNX Runtime Web CI Pipeline,Windows GPU CI Pipeline,Linux Android Emulator QNN CI Pipeline

@guschmue
Copy link
Contributor

/azp run Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,Windows ARM64 QNN CI Pipeline,Windows CPU CI Pipeline

@guschmue
Copy link
Contributor

/azp run Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline,Big Models

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@guschmue
Copy link
Contributor

/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 4 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

@fdwr fdwr requested a review from fs-eire December 18, 2024 22:13
@fdwr
Copy link
Contributor

fdwr commented Jan 9, 2025

/azp run ONNX Runtime Web CI Pipeline,Windows GPU CI Pipeline,Linux Android Emulator QNN CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Jan 9, 2025

/azp run Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,Windows ARM64 QNN CI Pipeline,Windows CPU CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Jan 9, 2025

/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Jan 9, 2025

/azp run Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline,Big Models

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 4 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

@egalli
Copy link
Contributor Author

egalli commented Jan 21, 2025

@fdwr on updates on this PR?

@fdwr
Copy link
Contributor

fdwr commented Jan 22, 2025

@fdwr on updates on this PR?

@fs-eire or @guschmue, can you review the Typescript interface JSEP changes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fdwr fdwr fdwr left review comments

@Honry Honry Honry approved these changes

@fs-eire fs-eire Awaiting requested review from fs-eire

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
ep:WebNN WebNN execution provider
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@egalli @fdwr @Honry @guschmue