Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update "Performance Adaptation" use case #207

Closed
wants to merge 5 commits into from
Closed

Update "Performance Adaptation" use case #207

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b2d4141
Adding a new use case for 'Framework Use Cases'
spshin3 Aug 30, 2021
756e85b
Revised use case
spshin3 Oct 19, 2021
e5b6e9c
Updated framework use case
spshin3 Dec 15, 2021
437de95
Update index.bs
spshin3 Dec 15, 2021
7b49df4
Revised 'Performance Adaptation' use case
spshin3 Feb 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions index.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,6 +369,19 @@ so that the application loads the tiny model in the case of CPU-only devices.

A JavaScript ML framework is responsible for loading, interpreting and executing a ML model. During the model execution phase, the framework iterates through the operations of the model and executes each operation on the hardware device, like CPU, GPU or ML accelerator. To avoid the unnecessary data copying across devices, the framework selects the same device to execute the operations. For a compute intensive operation, such as convolution 2D or matrix multiplication, the framework uses WebNN API to execute it with the ML-specific acceleration available on that selected device.

### Neural Network Deployment ### {#usecase-neural-net-deploy}
An application developer wants to develop a DNN model which outperforms in
Natural Language Processing such as [[GPT-3]] on the web. The model has pre-trained
with an enormous and diverse data, and she wants to deploy this DNN model in
various devices environments with efficiency. To address this issue, she develops
a framework using WebNN API for interoperability, so that the DNN model can
select the best options among CPU, GPU, or ML accelerators by checking
automatically the hardware capabilities of devices
Copy link
Collaborator

@wchao1115 wchao1115 Sep 16, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wording seems to imply that WebNN is responsible for automatically pick "the best" option to accelerate a given model based on what is available on the platform without necessarily relying on the input from either the web application or the framework that uses it. This is a big requirement that goes beyond simply:

checking automatically the hardware capabilities of devices

as it also involves analyzing model's complexity and memory bandwidth requirements, etc. Also note that "best" is subjective e.g. is it best for inference latency, data throughput, or battery life?

I understand that this is a new use case. I just want to flag that this may be a design requirement that we have yet to agree as one of the design goals.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment. We revised the use case for reflecting the comment.
The descriptions of automatic switching function are revised with considerations of user's preference. In the revised use-case, the 'best' option for the user can be subjectively changed by user's preference.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing my question. But I'm still not sure I fully understand this use case. Are we saying that the use case is for a framework to use WebNN for transfer learning on the user-machine via the web browser?

Copy link
Contributor Author

@spshin3 spshin3 Dec 15, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thanks for your good catch!

Unfortunately, we missed to consider the web browser scenario when we discussed this issue. We just focused the scenario for implementing DNN model in the mobile devices. Actually, the automatic selection of hardware in mobile devices are being researched in point of view for designing the mobile chip product such as Qualcom's snapdragon.

During several months, we investigated automatic hardware selection in web browser to resolve your sharp question. However, this is very premature issue to implement with the above approach. Just checking the hardware capability is not enough to implement automatic hardware selection function and it seems that this function may be related with ML model-specific matter rather than hardware-specific matter.

To deal with automatic hardware selection, web application developer may need to measure their DNN model performance by using the benchmark framework such as tensorflow benchmark library. See this reference Tensorflow's Performance measurement, and issue #169.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @spshin3 for the revised use case.

Do you think the proposed "Performance Benchmark" use case (as in this PR) would benefit from additional API surface or features in the WebNN API? There's a separate Battery Status API that may address your requirements related to identifying when the battery of the user's device is charging or the device is plugged in. I think these two APIs used together might help address your use case.

Feel free to follow up with your suggestions in #169 for any device preference hints or similar features that'd help this use case. Or if changes would be suggested to the Battery Status API, feel free to file issues for that spec.

Thank you for your continued contributions!

Copy link
Contributor Author

@spshin3 spshin3 Feb 16, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for carefull reviewing again! @wchao1115
I think that the 'option 2' is similar with our intention for the functionality of 'benchmark'.
If some developers want to test across different hardware like option1, they needs to set each target configuration of ardware offline.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given 'option 2', it looks like we could update the existing use case https://www.w3.org/TR/webnn/#usecase-perf-adapt

My assumptions are that the web developer may want to:

  • Change the model to better fit the hardware (current use case)
  • Choose the execution device to better fit the model (proposed use case)

Or we can do both.

I tried to incorporate these approaches into the existing use case merging with @spshin3's use case, proposal:

Performance Adaptation

A web application developer has a concern about performance of her DNN model. The model needs to run on both a mobile device with a low power CPU as well as on a laptop with a powerful CPU, GPU and a dedicated AI accelerator.

She has confirmed that the model may run too slow on the mobile device which does not have GPU acceleration. To address this issue, her web application refers to the WebNN API to confirm whether acceleration is available or not, so that the application can display a warning for devices without acceleration. After several weeks, she has developed a tiny DNN model that can even run on a CPU. In order to accommodate CPU execution, she modifies the application so that the application loads the tiny model in the case of CPU-only devices.

When executing the DNN model on a laptop with a more powerful CPU, GPU and a dedicated AI accelerator, she wants to use the execution device that minimizes the inference time. To address this issue, she runs the model on each execution device and measures the inference time for each test run. This information helps her release a web application that provides the best possible user experience given available hardware.

WDYT @spshin3? Suggestions welcome!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fully agreed @anssiko 's comment because we changed our approach now (automatic device selection -> test device performance).
I reflected the comment by revising our proposal. Please check the new version!

Thanks @anssiko @wchao1115 !

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anssiko The text that you propose here is still unclear to me. The wording makes it hard to understand what it tries to say. I don't know how to help because I may misrepresent it.

If we go with my wild guess of #2 above, then the issue we should consider is whether WebNN is the right API to "test" that whether a specific type of accelerator is available in the system it currently runs on. And even if it could, the presence of certain accelerator in the system doesn't necessarily imply the performance of the model being run on it. I just can't see the WebNN API as the right API for this purpose.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wchao1115 I think I understand your concerns. My proposal was too idealistic.

A concern that I stated earlier but failed to factor into this latest proposal was that the device selection is a hint in part due to privacy reasons. Now, I think that alone might make this use case not possible to be implemented reliably.

Then, if we in addition consider "the presence of certain accelerator in the system doesn't necessarily imply the performance of the model being run on it" I'm afraid the use case is not practical.

I don't immediately see how we could change the API to better cater for this use case while retaining its privacy-preserving qualities. @spshin3 thoughts?

Should we drop the existing performance adaption use case entirely?


After she releases the DNN model, her DNN model are applied to various devices
from server-level devices and mobile devices and be used in AI applications
such as language translation, chatbots, random writing, and others.

Security Considerations {#security}
===================================

Expand Down Expand Up @@ -2586,6 +2599,44 @@ Benjamin Poulain for their contributions to the API specification.
"Hartwig Adam"
],
"date": "November 2019"
},
"GPT-3": {
"href": "https://arxiv.org/abs/2005.14165",
"title": "Language Models are Few-Shot Learners",
"authors": [
"Tom B. Brown",
"Benjamin Mann",
"Nick Ryder",
"Melanie Subbiah",
"Jared Kaplan",
"Prafulla Dhariwal",
"Arvind Neelakantan",
"Pranav Shyam",
"Girish Sastry",
"Amanda Askell",
"Sandhini Agarwal",
"Ariel Herbert-Voss",
"Gretchen Krueger",
"Tom Henighan",
"Rewon Child",
"Aditya Ramesh",
"Daniel M. Ziegler",
"Jeffrey Wu",
"Clemens Winter",
"Christopher Hesse",
"Mark Chen",
"Eric Sigler",
"Mateusz Litwin",
"Scott Gray",
"Benjamin Chess",
"Jack Clark",
"Christopher Berner",
"Sam McCandlish",
"Alec Radford",
"Ilya Sutskever",
"Dario Amodei"
],
"date": "July 2020"
}
}
</pre>