Update "Performance Adaptation" use case #207
Conversation
- Adding a new use case of neural network deployment for 'Framework Use Cases'
|
@spshin3 thanks for the proposal. Is the key requirement needed to enable this proposed use case to allow "checking automatically the hardware capabilities of devices"? If so, this use case seems similar to https://webmachinelearning.github.io/webnn/#usecase-perf-adapt and we could consider expanding it or merging the two where applicable. |
index.bs
Outdated
| various devices environments with efficiency. To address this issue, she develops | ||
| a framework using WebNN API for interoperability, so that the DNN model can | ||
| select the best options among CPU, GPU, or ML accelerators by checking | ||
| automatically the hardware capabilities of devices |
There was a problem hiding this comment.
This wording seems to imply that WebNN is responsible for automatically pick "the best" option to accelerate a given model based on what is available on the platform without necessarily relying on the input from either the web application or the framework that uses it. This is a big requirement that goes beyond simply:
checking automatically the hardware capabilities of devices
as it also involves analyzing model's complexity and memory bandwidth requirements, etc. Also note that "best" is subjective e.g. is it best for inference latency, data throughput, or battery life?
I understand that this is a new use case. I just want to flag that this may be a design requirement that we have yet to agree as one of the design goals.
There was a problem hiding this comment.
Thanks for the comment. We revised the use case for reflecting the comment.
The descriptions of automatic switching function are revised with considerations of user's preference. In the revised use-case, the 'best' option for the user can be subjectively changed by user's preference.
There was a problem hiding this comment.
Thanks for addressing my question. But I'm still not sure I fully understand this use case. Are we saying that the use case is for a framework to use WebNN for transfer learning on the user-machine via the web browser?
There was a problem hiding this comment.
First of all, thanks for your good catch!
Unfortunately, we missed to consider the web browser scenario when we discussed this issue. We just focused the scenario for implementing DNN model in the mobile devices. Actually, the automatic selection of hardware in mobile devices are being researched in point of view for designing the mobile chip product such as Qualcom's snapdragon.
During several months, we investigated automatic hardware selection in web browser to resolve your sharp question. However, this is very premature issue to implement with the above approach. Just checking the hardware capability is not enough to implement automatic hardware selection function and it seems that this function may be related with ML model-specific matter rather than hardware-specific matter.
To deal with automatic hardware selection, web application developer may need to measure their DNN model performance by using the benchmark framework such as tensorflow benchmark library. See this reference Tensorflow's Performance measurement, and issue #169.
There was a problem hiding this comment.
Thank you @spshin3 for the revised use case.
Do you think the proposed "Performance Benchmark" use case (as in this PR) would benefit from additional API surface or features in the WebNN API? There's a separate Battery Status API that may address your requirements related to identifying when the battery of the user's device is charging or the device is plugged in. I think these two APIs used together might help address your use case.
Feel free to follow up with your suggestions in #169 for any device preference hints or similar features that'd help this use case. Or if changes would be suggested to the Battery Status API, feel free to file issues for that spec.
Thank you for your continued contributions!
There was a problem hiding this comment.
Thanks for carefull reviewing again! @wchao1115
I think that the 'option 2' is similar with our intention for the functionality of 'benchmark'.
If some developers want to test across different hardware like option1, they needs to set each target configuration of ardware offline.
There was a problem hiding this comment.
Given 'option 2', it looks like we could update the existing use case https://www.w3.org/TR/webnn/#usecase-perf-adapt
My assumptions are that the web developer may want to:
- Change the model to better fit the hardware (current use case)
- Choose the execution device to better fit the model (proposed use case)
Or we can do both.
I tried to incorporate these approaches into the existing use case merging with @spshin3's use case, proposal:
Performance Adaptation
A web application developer has a concern about performance of her DNN model. The model needs to run on both a mobile device with a low power CPU as well as on a laptop with a powerful CPU, GPU and a dedicated AI accelerator.
She has confirmed that the model may run too slow on the mobile device which does not have GPU acceleration. To address this issue, her web application refers to the WebNN API to confirm whether acceleration is available or not, so that the application can display a warning for devices without acceleration. After several weeks, she has developed a tiny DNN model that can even run on a CPU. In order to accommodate CPU execution, she modifies the application so that the application loads the tiny model in the case of CPU-only devices.
When executing the DNN model on a laptop with a more powerful CPU, GPU and a dedicated AI accelerator, she wants to use the execution device that minimizes the inference time. To address this issue, she runs the model on each execution device and measures the inference time for each test run. This information helps her release a web application that provides the best possible user experience given available hardware.
WDYT @spshin3? Suggestions welcome!
There was a problem hiding this comment.
I fully agreed @anssiko 's comment because we changed our approach now (automatic device selection -> test device performance).
I reflected the comment by revising our proposal. Please check the new version!
Thanks @anssiko @wchao1115 !
There was a problem hiding this comment.
@anssiko The text that you propose here is still unclear to me. The wording makes it hard to understand what it tries to say. I don't know how to help because I may misrepresent it.
If we go with my wild guess of #2 above, then the issue we should consider is whether WebNN is the right API to "test" that whether a specific type of accelerator is available in the system it currently runs on. And even if it could, the presence of certain accelerator in the system doesn't necessarily imply the performance of the model being run on it. I just can't see the WebNN API as the right API for this purpose.
There was a problem hiding this comment.
@wchao1115 I think I understand your concerns. My proposal was too idealistic.
A concern that I stated earlier but failed to factor into this latest proposal was that the device selection is a hint in part due to privacy reasons. Now, I think that alone might make this use case not possible to be implemented reliably.
Then, if we in addition consider "the presence of certain accelerator in the system doesn't necessarily imply the performance of the model being run on it" I'm afraid the use case is not practical.
I don't immediately see how we could change the API to better cater for this use case while retaining its privacy-preserving qualities. @spshin3 thoughts?
Should we drop the existing performance adaption use case entirely?
This change reflects the previous comments from @wchao1115 and @anssiko.
Updated an use case with considerations of hardware selection
This is the updated version of use case which we discussed.
anssiko
changed the title
|
@spshin3 recently we landed a change that made device type normative while defined power preference as a hint: #169 (comment) -- see also other related discussion in that issue as well as related discussion at https://www.w3.org/2022/06/02-webmachinelearning-minutes.html#t07 Please review the latest spec and let us know if this spec change impacts your expectations with regard to this use case. We are trying to freeze the CR scope for the spec and would like to make a decision on this PR soon. Hearing no concerns, we'd be inclined to label this use cases as post-CR. Thank you for your contributions. |
|
Dear Anssi,
I'm sorry for being late for the discussion of my PR.
Actually, I had been away from office for a long time due to the private schedule.....
I will check the latest work of WebML and contributes for the issue.!
Thanks and Regards.
- Sungpil
=======================================================
Sungpil Shin / Senior ResearcherStandards & Open Source Research Division 218 Gajeong-ro, Yuseong-gu, Daejeon, 34129, Republic of KoreaTel 042-860-1379 Mobile 010-5051-7536 Email ***@***.*** ***@***.***
…-----Original Message-----
From: "Anssi Kostiainen" ***@***.***>
To: "webmachinelearning/webnn" ***@***.***>;
Cc: "Sungpil Shin" ***@***.***>; "Mention" ***@***.***>;
Sent: 2022-06-09 (목) 16:15:23 (UTC+09:00)
Subject: Re: [webmachinelearning/webnn] Update "Performance Adaptation" use case (#207)
@spshin3 https://github.com/spshin3 recently we landed a change that made device type normative while defined power preference as a hint: #169 (comment) #169 (comment) -- see also other related discussion in that issue as well as related discussion at https://www.w3.org/2022/06/02-webmachinelearning-minutes.html#t07 https://www.w3.org/2022/06/02-webmachinelearning-minutes.html#t07
Please review the latest spec and let us know if this spec change impacts your expectations with regard to this use case. We are trying to freeze the CR scope for the spec and would like to make a decision on this PR soon. Hearing no concerns, we'd be inclined to label this use cases as post-CR. Thank you for your contributions.
—Reply to this email directly, view it on GitHub #207 (comment), or unsubscribe https://github.com/notifications/unsubscribe-auth/ANYEXJH6MFCUFYRCIXVRH63VOGKXZANCNFSM5DBEHC3Q.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@spshin3 we have created a dedicated repo for future work: https://github.com/webmachinelearning/proposals Please see the readme for submission instructions. Based on our discussions in the WG, we'd welcome you to submit this use case to that repo so it can be appropriately considered. Thanks! |
|
@spshin3, thank you for this proposal. After careful consideration, the WG suggested this use case should be submitted as a future work proposal to https://github.com/webmachinelearning/proposals Per https://www.w3.org/2022/09/08-webmachinelearning-minutes.html#t08 I'm closing this PR. |
Preview | Diff
Preview | Diff