Seeking feedback on delayed clipboard rendering proposal #417
Comments
anaskim
added
the
Delayed-Clipboard-Rendering
|
Tagging few people who would be interested in this proposal. @annevk @whsieh @a-sully @smaug---- |
|
Adding @EdgarChen too |
sanketj
added
the
Agenda+
|
2023-01-12: ACTION: everyone to take a look at the existing API and check if it can be used for the delayed rendering use case. |
|
2023-02-09 call:
|
|
FWIW, I find "rendering" a bit confusing here, but the description is relatively clear fortunately. I tend to agree with the feedback in the call that the existing API allows for this, but might need some refinements. |
|
Details from 2023-01-12 call: [torsdag 12 januari 2023] [17:04:49 CET] topic: https://github.com/w3c/editing/issues/417 |
|
We did some investigations into whether the existing read/write APIs are enough to implement delayed generation of clipboard data. Here are our findings: Problems with the existing API e.g.
Here the executor function There are couple of solutions that solves the issues mentioned above:
In this case the web author can decide which formats to delay generate and which ones to resolve immediately without having to provide the format and callback info in a separate map. This provides better developer ergonomics and easier to reason about as to which formats should be written immediately to the system clipboard and which ones should be marked as delayed generate. |
|
Ah, I see — it seems like DOM promise callbacks are always invoked immediately upon construction per the JS spec, rather than lazily. In that case, I agree that a callback is a reasonable way forward. |
|
Slight update to option 2: Per the clarification provided in whatwg/webidl#1278, the syntax proposed above won't work as is. The syntax would need to be:
|
|
Discussed at editing meeting 3/9/2023 |
|
A comment I made that didn't make it into the notes: UAs could consider integrating with the Reporting API for the scenario where a site places deferred content on the clipboard, the user navigates away from the site, and the user attempts to paste. This could act as a signal to sites that they should consider doing something in "beforeunload" handling, e.g. prompting the user that clipboard data will be lost. I think I've seen native Excel do something like this at least on macOS; I think it has heuristics and makes the clipboard content deferred if over a certain size, and warns on app shutdown? ETA: To clarify: someone else in the call suggested the beforeunload approach, which would be good to document as part of the proposal. I was suggesting that if a site doesn't do that, the Reporting API might be a way for the UA to provide a signal to web developers that it's something they should consider implementing. |
|
From the Adobe perspective, speaking on behalf of the PSWeb team we would indeed have use for a delayed clipboard rendering interface. Our underlying native code already supports and uses this on OS's that support it. So we are definitely in support of having similar capabilities on Web. We would however need clarification on the path to ensuring that a pending clipboard copy is fully rendered before the window/tab is closed (beforeunload might be sufficient, but I'm unclear on how this would work). It would also be interesting if the API could support streaming results, since PS can be dealing with very large data copies, and unnecessary buffer allocations may make that prohibitive. |
|
Kind of a tangent, but I'm a bit surprised to not see any mention of the previous attempt of including this kind of functionality in the spec (which still exists in the working draft) in the explainer, i.e. the https://www.w3.org/TR/clipboard-apis/#dom-clipboarditem-createdelayed method. I don't think anybody every tried actually figuring out how that method should work, but I would expect to at least see it mentioned as a considered alternative API shape? |
|
I believe https://docs.google.com/document/d/1lpi3-9vBP_1b7hZc2xBs0s_HaACJ6UigZZqHlJSNeJg/edit#heading=h.cuyqt05mqd5i was the explainer for that feature at the time. |
I thought I removed this from the spec, but looks like we haven't sync'd the latest changes to the spec with the working draft for some reason? |
|
drive-by: I noticed that there are some discussions about what to do when the document is navigated away, but it seems like it assumes that the document will be destroyed after that, but that's not always true: the document can get BFCached and stay in a non-fully active state, and later restored. So we can still get the contents of the document for that case, although I'm not sure if we want to behave differently from the non-BFCache case (maybe there's a privacy problem if we do that too?). See also this section and this section of the BFCache guide. |
|
Thanks for raising this. I think it's reasonable to make pages that own the clipboard with lazy data non-bfcacheable. It shouldn't come up very often. Edit: to clarify, I don't think a page holding the clipboard can be allowed into bfcache because if the user tries to paste, we we have no way to resurrect the page so the paste will fail. This is just like a page having an unload handler, which I presume prevents deactivation. |
|
@benjamind what does the adobe desktop application do if the user requests shutdown? Does it block until the clipboard content has been produced? Doing the same thing on the web is (generally) no OK. We do not trust pages to behave well. We have a limit on the time they can keep computing after being closed. We do not want pages that no longer have a visible UI to consume resources indefinitely as it's much harder for the user to identify and kill them. One possibility is for apps that use this API to install a |
New APIs shouldn't make pages non-bfcacheable. It will either make the user experience worse by making history page load times worse (as it could've been instant), or it will make web developers not want to use the API because they want to avoid losing the instant load. We've disabled BFCache on old APIs, but that's to prevent existing pages from breaking because they didn't expect the page to be BFCached and the API to behave differently. I think the API here should behave the same way on navigation, regardless of whether the document got bfcached or not. Whether we drop the contents on normal navigations or immediately do the work then, I don't see why we can't do that on navigations where we bfcache instead. |
|
@fergald I have the same concern and I think it's an important detail to work out before proceeding with the API. At least for Chromium we cap the duration of @rakina sure, in a world where navigating away eagerly fetched the clipboard contents, we could do that whether or not the page was going into bfcache. I thought you were suggesting that we wouldn't need to grab the data because the page might come back. If developers are worried about snappy navigations they probably won't use this API regardless, since this proposed work-on-shutdown model is going to be slow one way or another. |
Yes it does run immediately, but you don't have to call I guess what you're saying is that there's currently no event for such a request? |
|
@annevk So, we don't want to just delay the write of the format's data for x seconds, we don't want to write the data for that format at all if the target app (where the paste is happening) doesn't need it. With just text in the cells, we see 22 different formats on the clipboard. Native Excel uses delayed clipboard rendering, so we don’t have the data for all the formats in the clipboard. The data for a particular format gets populated when the user pastes the content in an app that supports that format. On the web, we support web custom formats that apps can use to copy/paste high fidelity content between web-to-web, web-to-native or vice versa. These formats are expensive to produce, and only the app that supports the custom format can parse its content, so these formats are ideal candidates for delay rendering. For Excel online specifically, the model lives on the server, so copy-paste involves data transfer between the client and the server. This leads to a lot of COGS due to server-side processing and large amounts of data being transferred over-the-wire, particularly for large payloads. With delayed clipboard rendering, Excel online app is looking to efficiently handle the web custom formats (that are expensive compared to text & html) when it’s not requested by the target app where the user pastes the content copied from Excel online. Scenario 2 (Adobe PS use case) |
|
To be pedantic, this could be done with promises with the addition of another signal, e.g. an event as @annevk mentions. I think this would be a poor developer experience compared to the callback approach proposed above. Here's what it could look like, just to demonstrate: navigator.clipboard.write(new ClipboardItem({
'text/html': new Promise(resolve => {
// somehow this event target is scoped to this clipboard item?
// event will only be fired once, even if paste happens again?
some_target.addEventListener('some_event_type', async e => {
// are we the type handler that's actually desired here?
if (e.requestedType === 'text/html') {
// do a bunch of stuff here, probably async
resolve(results);
}
};
}),
/* repeat the above for every supported type, but we'll only ever call resolve() for one Promise */
}));There may be a cleaner way, but IMHO a callback seems much cleaner. |
|
A callback definitely seems like the way. Is there any value in having the clipboard item data ever be a Promise? In other words, rather than this: I might expect this: That signature lets the value be provided immediately, deferred, or deferred and resolved asynchronously. |
Yes, because we want to allow sites to construct the data asynchronously, but eagerly. (Which is how the API currently behaves.) I believe what we actually would like is: However, it's apparently not possible for a Promise to be part of a union. Thus I think the next best option is to be able to provide additional data to the |
Right, that makes sense. Four options, then:
If that's the case, then the callback will have to return a promise. That seems okay, but I think it'd be more ergonomic for the application developer to return a value synchronously if they want to. |
Are these suggested alternative ctor param types? If so, they can't be or'd because of the promise. But I agree with the premise they could all be useful. I agree it would be nicer not to have to wrap your argument with |
|
Not alternative params, no, just trying to enumerate the usage scenarios. |
|
@snianu 2 general comments
|
sanketj
removed
the
Agenda+
|
To support streaming, and async data fetch, could ClipboardItemData be Promise<(DOMString or Blob or ReadableStream)>? If ReadableStream was used, UA could pull the data when needed. |
Should be doable, yes, but not with a custom controller but with a custom read request perhaps. But I'd rather expect the caller to do the needed conversion with |
|
Note that we have a significant privacy concern with callback-based production of clipboard data. In particular for non-built-in-types this could allow websites to determine the target application. (See WebKit/standards-positions#144 (comment).) |
Delayed clipboard rendering is the ability to delay the generation of clipboard data until it is needed by the target applications. It is especially useful when the clipboard formats supported by target applications are expensive to produce. Delayed clipboard rendering enables web authors to specifically mark delayed rendered payloads in the source application and avoid producing them if they’re not requested by target applications. Our goal is to leverage the existing Async Clipboard API to allow websites exchange large data payloads and improve performance by only producing clipboard payload when it’s needed by target applications.
We detail the proposal in this explainer.
We would love to get feedback on our proposed solution, considered alternatives, and open questions.
Related discussion: w3c/clipboard-apis#41
@snianu @sanketj
The text was updated successfully, but these errors were encountered: