Add Web Custom format

[snianu]

Closes #165

For normative changes, the following tasks have been completed:

• Modified Web platform tests (link to pull request)

Implementation commitment:

• WebKit (https://bugs.webkit.org/show_bug.cgi?id=)
• Chromium (https://bugs.chromium.org/p/chromium/issues/detail?id=)
• Gecko (https://bugzilla.mozilla.org/show_bug.cgi?id=)

---

Preview | Diff

[snianu]

@annevk @mbrodesser This PR has the web prefix changes. PTAL. Thanks!

[annevk]

Upon doing code inspection in Gecko we found that the legacy clipboard API might have some support for custom types as well. @EdgarChen is investigating this. We don't really want to end up with a situation where we have two incompatible custom types APIs.

It might be that other browsers have something similar as there were at least some hints of cross-browser collaboration?

[snianu]

@annevk Chromium and Safari have browser specific custom type that is undocumented and doesn't interop with other apps. The format parsing rules are convoluted since it uses browser specific pickling types for serialization/deserialization. This proposal aims to standardize the web custom formats and the format structure so its well documented and is interop across apps on a platform. This doesn't affect the existing pickling types.

In Firefox, I see that it allows writing custom types in setData, but I don't see any restriction on the number of custom types that can be registered. This is a major security issue on Windows and Linux because of the restriction on the number of clipboard formats that can be registered with the OS. This proposal addresses this issue by introducing a fixed set of web custom formats and a map that maps the mime types to native web custom formats.

[BoCupp-Microsoft]

Thanks for the changes! I think if you address these comments I'll be ready to approve for a merge.

[BoCupp-Microsoft]

why don't you call this |customFormatKey| and include the parse a mime-type steps in this algorithm to transform that into a "web " prefixed and parsed name that can be used to construct the clipboard item. the name you construct should probably be an output parameter.

[BoCupp-Microsoft]

Also, to make it clear what's an acceptable key, when parsing a mime-type, you actually use the ParseMimeTypeWithoutParameter function in Chromium, so can you clarify in the steps that you take the type and sub-type and concatenate those outputs from the parse a mime-type algorithm with '/', but throw away the parameters?

https://chromium-review.googlesource.com/c/chromium/src/+/3650952

[BoCupp-Microsoft]

Lastly, the parse a mime type can fail (by returning failure per the spec) so can you clarify that if failure is returned what you will do? I believe this function should also return failure and then you simply don't append anything to the collection of data used to construct the clipboarditems that will be given to the author.

[snianu]

To be a bit more resilient to the parsing mime type errors, I added steps to ignore the mime types that returned failure during parsing.

[BoCupp-Microsoft]

This step is getting a bit long and does a lot. Also, it sounds like if data is a copy of the system clipboard data that there will be more than just the mandatory mime-types and web custom formats in data.

I believe we want a loop over the mandatory data types where we insert each into a data object that will be used to construct a single clipboarditem with multiple representations corresponding to each of those types.

Additionally, we want a step here to first read the JSON from the Web Custom Format Map, and then we can loop over its keys too. For each key we can obtain the name of the format corresponding to that mime-type on the system clipboard, read it, and then add its blob to data using a new key that is the result of running the key we got from the JSON through the parse a mime-type algorithm with "web " prepended.

Most of that can happen in the read helper function you have defined. Consider returning the "web " prefixed and parsed name from that method in addition to the blob itself.

[BoCupp-Microsoft]

Two additional nits:

1. Be sure to include MAY be sanitized as a step as you iterate over the mandatory data types to preserve what you added in your previous PR.
2. Instead of looping over mandatory data types, you can loop over a set of data types supported by the user agent as a well-known format, which must at least include the mandatory data types listed in this spec.

[snianu]

Added these additional steps.

[BoCupp-Microsoft]

This should say something like insert the JSON serialized representation of |webCustomFormatMap| into the clipboard.

[snianu]

I added a step above to insert the mime type as key and |webCustomFormatIdx| as value into the |webCustomFormatMapString| using a JSON serializer.

[BoCupp-Microsoft]

Before inserting mimeType, you should be stripping the "web " prefix. I think the what follows the space is guaranteed to be a parsed as a mime-type value per the steps in the construction of the clipboarditem.

[BoCupp-Microsoft]

It looks like we didn't update the ClipboardItem constuctor. We have a step in there that runs the parse a mime-type algorithm, but it needs to test and allow for "web " prefixes.

[snianu]

Done.

[EdgarChen]

This proposal aims to standardize the web custom formats and the format structure so its well documented and is interop across apps on a platform. This doesn't affect the existing pickling types.

Do we care about the interop across legacy clipboard API and async clipboard API for custom type? E.g. write custom type via legacy clipboard API and read via async clipboard API, or vice versa.

[snianu]

Do we care about the interop across legacy clipboard API and async clipboard API for custom type?

@EdgarChen The async clipboard API model is fairly different, so I'm not sure if we can interop with legacy types. The web custom types are stored inside a JSON map structure, with fixed number of native web clipboard formats, and a well-defined format structure that helps the native apps parse the content of the format. At least in Chromium, the parsing of the custom types requires browser specific pickle types to serialize and deserialize the content, so it's not possible to add support for this format to the new web custom format.

[EdgarChen]

The web custom types are stored inside a JSON map structure, with fixed number of native web clipboard formats, and a well-defined format structure that helps the native apps parse the content of the format.

@snianu We could also write arbitrary type with "web" prefix via legacy API, e.g.

document.addEventListener("copy", function(e) {
  e.preventDefault();
  
  let clipboardData = e.clipboardData;
  clipboardData.setData("web text/foo", "foo");
  clipboardData.setData("text/bar", "bar");
});

I would expect the type with "web" prefix to follow the well-defined format/structure, too. So it could be interop with async clipobard API/browsers/native apps. for example, if we write the data into clipboard via above example and read clipboard data via async clipboard API

const items = await navigator.clipboard.read();
items[0].getType("web text/foo"); // get "foo"
items[0].getType("text/bar"); // promise reject 

(and vice versa)

Yes, this introduce a behavior changes in legacy API, but I think this also make web custom type less surprise to user? And the impact in legacy API should be small?

[snianu]

@EdgarChen While it's true that you could write custom types with "web " prefix using DataTransfer APIs, those custom types do not follow the same model - there isn't a restriction on the number of custom format types that can be written by the web authors, no web custom JSON map for mapping the web custom types to a fixed number of native web clipboard formats etc.
We don't want to make any changes in the legacy DataTransfer APIs as it has high regression risks and the model is fairly different there(at least in Chromium). The native apps on Windows do not know how to parse Chromium specific custom types, so it's not possible for them to exchange formats between web and native apps.
Chrome doesn't expose the custom mime types exactly as is - it exposes it via a Chromium specific custom format that uses browser specific parsing types, and it is not standardized, so the native apps don't understand how to parse this structure.

[EdgarChen]

Per https://w3c.github.io/clipboard-apis/#dom-clipboarditem-types, type getter of ClipboardItem returns types array, but it is not clear how does types array be constructed. It would be nice that we could have some steps for that.

Not sure if this makes sense: maybe we could construct types array from list of representations, but which means we would need to read custom types first as we would like the custom types listed at the top.

[snianu]

I agree. We should definitely add this to the spec.

[polx]

Hello @snianu ,

I've tried to review the current state after I have seen your presentation at the TPAC.

Here are my rather nitty gritty comments:

• sec 7.2 (ClipboardItem Interface): should is custom really be written in two words and not in one? The link in the version I could build did not work but that might be a local issue.
• the spreadsheet example close to that is nice, could it not also include a custom type? (e.g. containing text/csv alternative?)
• A web custom format has is custom set to true. => A representation in a web custom format has is custom set to true ?
• 7.3.3: write algorithm: f type is not in the mandatory data types list and has "web ": did you not mean “or has...” ?
• write blobs and option to the clipboard:
  • step 1: why is the variable called webCustomFormat ? It should be formats or something universal, right?
  • substeps under 2.2: "If formatString is empty then follow the below steps:": the steps here does not define a (non-empty) formatString; a value is however needed in order for the recipients to read (suggestion: “set formatString to a randomly chosen format name”)
  • step 2.3: Let payload be the result of UTF-8 decoding item’s underlying byte sequence. : this must be a misunderstanding as it would only apply to text. Maybe the whole section was about text? Then it was not readable
• the steps of “write web custom formats” sound quite solid to me but why is it independent from the previous?

Tell me if I should make my comments more precise (e.g. with line numbers).

I am quite excited by the implementation found in Chromium (and blogged about it). I suspect that we are getting close to acceptance, at least provided that Mozilla will consider implementing as it seems it said at the start.

thanks and sorry to be slow to have reviewed.

Paul

[snianu]

Thanks @polx for the review and apologies for the delay!

A web custom format has is custom set to true. => A representation in a web custom format has is custom set to true ?

Note that in the spec it's referred to as [=representation/isCustom=]

step 1: why is the variable called webCustomFormat ? It should be formats or something universal, right?

This is referring to the custom format with a web prefix, so it can be anything. In the below step, we strip out the web prefix:
If |webCustomFormatString| has "web "("web" followed by U+0020 SPACE) prefix, then remove the "web " prefix and store the remaining string in |webCustomFormat|'s {{Blob/type}}'s [=MIME type/essence=], else abort all steps.

step 2.3: Let payload be the result of UTF-8 decoding item’s underlying byte sequence. : this must be a misunderstanding as it would only apply to text. Maybe the whole section was about text? Then it was not readable

This is by-design that we write raw bytes sequence to the clipboard for the custom formats so the reader has to know how to parse its content (which is also by-design).

the steps of “write web custom formats” sound quite solid to me but why is it independent from the previous?

This is because we need to perform an additional step to produce a native clipbaord format and insert the key-value pair into the webCustomFormatMap. This step is not performed for mandatory data types.

[snianu]

@sanketj could you please take a look at this PR so we can merge it into the clipboard repo? Thanks!

[snianu]

@garykac Thank you so much!