Specify order of flavors exported to macOS's pasteboard

[mbrodesser]

When exporting images to the clipboard (via right-click and "Copy Image"), Chrome and Firefox currently order the flavors differently on macOS's pasteboard.

Chrome adds:

public.tiff
NeXT TIFF v4.0 pasteboard type
public.html
APPLE HTML pasteboard type

Firefox adds:

public.html
Apple HTML pasteboard type
public.tiff
NeXT TiFF v4.0 pasteboard type

Despite Apple's Pasteboard Programming Guide requiring the pasteboard readers to not rely on the order:

"A pasteboard reader must find the data type that best suits its capabilities (if any). Typically, this means selecting the richest type available. In the previous example, a rich text editor might provide RTFD, RTF, and NSString representations of copied data. An application that supports rich text but without images should retrieve the RTF representation; an application that only supports plain text should retrieve the NSString object, whereas an image-editing application might not be able to use the text at all."

some applications, like Pages rely on it. See here.

So for compatibility, the order should be specified.

[annevk]

cc @rniwa @whsieh

[rniwa]

Ah, yes. We had to deal with this problem specifically with Pages. The justification of their behavior comes from:
https://developer.apple.com/documentation/appkit/nspasteboard/1533561-declaretypes?language=objc

The types should be ordered according to the preference of the source application, with the most preferred type coming first (typically, the richest representation).

That Pasteboard Programming Guide's mentioning of preferring the most fluid / rich content type is relatively new thing. It happened as a result of continuity pasteboard since iOS's UIPasteboard didn't use to allow specifying the ordering between types.

This ordering issue is precisely what prompted us to suggest that the types in ClipboardItem should be ordered.

Now, I'm not certain that there is much we can do in terms of specification since this issue is very much platform dependent. We have a bunch of heuristics to determine in what order we want to write stuff into the pasteboard with various compatibility hacks but that's not something we want to standardize as we can't even agree on what to do in macOS vs. iOS.

[rniwa]

I'm gonna just drop this thing right here for ourselves... rdar://problem/35158037

[mbrodesser]

Ah, yes. We had to deal with this problem specifically with Pages. The justification of their behavior comes from:
https://developer.apple.com/documentation/appkit/nspasteboard/1533561-declaretypes?language=objc

The types should be ordered according to the preference of the source application, with the most preferred type coming first (typically, the richest representation).

That Pasteboard Programming Guide's mentioning of preferring the most fluid / rich content type is relatively new thing. It happened as a result of continuity pasteboard since iOS's UIPasteboard didn't use to allow specifying the ordering between types.

This ordering issue is precisely what prompted us to suggest that the types in ClipboardItem should be ordered.

Now, I'm not certain that there is much we can do in terms of specification since this issue is very much platform dependent. We have a bunch of heuristics to determine in what order we want to write stuff into the pasteboard with various compatibility hacks but that's not something we want to standardize as we can't even agree on what to do in macOS vs. iOS.

Which compatibility hacks?

It would be nice specify this somehow. Perhaps even OS specific? Otherwise, it's hard for browsers to stay compatible.

[rniwa]

Ah, yes. We had to deal with this problem specifically with Pages. The justification of their behavior comes from:
https://developer.apple.com/documentation/appkit/nspasteboard/1533561-declaretypes?language=objc

The types should be ordered according to the preference of the source application, with the most preferred type coming first (typically, the richest representation).

That Pasteboard Programming Guide's mentioning of preferring the most fluid / rich content type is relatively new thing. It happened as a result of continuity pasteboard since iOS's UIPasteboard didn't use to allow specifying the ordering between types.
This ordering issue is precisely what prompted us to suggest that the types in ClipboardItem should be ordered.
Now, I'm not certain that there is much we can do in terms of specification since this issue is very much platform dependent. We have a bunch of heuristics to determine in what order we want to write stuff into the pasteboard with various compatibility hacks but that's not something we want to standardize as we can't even agree on what to do in macOS vs. iOS.

Which compatibility hacks?

It would be nice specify this somehow. Perhaps even OS specific? Otherwise, it's hard for browsers to stay compatible.

I don't think that makes much sense. Maybe it cold be an informal note somewhere but it certainly doesn't belong in an official W3C specification for the clipboard API. Having said that, you can take a look at https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/platform/mac/PlatformPasteboardMac.mm and all the changes we made to that file in recent years.

[yisibl]

@tomayac We also need to look into this before Chrome ships the copied SVG.

@snianu Can you confirm that the current implementation in Chrome supports ordering ClipboardItem?
I tested this on a Mac and found that when adding image/svg+xml and text/plain at the same time, text/plain always came first, which prevented Apps such as Pages from pasting SVG correctly.

[snianu]

@snianu Can you confirm that the current implementation in Chrome supports ordering ClipboardItem?

In Chromium, the ordering of formats is defined here. Currently we ignore the ordering of formats in the ClipboardItems.

[yisibl]

@snianu From a simple approach, can we put SvgData in front of TextData? This is also in line with the recommended practice in the operating system, where SVG has richer information.

[yisibl]

Microsoft's development documentation has similar instructions:

Clipboard formats that contain the most information should be placed on the clipboard first, followed by less descriptive formats. A window pasting information from the clipboard typically retrieves a clipboard object in the first format it recognizes. Because clipboard formats are enumerated in the order they are placed on the clipboard, the first recognized format is also the most descriptive.

https://learn.microsoft.com/en-us/windows/win32/dataxchg/clipboard-formats#multiple-clipboard-formats

[yisibl]

It is difficult for the browser to determine which format is richer in information. For example, when copying image/svg+xml and text/html at the same time, which type is richer? This is ambiguous, so it should be left to the developer to decide based on their usage scenario.

In my testing, I found that Safari has the ability to customize the order in which information is written to the clipboard based on the type the developer passes in the ClipboardItem.

For better interoperability, the specification may be able to move in that direction..

[snianu]

Adding @evanstade to this thread as well. I agree that maintaining order in the system clipboard is important for native apps to pick the most desired format during paste. On Windows, the guidance is to place the high fidelity formats at the top:

Clipboard formats that contain the most information should be placed on the clipboard first, followed by less descriptive formats. A window pasting information from the clipboard typically retrieves a clipboard object in the first format it recognizes. Because clipboard formats are enumerated in the order they are placed on the clipboard, the first recognized format is also the most descriptive.

https://learn.microsoft.com/en-us/windows/win32/dataxchg/clipboard-formats#multiple-clipboard-formats

I'm not sure if we can standardize the ordering of the formats in the system clipboard, but perhaps we can maintain the order of the formats in the ClipboardItem as specified by the web authors during navigator.clipboard.write?

[yisibl]

I'm not sure if we can standardize the ordering of the formats in the system clipboard, but perhaps we can maintain the order of the formats in the ClipboardItem as specified by the web authors during navigator.clipboard.write?

Yes, Safari currently does that.

navigator.clipboard.write([
    new ClipboardItem({
        // Adjusting the order here in Safari will change the order in which it is written to the clipboard
        'image/png': new Promise(async resolve => {
            const response = await fetch("https://upload.wikimedia.org/wikipedia/commons/8/89/PNG-Gradient.png");
            resolve(await response.blob());
        }),
        'text/plain': new Promise(resolve => {
            const blob = new Blob([text], { type: 'text/plain' });
            resolve(blob);
        }),
    })
]);

[snianu]

Added Agenda+ tag to discuss in the EditingWG meeting.

[css-meeting-bot]

The Web Editing Working Group just discussed https://github.com/w3c/clipboard-apis/issues/137.

The full IRC log of that discussion

<dandclark> topic: https://github.com//issues/137
<dandclark> github: https://github.com//issues/137
<dandclark> snianu: All 3 browsers have different behaviors
<dandclark> ...: Want to standardize the order that formats get written to clipboard
<dandclark> ...: Clients tend to read topmost format, expect that to have highest fidelity
<whsieh> q+
<dandclark> ...: Chromium has fixed order, I think FF order is fixed but it seems to be wrong
<dandclark> ...: Custom formats seem to get on the top, image formats come after HTML
<dandclark> ...: Safari I haven't tested personally, someone said it preserves formats of web author
<dandclark> whsieh: We do preserve order, that's intentional
<dandclark> ...: Article you pointed out is old. Modern APIs use NSItemProvider where order does matter. Same as Windows, highest fidelity are on the top. Devs are expected to go through types in order of fideliety
<dandclark> ...: Idea is that richest supported format is the one they should consume
<dandclark> ...: In Safari we preserve order for that purpose
<dandclark> snianu: If the web authors write img after text/plain, and the user pastes in native app, does the it get reordered?
<dandclark> whsieh: We preserve order in that case
<dandclark> snianu: I wanted to discuss that; I don't know if web authors are aware that order is important. Would be good to standardize it. Should we define fixed order?
<dandclark> ...: Or leave it up to web authors?
<dandclark> smaug: Should leave it up to the webpage authors, following what Safari's doing. Don't know why Firefox is doing something else, it's some legacy behavior
<dandclark> snianu: FF had some surprising results
<dandclark> smaug: chromium's order is also strange
<dandclark> ...: We should follow Safari's model here
<dandclark> snianu: If we do that, does it affect DataTRansfer APIs?
<dandclark> ...: In Chromium we have fixed order for those, would be hard to have different order based on APIs being used by web authors
<dandclark> ...: In browser process where we interact with sys clipboard, can't know if we should preserve order or the legacy behavior
<dandclark> ...: So that could cause breakage if we change it
<dandclark> ...: Original issue was opened because of image pasting issues in Pages and Keynote.
<dandclark> ...: I only heard reports of Pages failures with Firefox. Has anyone investigated why it's failing?
<dandclark> ...: Someone commented on issue that order is the reason the paste failed. But my experience testing Office and Windows stuff, order matters when pasting images and plaintext.
<dandclark> ...: I don't see issues with paste, svg is still pasted in native app
<dandclark> smaug: I wonder if in Windows apps just do different things vs MacOS
<dandclark> smaug: On Windows it's worked the same since Win32
<dandclark> (That last comment was snianu, not smaug)
<dandclark> smaug: We should investigate if we can follow Safari's model elsewhere, see what breaks. Hard to know otherwise.
<dandclark> snianu: Proposal is to make changes in nightly build in both APIs? Worried about changing DataTransfer. I'm sure there will be breakage in Office.
<dandclark> smaug: What's the behavior in Safari for DataTransfer?
<dandclark> whsieh: Not sure, I think we respect order, if we don't we should.
<dandclark> ...: as we try to align DataTransfer to more modern async clipboard
<dandclark> johanneswilm: Are we proposing spec changes?
<dandclark> snianu: Spec says order is specified by web author.
<dandclark> ...: but spec is iterating clipboard item, doesn't define any order there.
<dandclark> ...: This order is more from system/OS perspective, if you mess that up then native app reads lower fidelity format
<dandclark> johanneswilm: So that's what we're sticking with, FF will try this out
<dandclark> smaug: It can't be just us
<dandclark> ...: Needs to be all browsers on Windows. Also don't know about Linux
<dandclark> snianu: Couldn't find Linux guidance
<dandclark> ...: MacOS & Windows, there are specific guidelines that richest format goes on top
<dandclark> smaug: Could be tricky to change if apps rely on specific behavior on Win
<dandclark> sanketj_: Is it defined about which formats are the most important?
<dandclark> smaug: I don't know which is more important. Would expect maybe HTML because it has more semantic info vs image
<dandclark> whsieh: But if you're copying single image...It's context dependent.
<dandclark> johanneswilm: What are we expecting from this meeting?
<dandclark> snianu: Was wondering if there's real issue here. If no then we're hesitant to change anything since this is very fragile.
<dandclark> ...: If we break copy/paste with old office apps, very hard to fix
<dandclark> ...: I initially proposed to keep Safari's behavior, because that's how the spec is written, but thought deeply about how to make the change in Chromium, ran into this issue that we can't figure what order the author chose
<dandclark> ...: From my own experience here it's very hard to change.
<dandclark> Christine Hollingsworth: Don't have a formal opinion, would be nice to standardize but only if reasonably possible
<dandclark> smaug: Can keep issue open but not be very active
<dandclark> sanketj_: If we mark it as closed, say we'll never do it. Leaving it open says we might do it in future. What new data would be needed such that we might do it ?
<dandclark> smaug: More testing, trying Safari's behaivor on Windows
<dandclark> sanketj_: Seems like it was a FF bug, pasting into Pages. Are you fixing that bug?
<dandclark> smaug: That was fixed years ago, looking at what was changed. Looks like the order was changed
<dandclark> ...: But it's not a generic solution, not following Safari's model. Order is still fixed.
<dandclark> sanketj_: You take image to be highest pri?
<dandclark> smaug: At least on mac
<dandclark> sanketj_: Can you confirm what's your order on different platforms? We could get inventory on different orders browsers have today on different platforms
<dandclark> ...: Based on that we could figure out what we need to change to all be spec compliant
<dandclark> smaug: Sounds good
<dandclark> ...: snianu , smaug , whsieh to document these in the issue for their respective browsers

[EdgarChen]

Summary of Gecko's behavior on different platform

Test script:

const clipboardItem = new ClipboardItem({
  "text/plain": "text",
  "image/png": imageBlob,
  "text/html": "<html></html>",
});
navigator.clipboard.write([clipboardItem]);

Order in system clipboard:

macOS	Linux	Windows
public.png	UTF8_STRING	CF_UNICODETEXT
Apple PNG pasteboard type	COMPOUND_TEXT	CF_TEXT
public.tiff	TEXT	application/x-moz-nativeimage
NeXT TIFF v4.0 pasteboard type	STRING	CF_DIBV5
public.utf8-plain-text	text/plain;charset=utf-8	CF_DIB
NSStringPboardType	text/plain	text/html
public.html	image/png	HTML Format
Apple HTML pasteboard type	image/jpeg
	image/bmp
	image/x-bmp
	image/x-MS-bmp
	image/x-icon
	image/x-ico
	image/x-win-bitmap
	image/vnd.microsoft.icon
	application/ico
	image/ico
	image/icon
	text/ico
	image/tiff
	text/html

On Linux and Windows, we use the order of the formats in the ClipboardItem as specified by the web authors.
On macOS, we put the image on the top.

And DataTransfer behave the same.

[css-meeting-bot]

The Web Editing Working Group just discussed Specify order of flavors exported to macOS's pasteboard.

The full IRC log of that discussion

<sanketj_> Topic: Specify order of flavors exported to macOS's pasteboard
<sanketj_> github: https://github.com//issues/137
<sanketj_> snianu: Not sure has changed since last discussion.
<sanketj_> ...: Chromium has fixed order. Order determined when clipboard item created. Order preserved when writing to clipboard.
<sanketj_> ...: Not sure if it is possible to standardize order of formats written to clibpoard.
<sanketj_> ...: On Windows, there is some historical precedence from legacy apps.
<sanketj_> ...: May be different orders are needed on different platforms.
<sanketj_> sanketj: Last time we discussed we wanted to find out order from each browser. Chrome and Firefox seem to do different things, not sure if can be reconciled without compat problems.
<sanketj_> smaug: Doesn't Safari write based on author order?
<snianu> my zoom client crashed
<sanketj_> snianu: Sounds right. But we may have to leave it up to the UA. Possibly based on platform.
<sanketj_> smaug: Does Chromium have fixed order, regardless of author choice?
<sanketj_> sanketj: Chromium has fixed order, seems to be written in that order on all platforms.
<sanketj_> wshieh: List is order of fidelity? Highest to lowest?
<sanketj_> sanketj: Yes.
<sanketj_> wshieh: Similar concept on Mac/iOS with Webkit, but we go with the developer's ordering choice.
<sanketj_> wshieh: MacOS documentation also old. Points to older APIs.
<sanketj_> johanneswilm: Are we likely to get agreement on a behavior across browsers here?
<sanketj_> smaug: Not sure if not possible or no strong opinion.
<sanketj_> ...: May need investigation from all browsers.
<sanketj_> sanketj: There is risk with making these changes. Is it worth doing?
<sanketj_> johanneswilm: How would this make things better?
<sanketj_> smaug: Just consistency.
<sanketj_> johanneswilm: Spec already mentions you can't rely on the order.
<sanketj_> smaug: Depends on what native apps choose.
<sanketj_> johanneswilm: Let's set this aside for further investigations from other browsers?
<sanketj_> smaug: Why does Chromium have the current order?
<sanketj_> snianu: Not sure why fixed order was chosen.
<sanketj_> ...: Do want consistent order of formats between DataTransfer and async clipboard API.
<sanketj_> ...: Different order between copy/paste and async clipboard API may lead to inconsistent copy/paste for users.
<sanketj_> ...: OS like Windows specify higher fidelity formats go on top.
<sanketj_> sanketj: Doesn't Windows spec mention order?
<sanketj_> snianu: Windows spec doesn't specify exact order, but says highest fidelity format should go on top.
<sanketj_> ...: Risk is that if the authors provide a different order, then that isn't respected.
<sanketj_> ...: Changing browser orders carries risk of copy/paste issues since native apps hard to change.
<sanketj_> sanketj: Can someone post in the issue to confirm the active web dev impact? Original issue seems to be a Firefox bug that fixed a long time ago.
<sanketj_> johanneswilm: If someone can help take this further, let us know. If not, let's move on until there is more info.