Allow surrogates in content, issue #895

[mihnita]

The goal was to allow surrogates in localizable content, but not in names or anything else that is "code".

This change reflects that agreement.

[catamorphism]

This would allow surrogates in both text and literals. I don't have an opinion on whether they should be included in literals or not, just pointing out that this change makes that possible since quoted-char depends on content-char and quoted-literal depends on quoted-char.

[mihnita]

Touching only text-char means that potentially localizable content does no allow surrogates.
For example:

... {img src=... alt=|This is localizable text|...}...

That would be a quoted literal.

Allowing surrogates in the quoted literal might also allow it in a couple of other places:

... {|allow it here| :string} ...

and

.match
|also here| {{ ... }}
* {{ ... }}

That is again non-code, and potentially localizable.

---

To help understanding what is going on I created a graph (click it for full size).

It represents the mf2 abnf as it is today, with colors added to the interesting nodes, and all links to ws, o, s, bidi commented out to reduce clutter.

[mihnita]

This would allow surrogates in both text and literals

I know. It is intentional, and I explained why.

As you can imagine, that graph took a while :-)
So I had it all prepared, and I was writing the explanation.

But you were too fast :-)

[aphillips]

I think this has to be discussed again if we propose to include literal. While I agree that the literal examples are valid, I think that makes my proposed solution more attractive (forbid surrogates in the grammar but permit implementations to not check for them (except in identifiers). Unpaired surrogates have no valid use and, as noted elsewhere, the first UTF-8 coder will eliminate any found in a message.

Note that this PR is incomplete, since the syntax spec also has to be updated.

[eemeli]

My understanding of the consensus reached during the 2024-10-07 call was that we'd be allowing unpaired surrogates in text only.

If allowing surrogates in text is not enough and we need to consider also allowing them in quoted literal content, I think we need to see some examples of such use to consider their utility. In text I can imagine them showing up from processes that are doing dumb concatenation or string slicing, but in literals we're talking about content that needs to be processed within a framework that has a pretty good understanding of MF2, and content that is further processed within MF2 rather than only in a consumer of the formatted output of MF2.

I would also strongly prefer to see such examples or other rationalisation for unpaired surrogates in quoted literals presented here, before we spend more meeting time on this topic.

My own base assumption is that any unpaired surrogate is an indicator of some broken localization process, and that silently ignoring them would be hiding an error.

[mihnita]

allowing unpaired surrogates in text only.

Literals are text. Sometimes localizable:

... {img src=... alt=|This is localizable text|...}...

an indicator of some broken localization process, and that silently ignoring them would be hiding an error.

Agree.
But hiding that minor error results in a usable application for the end user instead of an exception.

[macchiati]

My thoughts are that we should allow unpaired surrogates in the text in *input parameters*, but I don't think we need to allow them in the message itself (literals, etc). That is, in parsing a message format, we could throw an exception if an unpaired surrogate were encountered, but the formatting software can accept unpaired surrogates in data passed in, and treat them as if they were reserved, or convert to U+FFFD "replacement character".

…

On Sat, Oct 12, 2024 at 3:15 AM Eemeli Aro ***@***.***> wrote: My understanding of the consensus reached during the 2024-10-07 call was that we'd be allowing unpaired surrogates in *text* only. If allowing surrogates in *text* is not enough and we need to consider also allowing them in *quoted literal* content, I think we need to see some examples of such use to consider their utility. In *text* I can imagine them showing up from processes that are doing dumb concatenation or string slicing, but in *literals* we're talking about content that needs to be processed within a framework that has a pretty good understanding of MF2, and content that is further processed within MF2 rather than only in a consumer of the formatted output of MF2. I would also strongly prefer to see such examples or other rationalisation for unpaired surrogates in *quoted literals* presented here, before we spend more meeting time on this topic. My own base assumption is that any unpaired surrogate is an indicator of some broken localization process, and that silently ignoring them would be hiding an error. — Reply to this email directly, view it on GitHub <#906 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMCCZGRCB3KVZU3PUTDZ3DZC5AVCNFSM6AAAAABPZQ37F2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBYGUYDSMZSG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[aphillips]

(chair hat)

In the 2024-10-14 call we agree to allow unpaired surrogates in text and literals, including changing mentions of characters to mentions of code points in the spec as needed and including a health warning.

[mihnita]

Updated the non-abnf part of the spec.

In most cases there was no need to change code point to code units.

From the Unicode spec, Chapter 3, section "3.8 Surrogates":

D71 High-surrogate code point: A Unicode code point in the range U+D800 to U+DBFF.
D72 High-surrogate code unit: A 16-bit code unit in the range D80016 to DBFF16, used in UTF-16 as the leading code unit of a surrogate pair.
D73 Low-surrogate code point: A Unicode code point in the range U+DC00 to U+DFFF.
D74 Low-surrogate code unit: A 16-bit code unit in the range DC0016 to DFFF16, used in UTF-16 as the trailing code unit of a surrogate pair.

Just because in localizable elements we allow code points in the surrogates range it does not mean that all of our processing and spec are all of a sudden written in terms of code units.

I think that the changes I made are sufficient.
I also included wording that unpaired surrogates are "likely an indication of mistakes or bad tooling" and a recommendation the linting can be use to detect and prevent them.

[aphillips]

Thank you @mihnita for getting this done before our next call!!!!

I like the changes generally, but am making some suggestions below.

[aphillips]

This is fine, but I think it would be okay to omit it too.

[eemeli]

How/why is this important to note as a security consideration?

[aphillips]

I previously suggested the same thing:

I... don't think this is a security consideration? Maybe move this note to the section on message, just after the note that begins:

This syntax is designed to be embeddable into many different programming languages...

I think that location is a good one because we put a whole wodge of notes there that are "read once and forget" about the content of messages.

[eemeli]

The specifier about where surrogates are valid is unnecessary esp. in this context, where the other "potentially confusing content" is for the most part only allowed in the same.

Suggested change

	U+100000 through U+10FFFD), unassigned code points, unpaired surrogates in messages and
	quoted literals only (U+D800 through U+DFFF), and other potentially confusing content.
	U+100000 through U+10FFFD), unassigned code points,
	unpaired surrogates (U+D800 through U+DFFF), and other potentially confusing content.

[eemeli]

This note would be better placed a bit further down in this section, and it needs to be separated from the surrounding content by an empty line. This would also be a more appropriate place for the concerns that are currently hidden away in the Security Considerations appendix.

[aphillips]

@mihnita Per today's call, ready to merge?