[DESIGN] Bidi usability

[aphillips]

Addresses #746.

Provide a design for bidi isolation of MFv2 syntax elements, allowing users and tools to provide messages in right-to-left languages that look normal or have minimal disruption from the Unicode Bidirectional Algorithm.

This design does not require the use of isolates or markers, so it is still possible to create messages that look funky.

This design ignores the UAX31 whitespace definition requirements addressed by #673. Note that the design provided here is more closely tailored to ensure isolate sequences, once opened, are closed and that they tightly wrap only the things desired. There is some funkiness around name that we might improve (see the doc).

[eemeli]

I didn't add line comments for them, but I continue to note that we should be consistent in calling this spec "MessageFormat 2" or "MF2".

[eemeli]

Why LRM/RLM rather than isolates?

Why allow for RLM?

[macchiati]

1. Some implementations don't handle bidi isolates well yet.
2. The LRM and RLM are not stateful, and may be preferred in some circumstances.

[aphillips]

We might allow paired isolates inside expressions. It might not be an either/or. Allowing LRM makes it easy to clean up the contents of an expression:

.input {$م1صر :م2صر} <- no bidi controls
.input {$م1صر‎ :م2صر} <- one LRM right after the id with 1 in it

I included RLM (and probably should have included ALM U+061C) so that an RTL literal or name that ends with a neutral can display correctly:

{م123+ :foo} <- with no RLM
{م123+‏ :foo} <- with RLM after +

[eemeli]

Given that we allow a number of neutral direction characters also in name-start, doesn't the same apply to the beginning of the name as well?

From an automation PoV, using FSI/PDI to wrap names seems like it would "just work", whereas LRM/RLM/ALM would require inspecting the contents of the string to figure out what might be needed.

[aphillips]

If we are strictly talking about machines doing the "bidi annotation", tightly wrapping with isolates will generally work.

FSI is not always going to get the right results, as not all tokens have the correct direction strongly directional character nearest the front. There is an element of judgement (machines don't have enough information, generally, to decide this, although sometimes they do).

But going back to my initial statement: humans write these strings and create translations of these strings. Sometimes the easiest way for them to make the message look correct is to add a strongly directional mark vs. wrapping. (Note that ICU produces marks on some number and date formats to coerce proper display).

You are correct that the design document should call out these use cases separately so that the reader (and ultimately the WG) can weight supporting these mechanisms appropriately.

[eemeli]

@macchiati:

Some implementations don't handle bidi isolates well yet.

Is there a list available anywhere of software that does not yet support bidi isolates?

[eemeli]

Should we also allow for an LRI/FSI pair immediately inside expressions and markup, or is there a reason not to do so?

[aphillips]

We could do that also. It doesn't solve the problem of expression/markup internal bidi, though.

[eemeli]

I'm mostly here thinking of content like:

a = 'אחד'
b = 'שתיים'
s = a + '{$' + b + '}'

where we have an RTL variable name inside a placeholder in an RTL pattern.

How, except with an LRI/FSI pair inside the braces, can we get that to render so that the $ is to the left of the name?

[macchiati]

See #discussion_r1542105763
For those implementations, RLM/LRM are the best one can do.

[eemeli]

Do all editors reset the paragraph direction after a newline? For example, if there's a newline between an LRI and an FSI, how is the paragraph direction of the second line determined?

[macchiati]

The normal application of the bidi algorithm requires a reset on each paragraph, wherein a newline breaks paragraphs.

"The algorithm reorders text only within a paragraph; characters in one paragraph have no effect on characters in a different paragraph. Paragraphs are divided by the Paragraph Separator or appropriate Newline Function (for guidelines on the handling of CR, LF, and CRLF, see Section 4.4, Directionality, and Section 5.8, Newline Guidelines of [Unicode]). Paragraphs may also be determined by higher-level protocols: for example, the text in two different cells of a table will be in different paragraphs."

[aphillips]

@macchiati is correct. That's why it's called "paragraph direction". Note that newlines don't help us that much: they are optional in our syntax (outside literals) and technically normalize to space (or nothing). That is, the newline doesn't help us if we end up writing the message as a single-line.

[eemeli]

Ok, so given that we allow for newlines within "code" and, specifically, expressions, I think we need to account for that so that we can keep the direction of the code as left-to-right, even when the first strongly directional character on the line is RTL.

As I understand it, not even an LRI/FSI pair inside the braces is always enough to keep the $ on the left side of its name if it's preceded by a newline:

a = 'אחד'
b = 'שתיים'
s = a + '{\u2066\n$' + b + '\u2069}'

[aphillips]

That's correct. Getting the sigils to stay on the left side needs a base direction of LTR. An LRM doesn't help in your example either (except to prevent spillover with the following annotation if there were any). My proposal is not 100% bulletproof (and requires some action on the part of tools or users).

A bulletproof design would require more isolates and would probably be limited to using LRI/PDI pairs. It would be difficult to work with, given that there would be a lot of invisible control characters inside subcomponents of an expression, e.g.:

<LRI><LRI>option<PDI>[whitespace]=[whitespace]<LRI>value<PDI><PDI>

[macchiati]

The W3C internationalization group might have some information on browsers, but one would also have to consider other applications and tools.

…

On Thu, Mar 28, 2024 at 12:59 PM Eemeli Aro ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In exploration/bidi-usability.md <#754 (comment)> : > +Permit the use of LRM or RLM controls immediately following: +- name (note that this includes _identifiers_ as well as names of + _functions_, _variables_, and _unquoted_ literals @macchiati <https://github.com/macchiati>: Some implementations don't handle bidi isolates well yet. Is there a list available anywhere of software that does not yet support bidi isolates? — Reply to this email directly, view it on GitHub <#754 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMFCLVV6ICVGRTSJZATY2RSAHAVCNFSM6AAAAABFLG4F7GVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSNRXGE2TOMJWGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[aphillips]

@macchiati noted:

The W3C internationalization group might have some information on browsers,
but one would also have to consider other applications and tools.

@eemeli already spotted this: https://www.w3.org/International/i18n-tests/results/bidi-algorithm

The browsers already support isolates. Other environments, of course, may not (although much progress has been made there too)

[catamorphism]

Feel free to ignore some of my comments if you think it's already clear to a less naïve reader. However, there are a few things that are really ambiguous to me, like the syntax of examples.

[catamorphism]

Is this worth adding to the "constraints" section? Constraint: we must allow bidi controls as either literal (interpreted by the MF2 parser) or escaped (treated as regular text). (Their position introduces an implicit escape.)

[aphillips]

Not sure I follow? The point of the note is to show that bidi controls are just normal text inside literal contexts (the body of a pattern or inside of quoted literals)

[catamorphism]

Suggested change

	- name (note that this includes _identifiers_ as well as names of
	_functions_, _variables_, and _unquoted_ literals
	- name (note that this includes _unquoted_ literals_, _identifiers_, and _variables_;
	and that _identifiers_ include the names of _functions_.)

(It's a bit confusing to say that an unquoted literal has a name.)

[aphillips]

Perhaps, but that's how unquoted is defined:

unquoted       = name / number-literal

[catamorphism]

I don't have time to re-read in detail, but given that it's a design doc, I think that's fine; I followed it for the most part, and the parts that weren't clear to me are probably due to my lack of familiarity with the area.

[aphillips]

@eemeli Per today's (2024-04-08) call, I added optional isolates in expressions and markup.

A couple of tricky things here. One is closing markup. The syntax currently makes clear that the opening markup sigil is not attached to the opening bracket. but the standalone closing sigil is ambiguous. It's currently attached to the the closing bracket (as no space is permitted): /}. However, if we put the closing isolate before the closing slash, the slash will be mirrored in an RTL context (like so):

(Recall that the brackets are mirrored because they are outside the isolate. The one on the left in the above picture is the trailing bracket.)

This adds pressure on the parser, since there are four "end of markup" sequences: }, \u2069}, /} and /\u2069}.

[eemeli]

This adds pressure on the parser, since there are four "end of markup" sequences: }, \u2069}, /} and /\u2069}.

Only two of those are valid at a time, though, as it's not ok to skip the end if you start it.

[eemeli]

I still think this is too messy, and doesn't solve the problem as well as isolation, but I'm ok with considering that separately.

My preferred overall solution would be to:

1. Optionally LR/RL/FS -isolate quoted-pattern, quoted, and name;
2. Optionally LR-isolate expression; and
3. Allow for a single LRM after a newline in code, or may at the end of whitespace containing a newline.

Put together, that should allow for rendering all code as LTR, and all possibly-RTL content as RTL.