TextMetrics.advances should define more details

[kojiishi]

TextMetrics.advances is a recent addition to the spec. It looks very useful, but needs a few more details.

1. Is the array index a code unit, code point, or glyph index? Font Metrics API currently defines it's code point. Does the "character" in this spec imply code point? Is the code point a good index for this member?
2. The current spec says "each advance is...distance from the beginning of the string". The member name advance suggests it's an advance of each glyph. Should it be so, or change the name if authors want cumulative widths up to each index?
3. When the index does not have corresponding glyph (e.g., missing glyph, ligatures, etc.) how would the array look like? I guess the first element has the advance and others are 0?
4. The ordering of the array for RTL/mixed-bidi text is not defined. Is it logical order or visual (after bidi reorder) order? How would RTL runs be represented?

Note this member is also defined in Font Metrics API.

Opinions appreciated: @litherum @FremyCompany @dbaron @jfkthame @fserb @domenic @eaenet

[annevk]

cc @whatwg/canvas

[annevk]

See also #3994.

[litherum]

Is the array index a code unit, code point, or glyph index?

Glyphs are an implementation detail of text drawing libraries. I'm worried about people getting trying to use glyphs as semantic information. For consistency with the rest of the Web Platform, these should be UTF-16 Code Unit.

The member name advance suggests it's an advance of each glyph.

This is almost certainly a mistake. I've never heard of anyone wanting this information, and we shouldn't force web developers to subtract in order to get what they really want.

When the index does not have corresponding glyph?

We already have to solve this problem for text selection with a mouse. We should use whatever we use for that.

We should also have the invariant that the sum of all the advances is the width of the entire string. (So, e.g. combining marks get a 0 advance)

The ordering of the array for RTL/mixed-bidi text is not defined.

As far as I know, every text engine does Bidi reordering before looking up characters in fonts. In order to be implementable in existing engines, we should make this visual order.

[kojiishi]

Is the array index a code unit, code point, or glyph index?

Glyphs are an implementation detail of text drawing libraries. I'm worried about people getting trying to use glyphs as semantic information. For consistency with the rest of the Web Platform, these should be UTF-16 Code Unit.

I agree. We should fix Houdini Font Metrics API as well, if consensus.

One edge case is when one code point is shaped to multiple glyphs. Maybe we can put the sum of advances of such glyphs?

The member name advance suggests it's an advance of each glyph.

This is almost certainly a mistake. I've never heard of anyone wanting this information, and we shouldn't force web developers to subtract in order to get what they really want.

Agree (I assume you meant to prefer each advance, not total from index 0, correct?)

When the index does not have corresponding glyph?

We already have to solve this problem for text selection with a mouse. We should use whatever we use for that.

I believe mouse selection is not defined in the spec, is that correct? Is there any definitions we can use for this purpose? @annevk @domenic

We should also have the invariant that the sum of all the advances is the width of the entire string. (So, e.g. combining marks get a 0 advance)

Good point, agree.

The ordering of the array for RTL/mixed-bidi text is not defined.

As far as I know, every text engine does Bidi reordering before looking up characters in fonts. In order to be implementable in existing engines, we should make this visual order.

Agree it's the most feasible to implement. I don't know how we can present advances in the logical order. On the other hand, if the intention of advances is to know the advance of specific character index, it makes hard to use for authors. Maybe it's fine to start with the visual order, and add another member if authors complain?

[litherum]

We already have to solve this problem for text selection with a mouse. We should use whatever we use for that.

I believe mouse selection is not defined in the spec, is that correct? Is there any definitions we can use for this purpose? @annevk @domenic

Does it have to be spec'ed? Advances are already platform / engine / font-specific.

Similarly, the mapping from glyph index -> character index has to survive font shaping, which is lossy (consider the Contextual Glyph Substitution Subtable inside the MORX table in AAT fonts). This mapping definitely shouldn't be spec'ed.

The ordering of the array for RTL/mixed-bidi text is not defined.

As far as I know, every text engine does Bidi reordering before looking up characters in fonts. In order to be implementable in existing engines, we should make this visual order.

Agree it's the most feasible to implement. I don't know how we can present advances in the logical order. On the other hand, if the intention of advances is to know the advance of specific character index, it makes hard to use for authors. Maybe it's fine to start with the visual order, and add another member if authors complain?

If there is demand, perhaps we can work with the ECMA-402 to provide a UBA? Both visual and logical orders have value for different purposes; we should start with the easy one and open it up to the more complicated one if necessary.

[jfkthame]

The ordering of the array for RTL/mixed-bidi text is not defined.

As far as I know, every text engine does Bidi reordering before looking up characters in fonts. In order to be implementable in existing engines, we should make this visual order.

Agree it's the most feasible to implement. I don't know how we can present advances in the logical order. On the other hand, if the intention of advances is to know the advance of specific character index, it makes hard to use for authors. Maybe it's fine to start with the visual order, and add another member if authors complain?

If there is demand, perhaps we can work with the ECMA-402 to provide a UBA? Both visual and logical orders have value for different purposes; we should start with the easy one and open it up to the more complicated one if necessary.

ISTM that an array of glyph advances in visual order, indexed by code units into the character string, is a fundamentally broken API, and if authors try to build things on top of it they'll end up with code that fails in peculiar ways when faced with complex-script text.

It's not just about bidi reordering; what about Indic-style rearrangement of glyphs such as vowels that appear to the left of the base character? When the shaping engine reorders the glyphs corresponding to "hindi" into the visual order "ihndi", how will a client know that the first element of advances has nothing to do with the first character in the text?

[fserb]

@jfkthame I'm not sure I understand why "an array of glyph advances in visual order, indexed by code units into the character string" is fundamentally broken with your example. Could you please clarify that?

In the example that you gave "hindi" (an let's assume that each glyph here has 10 logical unit of advance size and we end up with 5 glyphs). If there's a rearrangement to "ihndi", the returning advances would be [10, 0, 20, 30, 40]. what would be fundamentally broken about that? The first element of advances would still be the first character in the text. It's still indexed by code units into the character string, if that's what you mean by that.

It seems some of those TextMetrics threads diverged a bit from people guessing what was the original intent of it. It's totally my fault, for not making it way more clear on the original spec what it should have been.

One of the original motivations for this API was to solve things like detecting cursor position, i.e., to answer the question "Where the editing cursor would have to be to be at the left of the glyph associated with this character". Which is exactly what @litherum hints at when they said "We already have to solve this problem for text selection with a mouse. We should use whatever we use for that." We are actually using the same information (more precisely, text edit cursor selection, but they are the same).

Implementation wise (and this was not on the spec writing and is all my fault), most of the issues that I've seen being brought up here were addressed, but they didn't end up into the spec, as I wrongly assumed they were implementation details. For example, @annevk brought up on the other thread "What happens if multiple code points get rendered as a single glyph?". In this case, we return the same advance for both code points.

It's possible that we forgot some cases that should be addressed and the spec needs some clarification (and maybe even directly address some of the cases brought up here? Although I'd argue that WPT could be better for this, but oh well...).

[jfkthame]

In the example that you gave "hindi" (an let's assume that each glyph here has 10 logical unit of advance size and we end up with 5 glyphs). If there's a rearrangement to "ihndi", the returning advances would be [10, 0, 20, 30, 40]. what would be fundamentally broken about that? The first element of advances would still be the first character in the text. It's still indexed by code units into the character string, if that's what you mean by that.

OK, I think that makes it clear that "advances" is the wrong name for this array. Those aren't the advances of the characters (or glyphs); they're positions.

So if a client wants to use this information to draw an underline below the first character of the text ("h"), how should it go about this? Draw a line from positions[0] to positions[1]? I expect that's what most authors would instinctively write; but it'll be wrong.

What if a single character results in multiple rendered glyphs? What if those glyphs are non-contiguous in the resulting visual order? Suppose the first "i" of "hindi" renders as two glyphs (let's call them i1 and i2) that surround the h, so the text renders as the 6 glyphs i1, h, i2, n, d, i. (This doesn't really happen in Devanagari, but does in some other Indic scripts.) What do we expect to be returned in the (5-element) array here?

If this array were renamed positions (or offsets) rather than advances, it would be more understandable, though I'm still concerned about the limitations of exposing a single "position" for a character that actually renders as several separate glyphs (with relative positions that may depend on the context -- e.g. the distance between i1 and i2 above depends on the advance of the intervening h glyph).

Implementation wise (and this was not on the spec writing and is all my fault), most of the issues that I've seen being brought up here were addressed, but they didn't end up into the spec, as I wrongly assumed they were implementation details. For example, @annevk brought up on the other thread "What happens if multiple code points get rendered as a single glyph?". In this case, we return the same advance for both code points.

This means a client cannot distinguish between a pair of spacing characters that happen to ligate and a base letter followed by a non-spacing mark, which ISTM is a useful distinction when dealing with issues such as caret positioning or selection highlighting.

[fserb]

So if a client wants to use this information to draw an underline below the first character of the text ("h"), how should it go about this? Draw a line from positions[0] to positions[1]? I expect that's what most authors would instinctively write; but it'll be wrong.

That's a good question. The actual algorithm for doing that would probably be "for LTR, draw frompositions[0] to the next position that is greater than position[0] or width if there are none" which is not horribly complex, but prone to errors.

What if a single character results in multiple rendered glyphs? What if those glyphs are non-contiguous in the resulting visual order? Suppose the first "i" of "hindi" renders as two glyphs (let's call them i1 and i2) that surround the h, so the text renders as the 6 glyphs i1, h, i2, n, d, i. (This doesn't really happen in Devanagari, but does in some other Indic scripts.) What do we expect to be returned in the (5-element) array here?

I was not aware that we had non-contiguous multiple glyphs from the same character. I'm almost sure that Chrome doesn't handle this at all, not sure about Safari and Firefox. Do you have a real text example of this that we could test against?

This means a client cannot distinguish between a pair of spacing characters that happen to ligate and a base letter followed by a non-spacing mark, which ISTM is a useful distinction when dealing with issues such as caret positioning or selection highlighting.

Sorry. I misstated that. If two code points get rendered as a singly glyph, there are two options: if they are separate unicode graphemes, we return the linear interpolation of the advance of the glyph for each code point (e.g. [0, 5] for a glyph with width 10). The linear interpolation is mostly wrong, but seems to be the current solution on all browsers, afaik. If they are a single unicode grapheme, we'd return the same position ([0, 0]).

[jfkthame]

I was not aware that we had non-contiguous multiple glyphs from the same character. I'm almost sure that Chrome doesn't handle this at all, not sure about Safari and Firefox. Do you have a real text example of this that we could test against?

This occurs in scripts such as Malayalam: the sequence U+0D15, U+0D4A is two characters (a consonant ക and a vowel ൊ) where the two glyphs that make up the vowel render on either side of the consonant: കൊ.

AFAIK, most (all?) browsers currently behave as if this entire cluster were a single glyph, for selection/cursor placement purposes; but it's not, it is three distinct glyphs (and none of them are zero-width, fwiw). I think a canvas API client should be able to determine things like this, as the point of using canvas is (at least in part) to give the author low-level control over exactly what/how they're drawing.

Perhaps we should first be creating canvas APIs to draw and measure glyphs (as opposed to characters), along with APIs to access the text-shaping process (mapping from a string of characters, with associated font/style information, to an array of glyphs and positions).

[litherum]

ISTM that an array of glyph advances in visual order, indexed by code units into the character string, is a fundamentally broken API, and if authors try to build things on top of it they'll end up with code that fails in peculiar ways when faced with complex-script text.

This is what the web author would need to implement text selection. (But only if the advances are layout advances, not paint advances.)

EDIT: Yeah, UBA makes a mess of this. You're right, it's pretty broken.