Support font ligatures

[Tyriar]

Support being removed in #938

It's certainly still possible, the renderer needs to know which characters to join though.

[mofux]

I think we are not interested into the regular ligatures (in fact, it would be bad to enable ligatures for li and such), but for things like ==, !==, => and so on. Here's a nice list of ligatures supported by the fira code monospace font:
https://github.com/tonsky/FiraCode/blob/master/showcases/all_ligatures.png

[LoganDark]

What would happen if half of the ligature is a different color? How would you handle that?

[Tyriar]

@LoganDark does that even work with regular ligatures? Or are they split up when they're different colors?

[LoganDark]

No, and no.

[devsnek]

i wonder if it would be possible to pull render code from txtjs, as they have figured out how to render ligatures, although i think they manually draw the text. http://txtjs.com/examples/Text/ligatures.html

[Tyriar]

@devsnek I don't think it's an issue actually doing the rendering of the text. The issue is knowing which character join so they can be drawn together (Currently "==" is drawn as "=" and "=", not "==").

[devsnek]

@Tyriar wouldn't the font renderer take care of that without our intervention

[Tyriar]

@devsnek yes, but each cell in the grid is drawn individually with a few exceptions (emojis, wide unicode chars). Ligatures need to somehow be included in that list. Look at #938 for more context

[LoganDark]

@devsnek IT'S YOU

disclaimer: completely off topic, just a random comment

[Qix-]

@LoganDark does that even work with regular ligatures? Or are they split up when they're different colors?

If we're going by how other emulators handle them, they do get split up if they are different colors, as one would expect. This is almost a requirement since this allows symbols to be correctly represented by some language highlighters in ViM, for example.

I think it's entirely acceptable to show the individual symbols if the render mode is not the same for all of the underlying characters.

[LoganDark]

@Qix- My suggestion then would be to draw all the text at once and then do coloring in post. That would eliminate any issues with ligatures, and wouldn't require detecting ligature pairs (although it would break compatibility with variable-width fonts, or even monospace fonts that are slightly off/don't have integer widths)

[Qix-]

@LoganDark multi-colored ligatures would look bizarre and there would be no clear way to color them IMO.

[Tyriar]

Yeah I don't think multi-color ligatures would work. It also goes against how they work underneath, where a single glyph is drawn at the start, not multiple.

As a clarification, this is waiting on a good solution for detecting which character sequences have ligatures. To do this properly it would probably involve low level code that checks font files (and thus would need to be a native node module and not work for web consumers), I don't think this information is exposed to the web platform.

[Spaxe]

Now that hyper released 2.0.0 stable, maybe ligature workarounds need a higher priority.

[iamstarkov]

ref vercel/hyper#914 (comment)

[princjef]

Determining the glyph mappings manually is a tough nut to crack. From what I can tell, making a decent experience out of this would require the following:

1. Map the selected font family back to the file/buffer containing the actual font data (otf/ttf/woff/etc)
2. Parse the data from the GSUB table of the font and translate that into a sensible set of rules for glyph replacement
3. Pass some sort of map or mapping function to xterm to determine what to render for a given character sequence

I've done some initial poking around with Fira Code (specifically its nerd font variant) to try to figure out how difficult each step might be. I haven't yet decided whether I'm feeling ambitious enough (or care about font ligatures enough) to take this on, but here's what I've found so the knowledge isn't lost:

• There is no way that I can find to fetch the font data using browser APIs, so this won't work as a direct feature of xterm.js, but more likely as a separate package/extension with a hook exposed by xterm.js

• Mapping the CSS font-family name of a font back to its font file in Windows is painful but appears doable. So far the only way I've found is to fetch everything in %WINDIR%\Fonts and parse every file I find (spoiler: it's really slow). Haven't tried other platforms yet. (Note: I also tried lifting the name from the registry but the naming doesn't line up for some fonts such as the ones from nerd fonts. They use a "preferred" family and subfamily which isn't picked up in the registry's name but is used in the css font-family. If you're curious, the registry key is in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Fonts)

• There's a library called opentype.js which does full parsing of OpenType font tables and even has a font.stringToGlyphs(str) function which handles basic ligatures, but the ligatures for Fira Code (and several if not all of the other common ligature fonts) use a feature called contextual alternates which is not yet supported by opentype.js (listed under Planned). The necessary GSUB table is parsed to JSON for you though, so theoretically all that's missing is the interpretation of the table data.

• Fira Code ligatures (and I think others) actually replace the original glyphs with equal numbers of glyphs, rather than a single extra wide one (to maintain the properties of the monospace font). For a string like ==>, the font will end up telling you to replace it with two "LIG" characters (essentially an empty space), followed by the actual glyph for the ligature, which is still technically only one monospace character wide. Despite being ostensibly single width, the path of the last character extends beyond the left side of the character's bounding box to cover the space occupied by the two LIG characters before it. See below for a visual (0 and 600 are the sides of the character's box). I don't know if this complicates the task of the renderer or if it would have to be transformed before being passed to xterm.js, but something to be aware of.

• Another wrinkle in this is determining when to re-evaluate the ligature. For example, if I type = four times consecutively, the desired behavior would be for me to see a single equals, then a double equals ligature, then a triple equals ligature, then four separate equals signs. There are actually mappings in the contextual alternates rules for clearing out the ligature (i.e. if the current input is '=' and the previous three characters are '===', don't remap it at all), but we'd have to figure out how to apply those rules.

• OpenType is complicated. Admittedly, I'm not a font rendering expert, but the number of possible variations that lead to different types of rendering is pretty extensive. Without a built-in library that does the mapping for us, I think the most reasonable way to attack this is incrementally. Fira Code specifically uses Chaining Context Substitution Format 3, but I'm sure there are other popular fonts that use different ones. Since each has slightly different semantics, it probably makes sense to start with one and go from there.

[mofux]

@princjef Thanks for sharing your explorations, really really helpful! I have also been putting some thoughts into this topic a couple days ago, and I came to the following conclusion:

• Detecting ligatures in webfonts using a build-in API seems impossible (just as you describe)
• There are certain ligatures that don't make sense in a terminal, even if the font supports it (e.g. li)
• There is a very small subset of ligatures that make sense to be supported, namely most of the Fira Code ligatures.
• Adding support for drawing and clearing a character that spans multiple cells is very hard to implement with our current single-character based rendering approach (CharAtlas).

TBH, I don't think it's worth the effort to support ligatures at the current state 😔

[LabhanshAgrawal]

I've opened a pr princjef/font-ligatures#22 as the first step in getting this done

[LabhanshAgrawal]

The pr's open for quite some time now, what to do?

[Tyriar]

I reached out to @princjef on Teams.

[LabhanshAgrawal]

I have opened a pr with the discussed idea. Do check it out and test. #3264

[LabhanshAgrawal]

As #3264 is merged now, what's the next step?

[Tyriar]

@LabhanshAgrawal just implemented character joiner support in the dom renderer with #3285, webgl still needs to happen which shouldn't be too difficult as well #3094. Pulling out the xterm.js only bits from microsoft/vscode#34103:

• DOM renderer support
• WebGL renderer support
• Hook up ligatures in demo
• Set up some puppeteer tests that verify on all renderers (dom, canvas, webgl) that ligatures are working
  • This might be difficult with the permissions 🤔
• Slim down the dependencies - I need to do a once over to see the state here
• Leverage font access API

[LabhanshAgrawal]

I can take a look at webgl if needed, but it will take some time as I'm not familiar with it at all right now.

[Tyriar]

@LabhanshAgrawal I'm on it atm 🙂

[LabhanshAgrawal]

That's great then 😊👍

[LabhanshAgrawal]

Oh, it's already done.
it's going to be hard waiting for the release now.

[jukrb0x]

Oh, it's already done. it's going to be hard waiting for the release now.

Please close this ticket when releases, I would be happy to try it immediately as it releases. :)

[Tyriar]

@jukrb0x the webgl ligature support is done, I was going to keep this open until it's done in VS Code to do the remaining mostly clean up items:

• Set up some puppeteer tests that verify on all renderers (dom, canvas, webgl) that ligatures are working
• Slim down the dependencies - I need to do a once over to see the state here
• Review/action area/addon/ligatures

[NewtonChutney]

Oh, it's already done.
it's going to be hard waiting for the release now.

@jukrb0x the webgl ligature support is done, I was going to keep this open until it's done in VS Code to do the remaining mostly clean up items:

• Set up some puppeteer tests that verify on all renderers (dom, canvas, webgl) that ligatures are working
• Slim down the dependencies - I need to do a once over to see the state here
• Review/action area/addon/ligatures

Knock knock 😶👀
Was ligature support finally added? 🥲

[Tyriar]

@NewtonChutney latest update for vscode: microsoft/vscode#34103 (comment)

[jerch]

@Tyriar Not sure if this is related - tried to run gh CI with node 16|18 and it failed with some weird output error for the ligature addon, whilte it works with node 14.

(Link to the action:https://github.com/jerch/xterm-addon-image/actions/runs/5222277777/jobs/9427553798)

[Tyriar]

@jerch passes fine on main, it's on Azure DevOps but also node 18 🤷

[jerch]

@jerch passes fine on main, it's on Azure DevOps but also node 18 shrug

Hmm weird, it also worked locally with node 16 (have not tested node 18 here).