JS bindings for color needed

[Lorp]

I built a version of hb.wasm to try on COLRv1 fonts. (I used the supplied build.sh script, but commented out #define HB_NO_COLOR in hb-config.hh)

The new hb.wasm works, but emits monochrome glyphs as before. I guess we need some JS bindings that will allow me to use color. When I came across a related problem for variable fonts (#34), @ebraminio was super helpful. Are you there for me again by any chance? :)

@behdad suggests @khaledhosny may also be able to fix the JS side. Any help appreciated.

[behdad]

I built a version of hb.wasm to try on COLRv1 fonts. (I used the supplied build.sh script, but commented out #define HB_NO_COLOR in hb-config.hh)

You should modify config-override.h, and undef HB_NO_COLOR as well as HB_NO_PAINT.

The new hb.wasm works, but emits monochrome glyphs as before.

Indeed.

I guess we need some JS bindings that will allow me to use color. When I came across a related problem for variable fonts (#34), @ebraminio was super helpful. Are you there for me again by any chance? :)

@behdad suggests @khaledhosny may also be able to fix the JS side. Any help appreciated.

Yes, we need to bind the hb-paint API, then there needs to be an implementation of it on the JS side for HTML5 canvas. It's a bit of work but easy to port from hb-cairo. For bindings would be great to get help from Khaled or Ebrahim indeed. But you should also be able to do it yourself if you look at how the hb-draw binding is done.

[chearon]

Yes, we need to bind the hb-paint API, then there needs to be an implementation of it on the JS side for HTML5 canvas. It's a bit of work but easy to port from hb-cairo

I got painting to a canvas context working here: chearon@7676278

Since it allows harfbuzz to make calls into JS, it raises the question "should SVG creation in hbjs.cc be ported to JS too?" but I was unsure if I should make such a large change. Either way, we could add painting color glyphs onto that.

[behdad]

I got painting to a canvas context working here: chearon@7676278

That's great! Is that upstream yet?

Since it allows harfbuzz to make calls into JS, it raises the question "should SVG creation in hbjs.cc be ported to JS too?"

I believe so. The SVG was a hack IMO to work around not binding the draw API.

but I was unsure if I should make such a large change.

Either way sounds fine to me.

Either way, we could add painting color glyphs onto that.

Yeah, that would be great. Then we can port the hb-cairo painting code to a canvas-based implementation.

[chearon]

That's great! Is that upstream yet?

Not yet, but I'll make a PR soon. I've got several patches I'm using for a JS text stack that I just need to touch up.

Okay, I'll move the SVG stuff into JS. That'll be nice!

Then we can port the hb-cairo painting code to a canvas-based implementation.

I just read the 7.0.0 release notes and it sounds like we should enable the hb-paint API (currently harfbuzzjs uses hb-draw to render to SVG), use that, and then for the emoji part, hand write JS that's based on the hb-cairo glyph drawing functions?

---

It's interesting that the hb-draw and hb-paint APIs draw directly to a context. I always assumed FreeType was the "right" way to draw glyphs because it cached the bitmaps or something. Is that not true? In my JS text layout project, I only draw contours manually if someone does something like color a diacritic. Whenever I can, I use ctx.fillText(string) because it is so much faster than moveTo, quadraticCurveTo, etc. Maybe those are only slow because of the JS->native overhead.

[Lorp]

This is really promising!

In parallel I’ve been generalizing the drawing instructions produced by the Samsa library, removing SVG-specific output, and expecting more from the consumer application to build the SVG (or canvas). This demo page shows Samsa-generated instructions being used for SVG and canvas, presented next to each other (and also native and harfbuzz rendering):

https://www.axis-praxis.org/samsa/ap3/ap3.html

Using the same approach to deal with COLRv1 will result in JS similar to what is needed for the JS bindings for Harfbuzz, whether I generate SVG or canvas. Note that the SVG solution will not handle radial or conic gradients, but canvas will.

BTW the harfbuzz wasm used in the demo page above is built with #undef HB_NO_BORING_EXPANSION in config-override.h so if you choose Roboto-Flex-avar2 and adjust wght, wdth or opsz, you see avar2 working in all 4 renderings 😀

[khaledhosny]

It's interesting that the hb-draw and hb-paint APIs draw directly to a context. I always assumed FreeType was the "right" way to draw glyphs because it cached the bitmaps or something. Is that not true? In my JS text layout project, I only draw contours manually if someone does something like color a diacritic. Whenever I can, I use ctx.fillText(string) because it is so much faster than moveTo, quadraticCurveTo, etc. Maybe those are only slow because of the JS->native overhead.

HarfBuzz does not currently have a rasterizer, so the draw API is akin to FT_Outline_Decompose, and the client does the rasterization. With paint API HarfBuzz will tell you to draw a glyph and you can draw it whatever you like, you don’t have to use the draw API to draw it, but using draw API has its advantages since it handles variable font instanciation, and newer font technology additions that HarfBuzz is working on.

As for ctx.fillText(), it takes text strings, but HarfBuzz output is glyph indices, so it can’t be used with HarfBuzz.

There is a hack, however, that Mozilla’s pdf.js does to work around this: before setting the font on the canvas, modify it by removing all cmap subtables, then adding a cmap subtable that maps all glyphs in the font to PUA, then when calling ctx.fillText(), convert the glyph ID to the corresponding PUA character. You still have to draw glyphs one by one since you will be doing the positioning yourself (or you have to check if shaped glyph advances match font ones and there is no x and y offsets). Sounds like too much work, but might be worth it if the performance difference is significant.

[behdad]

It's interesting that the hb-draw and hb-paint APIs draw directly to a context. I always assumed FreeType was the "right" way to draw glyphs because it cached the bitmaps or something. Is that not true?

FreeType does have a caching subsystem, but that is rarely used. It's still up to the FreeType client to cache the bitmaps. With HarfBuzz, the rasterization is also offloaded to the client, which would then need to cache the canvas rasters in an atlas to do fast blitting...

Or the hack Khaled mentioned, if platform-level speed and rasterization is desired but with HarfBuzzJS shaping.

[chearon]

@khaledhosny

As for ctx.fillText(), it takes text strings, but HarfBuzz output is glyph indices, so it can’t be used with HarfBuzz.

As a consumer of harfbuzzjs you can shape and wrap a paragraph and then have fast painting to a canvas if you find safe boundaries to call fillText.

For example if you have passe<span style="color: red;"> ́</span> it's safe to fillText("pass") followed by hbFont.drawGlyph(ctx, /* e */) and hbFont.drawGlyph(ctx, /* ́ */). I've also tested this with coloring the middle of an Arabic word, which leverages UNSAFE_TO_BREAK boundaries rather than grapheme boundaries.

It's still a hack since you're assuming the same shaping results as native. Maybe one day we can have shaping and glyph drawing APIs in the browser.

There is a hack, however, that Mozilla’s pdf.js does to work around this

That is very interesting, thank you! Since I found quadraticCurveTo to be slower than fillText, I'm not sure if one fillText per glyph would be faster or not, but if pdf.js is using it it's worth testing, and this would not have the assumption I mentioned above.

@behdad

FreeType does have a caching subsystem, but that is rarely used. It's still up to the FreeType client to cache the bitmaps.

Ok, thanks for the answer. cairo_show_glyphs was about 3x faster than cairo_curve_to/etc, but when I tested that, I was doing it through node-canvas so maybe the former reducing calls from JS->native is what did it.

It would be interesting to try the bitmap caching in a canvas, but since those have to be colored upon drawing, not sure if that negates it. Sounds like I have some profiling to do...

[khaledhosny]

I managed to re-implment the glyphToPath in JS here #97, it should serve as an example on how to do callbacks. I’m not very well versed in JS or Emscripten, so if actual users of the library can look into this and let me know if it works for them. We probably should do the same to shapeWithTrace and drop hbjs.cc entirely.