Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Variable fonts support #1672

Merged
merged 29 commits into from
Jan 25, 2023
Merged

Variable fonts support #1672

merged 29 commits into from
Jan 25, 2023

Conversation

khaledhosny
Copy link
Contributor

Fixes #1182

This uses HarfBuzz subsetter to instanciate the font, saves it to a temporary file and passes it to libtextpdf for embedding into PDF file.

Fonts with CFF2 table are not yet supported and will result in an error.

shapers/harfbuzz.lua Outdated Show resolved Hide resolved
@alerque alerque added the enhancement Software improvement or feature request label Dec 26, 2022
@alerque alerque added this to the v0.x.y milestone Dec 26, 2022
@khaledhosny khaledhosny marked this pull request as ready for review December 26, 2022 22:18
@khaledhosny khaledhosny requested review from a team, alerque and simoncozens as code owners December 26, 2022 22:18
@khaledhosny
Copy link
Contributor Author

The test failures seem to be cause by outdated test expectations. How are these updated?

@khaledhosny
Copy link
Contributor Author

I think I’m done here, this should be ready for review/testing for real now.

@ctrlcctrlv
Copy link
Member

Got it. Do you expect color fonts will work? If so I'll give the FRBAC specimen build a spin and add the Cantarell TTF as the font used to show the labels.

@khaledhosny
Copy link
Contributor Author

Color fonts should work.

@ctrlcctrlv
Copy link
Member

I'm sorry to say that opsz variations don't seem to be working for me.

Dumping "dist/ttf/FRBAmericanPrint-900-GuidelinesDottedBlack.ttf" to "dist/ttf/FRBAmericanPrint-900-GuidelinesDottedBlack.ttx"...
Dumping 'GlyphOrder' table...
Dumping 'head' table...
Dumping 'hhea' table...
Dumping 'maxp' table...
Dumping 'OS/2' table...
Dumping 'hmtx' table...
Dumping 'cmap' table...
Dumping 'loca' table...
Dumping 'glyf' table...
Dumping 'name' table...
Dumping 'post' table...
Dumping 'COLR' table...
Dumping 'CPAL' table...
Dumping 'GDEF' table...
Dumping 'GPOS' table...
Dumping 'GSUB' table...
Dumping 'HVAR' table...
Dumping 'STAT' table...
Dumping 'XOLR' table...
Dumping 'fvar' table...
Dumping 'gvar' table...

Could it be a table check gone awry? This isn't a traditional variable font in that I only use FeatureVariations in it.

@ctrlcctrlv
Copy link
Member

Interesting, some of my fonts are causing crashes, but not all.

! Unexpected Lua error

stack traceback:
	/opt/share/sile/lua_modules/share/lua/5.4/pl/class.lua:20: in upvalue 'call_ctor'
	/opt/share/sile/lua_modules/share/lua/5.4/pl/class.lua:171: in field 'measurement'
	/opt/share/sile/core/font.lua:80: in function 'core.font._key'
	/opt/share/sile/core/font.lua:118: in function 'core.font.cache'
	/opt/share/sile/shapers/harfbuzz.lua:33: in function 'shapers.harfbuzz.shapeToken'
	/opt/share/sile/shapers/base.lua:94: in function 'shapers.base.createNnodes'
	/opt/share/sile/core/nodefactory.lua:250: in method 'shape'
	/opt/share/sile/typesetters/base.lua:302: in function 'typesetters.base.shapeAllNodes'
	/opt/share/sile/packages/bidi/init.lua:159: in function 'typesetters.base.boxUpNodes'
	/opt/share/sile/typesetters/base.lua:596: in function 'typesetters.base.leaveHmode'
	...	(skipping 210495 levels)
	/opt/share/sile/classes/plain.lua:98: in field '?'
	/opt/share/sile/core/sile.lua:391: in function 'core.sile.call'
	/opt/share/sile/core/sile.lua:236: in function 'core.sile.process'
	/opt/share/sile/core/font.lua:53: in field '?'
	/opt/share/sile/core/sile.lua:391: in function 'core.sile.call'
	/opt/share/sile/core/sile.lua:236: in function 'core.sile.process'
	(...tail calls...)
	/opt/share/sile/core/sile.lua:300: in function 'core.sile.processString'
	/opt/share/sile/core/sile.lua:332: in function 'core.sile.processFile'
	(...tail calls...)
	[C]: in function 'xpcall'
	/opt/bin/sile:125: in main chunk
	[C]: in ?

error summary:
	Processing at: /tmp/silespecimens/eachfontFRBAmericanCursive-1000-GuidelinesArrowsUltra.ttf.sil:20:1: in \bigskip
	Using code at: /opt/share/sile/lua_modules/share/lua/5.4/pl/class.lua:20: stack overflow

@ctrlcctrlv
Copy link
Member

Determined it's only the color ones that cause crashes.

@ctrlcctrlv
Copy link
Member

@khaledhosny I added a quick hack to my build script so I could see what happens when we don't crash.

for f in `find /tmp/silespecimens -iname '*.pdf'`; do
    PDFHEADERONLY='00000000: 2550 4446 2d31 2e35 0a25 e4f0 edf8 0a    %PDF-1.5.%.....'
    if [ "$(head -c 30 "$f" | xxd)" = "$PDFHEADERONLY" ]; then
        rm "$f"
    fi
done

It's possible I'm doing it wrong, but it's not working for me at all for some reason.

image

I expected that Cantarell to be at its heaviest weight.

Here's the .sil

\begin[papersize=9in x 3in]{document}
% Note: This file is intended to be compiled by scripts/each_font_specimen.sh which expects you to also have installed pdftk
\script[src=packages/complex-spaces]
\script[src=packages/color-fonts]
\script[src=packages/frametricks]
\script[src=packages/rebox]
\set[parameter=font.filename,value=/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf,makedefault=true]
\set[parameter=font.size,value=20pt]
\frame[id=specimen,left=5%pw,right=95%pw,top=5%ph,bottom=top(footnotes)]
\frame[id=specimen_overlay,left=5%pw,right=95%pw,top=5%ph,bottom=top(footnotes)]
\frame[id=arrows,left=5%pw,right=95%pw,top=5.22%ph,bottom=top(footnotes)]
\set[parameter=shaper.spaceenlargementfactor,value=1]
\frame[id=fontname,left=5%pw,right=95%pw,top=1%ph,bottom=3% + height(folio)]
\typeset-into[frame=fontname]{
\noindent\font[size=5pt,variations=wght=1000]{FRBAmericanCursive-1000-DashedUltra.ttf}
}
\begin[filename=dist/ttf/FRBAmericanCursive-1000-DashedUltra.ttf]{font}
\par \font[size=48pt]{FRB American Cursive}
\bigskip
\par FRB American Cursive Dashed Ultra
\bigskip
\par Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz
\par ÁáÀàẢảÃãẠạĂăẮắẰằẲẳẴẵẶặÂâẤấẦầẨẩẪẫẬậĐđÉéÈèẺẻẼẽẸẹÊêẾếỀềỂểỄễỆệÍíÌìỈỉĨĩỊịÓóÒòỎỏÕõỌọ
\par ÔôỐốỒồỔổỖỗỘộƠơỚớỜờỞởỠỡỢợÚúÙùỦủŨũỤụƯưỨứỪừỬửỮữỰựÝýỲỳỶỷỸỹỴỵ
\par abcdefghijklmnopqrstuvwxyz
\par 1234567890 !@#$\%&*() []{};':",./ \\ № ₿ ₱ ₹ †
\par The quick brown fox jumps over the lazy dog.
\par Ekvidu! Laŭ Zamenhof bongustas Ĉeĥuje manĝaĵo freŝa spica.
\par В чащах юга жил бы цитрус? Да, но фальшивый экземпляр!
\par Şaban Türk'tür. Bir şıpka torpido yakalar.
\end{font}
\end{document}

@khaledhosny
Copy link
Contributor Author

I expected that Cantarell to be at its heaviest weight.

It works for me. Can you pull the last commit and run with sile --debug=fonts --debug=versions and show me the output?

@khaledhosny
Copy link
Contributor Author

Determined it's only the color ones that cause crashes.

Please send me and fonts and documents that are failing.

@ctrlcctrlv
Copy link
Member

Cantarell.zip is in #1182.

Here's an example of a successful invocation.

Shapers enabled: ot, fallback
ICU support enabled

Fonts used:
dist/ttf/FRBAmericanCursive-400-Regular.ttf:0	Version 2.0;Modular Font Editor K font №1
/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf:0	Version 0.303
/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf:0	Version 0.303
dist/ttf/FRBAmericanCursive-400-Regular.ttf:0	Version 2.0;Modular Font Editor K font №1
SILE v0.14.6.r20-g605aed8-dirty (Lua 5.4)
</tmp/silespecimens/eachfontFRBAmericanCursive-400-Regular.ttf.sil> as sil

[fonts] Looking for ;5;400;;normal;;'wght=1000';LTR;/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Resolved font family  -> /home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Looking for ;10;400;;normal;;;LTR;dist/ttf/FRBAmericanCursive-400-Regular.ttf
[fonts] Resolved font family  -> dist/ttf/FRBAmericanCursive-400-Regular.ttf
[fonts] Looking for ;48;400;;normal;;;LTR;dist/ttf/FRBAmericanCursive-400-Regular.ttf
[fonts] Resolved font family  -> dist/ttf/FRBAmericanCursive-400-Regular.ttf[1] 
[fonts] Looking for ;10;400;;normal;;;LTR;/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Resolved font family  -> /home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf

And here's a failed one:

99% 104:1=0s dist/ttf/FRBAmericanCursive-1000-Ultra.ttf                                                                                                       SILE v0.14.6.r20-g605aed8-dirty (Lua 5.4)
</tmp/silespecimens/eachfontFRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf.sil> as sil

[fonts] Looking for ;5;400;;normal;;'wght=1000';LTR;/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Resolved font family  -> /home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Looking for ;10;400;;normal;;;LTR;dist/ttf/FRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf
[fonts] Resolved font family  -> dist/ttf/FRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf
[fonts] Looking for ;48;400;;normal;;;LTR;dist/ttf/FRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf
[fonts] Resolved font family  -> dist/ttf/FRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf
! Unexpected Lua error

stack traceback:
	/opt/share/sile/lua_modules/share/lua/5.4/pl/class.lua:20: in upvalue 'call_ctor'
	/opt/share/sile/lua_modules/share/lua/5.4/pl/class.lua:171: in field 'measurement'
	/opt/share/sile/core/font.lua:80: in function 'core.font._key'
	/opt/share/sile/core/font.lua:118: in function 'core.font.cache'
	/opt/share/sile/shapers/harfbuzz.lua:33: in function 'shapers.harfbuzz.shapeToken'
	/opt/share/sile/shapers/base.lua:94: in function 'shapers.harfbuzz.createNnodes'
	/opt/share/sile/core/nodefactory.lua:250: in method 'shape'
	/opt/share/sile/typesetters/base.lua:302: in function 'typesetters.base.shapeAllNodes'
	/opt/share/sile/packages/bidi/init.lua:159: in function 'typesetters.base.boxUpNodes'
	/opt/share/sile/typesetters/base.lua:596: in function 'typesetters.base.leaveHmode'
	...	(skipping 210495 levels)
	/opt/share/sile/classes/plain.lua:98: in field '?'
	/opt/share/sile/core/sile.lua:391: in function 'core.sile.call'
	/opt/share/sile/core/sile.lua:236: in function 'core.sile.process'
	/opt/share/sile/core/font.lua:53: in field '?'
	/opt/share/sile/core/sile.lua:391: in function 'core.sile.call'
	/opt/share/sile/core/sile.lua:236: in function 'core.sile.process'
	(...tail calls...)
	/opt/share/sile/core/sile.lua:300: in function 'core.sile.processString'
	/opt/share/sile/core/sile.lua:332: in function 'core.sile.processFile'
	(...tail calls...)
	[C]: in function 'xpcall'
	/opt/bin/sile:125: in main chunk
	[C]: in ?

error summary:
	Processing at: /tmp/silespecimens/eachfontFRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf.sil:21:1: in \bigskip
	Using code at: /opt/share/sile/lua_modules/share/lua/5.4/pl/class.lua:20: stack overflow

And here's the font that failed w/the document above:
FRBAC_EXAMPLE_FONT_OT_LICENSE_INVALID_DO_NOT_REDISTRIBUTE.tar.gz

@ctrlcctrlv
Copy link
Member

(Note that as mentioned I'm not seeing Cantarell at the weight I expected even when the invocation of sile does not crash.)

image

eachfontFRBAmericanCursive-400-Regular.ttf.pdf

@khaledhosny
Copy link
Contributor Author

Cantarell.zip is in #1182.

Here's an example of a successful invocation.

Shapers enabled: ot, fallback
ICU support enabled

Fonts used:
dist/ttf/FRBAmericanCursive-400-Regular.ttf:0	Version 2.0;Modular Font Editor K font №1
/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf:0	Version 0.303
/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf:0	Version 0.303
dist/ttf/FRBAmericanCursive-400-Regular.ttf:0	Version 2.0;Modular Font Editor K font №1
SILE v0.14.6.r20-g605aed8-dirty (Lua 5.4)
</tmp/silespecimens/eachfontFRBAmericanCursive-400-Regular.ttf.sil> as sil

[fonts] Looking for ;5;400;;normal;;'wght=1000';LTR;/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Resolved font family  -> /home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Looking for ;10;400;;normal;;;LTR;dist/ttf/FRBAmericanCursive-400-Regular.ttf
[fonts] Resolved font family  -> dist/ttf/FRBAmericanCursive-400-Regular.ttf
[fonts] Looking for ;48;400;;normal;;;LTR;dist/ttf/FRBAmericanCursive-400-Regular.ttf
[fonts] Resolved font family  -> dist/ttf/FRBAmericanCursive-400-Regular.ttf[1] 
[fonts] Looking for ;10;400;;normal;;;LTR;/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Resolved font family  -> /home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf

I was expecting to see the HarfBuzz version here, and some output about instanciation, but I see neither. You need harfbuzz-subset >= 6.0.0 for instanciation to happen.

And here's a failed one:

99% 104:1=0s dist/ttf/FRBAmericanCursive-1000-Ultra.ttf                                                                                                       SILE v0.14.6.r20-g605aed8-dirty (Lua 5.4)
</tmp/silespecimens/eachfontFRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf.sil> as sil

[fonts] Looking for ;5;400;;normal;;'wght=1000';LTR;/home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Resolved font family  -> /home/fred/Workspace/cantarell-fonts/src/variable_ttf/Cantarell-VF.ttf
[fonts] Looking for ;10;400;;normal;;;LTR;dist/ttf/FRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf
[fonts] Resolved font family  -> dist/ttf/FRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf
[fonts] Looking for ;48;400;;normal;;;LTR;dist/ttf/FRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf
[fonts] Resolved font family  -> dist/ttf/FRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf
! Unexpected Lua error

stack traceback:
	/opt/share/sile/lua_modules/share/lua/5.4/pl/class.lua:20: in upvalue 'call_ctor'
	/opt/share/sile/lua_modules/share/lua/5.4/pl/class.lua:171: in field 'measurement'
	/opt/share/sile/core/font.lua:80: in function 'core.font._key'
	/opt/share/sile/core/font.lua:118: in function 'core.font.cache'
	/opt/share/sile/shapers/harfbuzz.lua:33: in function 'shapers.harfbuzz.shapeToken'
	/opt/share/sile/shapers/base.lua:94: in function 'shapers.harfbuzz.createNnodes'
	/opt/share/sile/core/nodefactory.lua:250: in method 'shape'
	/opt/share/sile/typesetters/base.lua:302: in function 'typesetters.base.shapeAllNodes'
	/opt/share/sile/packages/bidi/init.lua:159: in function 'typesetters.base.boxUpNodes'
	/opt/share/sile/typesetters/base.lua:596: in function 'typesetters.base.leaveHmode'
	...	(skipping 210495 levels)
	/opt/share/sile/classes/plain.lua:98: in field '?'
	/opt/share/sile/core/sile.lua:391: in function 'core.sile.call'
	/opt/share/sile/core/sile.lua:236: in function 'core.sile.process'
	/opt/share/sile/core/font.lua:53: in field '?'
	/opt/share/sile/core/sile.lua:391: in function 'core.sile.call'
	/opt/share/sile/core/sile.lua:236: in function 'core.sile.process'
	(...tail calls...)
	/opt/share/sile/core/sile.lua:300: in function 'core.sile.processString'
	/opt/share/sile/core/sile.lua:332: in function 'core.sile.processFile'
	(...tail calls...)
	[C]: in function 'xpcall'
	/opt/bin/sile:125: in main chunk
	[C]: in ?

error summary:
	Processing at: /tmp/silespecimens/eachfontFRBAmericanCursive-1000-GuidelinesArrowsDottedUltra.ttf.sil:21:1: in \bigskip
	Using code at: /opt/share/sile/lua_modules/share/lua/5.4/pl/class.lua:20: stack overflow

I can reproduce this. Investigating, though since you don’t seem to have any instanciation going on, I’m suspecting it is an unrelated issue.

@khaledhosny
Copy link
Contributor Author

I can reproduce this. Investigating, though since you don’t seem to have any instanciation going on, I’m suspecting it is an unrelated issue.

I can’t run using this font with SILE v0.14.5 either. Looks like a bug in the color-fonts package (I would have replaced this code with the better maintained HarfBuzz API (which now supports COLRv1, too), but it when I suggested using HarfBuzz API to read MATH table others didn’t like it).

@khaledhosny
Copy link
Contributor Author

If you add --debug=color-fonts you can see it is getting into some kind of infinite loop before exhausting lua stack.

@khaledhosny
Copy link
Contributor Author

The infinite loop is caused by leaveHmode in, but I don’t understand this code to debug further:

sile/core/font.lua

Lines 130 to 134 in 771d87f

if SILE.shaper._name ~= "harfbuzzWithColor" then
SU.debug("color-fonts", "Switching to color font Shaper")
SILE.typesetter:leaveHmode(true)
lastshaper, SILE.shaper = SILE.shaper, SILE.shapers.harfbuzzWithColor()
end

Also, conceptually, color support should be handled by the outputters not the shapers (I know, I wrote the initial color-fonts package, guilty as charged).

@alerque
Copy link
Member

alerque commented Dec 28, 2022

An issue with looping after using color fonts sounds like the modified shaper object not getting relpaced with the original when done. I'll see if I can replicate that, but I ran into something like that recently with the fallback font stack so it sounds familiar.

@alerque alerque modified the milestones: v0.x.y, v0.14.8 Dec 28, 2022
khaledhosny and others added 25 commits January 25, 2023 16:07
Creating HarfBuzz fonts was all over the place, done in a slightly
different way each time and with potential memory leaks (e.g. often
destroying the font, but not the face or the blob).

It was also reading the whole font data in Lua, then passing it to
HarfBuzz, which is inefficient—especially for large fonts. The code is
now using hb_blob_create_from_file() instead which will mmap the file
whenever it can.

The created hb_font_t is now cached in the Lua font and retrieved from
there when needed.
Each call hb_font_set_variations() resets all font variations, so we
need to combine all variation settings and apply them at once.
FontConfig will use the upper bits of the face index for named instance
index, and hb_font_create() handles this, but we were stripping these
bits prematurely. Strip them before sending the index to libtextpdf
instead.

With named instances supported, saying \font[name=Foo, weight=700] of a
variable font will set the correct variation coordinates for the named
instance, instead of always using the default instance.
Extend the hack used for finding face index of TTC files to find named
instance index of variable fonts. Switch the code to use HarfBuzz all
the way to reduce duplication.
With the last commits, FreeType is no longer directly used. It is still
needed by FontConfig making it a transitive dependency.

This also paves the way to allow building on macOS without FontConfig as
well, and maybe provide a DirectWrite-based Windows font manager and
build without FontConfig there, too.
* Build HarfBuzz 6.0.0 and harfbuzz-subset library.

* Don’t build FreeType with HarfBuzz support.

    FreeType uses HarfBuzz in the autohinter, but the autohinter is not
    used since FreeType is only used by FontConfig to load the fonts and
    it is not used for any rasterization.

* Don’t build HarfBuzz with ICU support, SILE does not use it.

* Build FreeType from upstream repository.

* Update to the latest release tag and use upstream repository.
This makes variable font support the default, but comes at the cost of
requiring the supper-new HarfBuzz 6. Folks can explicitly disable this
for older HB support but they should be aware they are limiting
features, hence the --disable-font-variations flag.
If we can’t instanciate the glyph outlines, there is no point in
applying variations during layout.
- Force `fontconfig` fontmanger as the test depends on `FONTCONFIG_FILE`.
- Correctly call `getFace()` to actually load the Libertinus Serif font
  not the default one.
- Check the name string not its length which might accidentally pass
  when loading a different font.
Copy link
Member

@alerque alerque left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay getting this merged and released. The Nix stuff I was waiting for happened in a couple days but my kids have been sick. For my youngest daughter with TSC that means more seizures and I haven't gotten much work in the past few weeks.

I think we're good to go here. It is easy to only run supported tests but I couldn't come up with a mechanism for only running tests not supported by other runs that I liked, so I just hard coded the HB6 specific tests for now. Eventually we'll probably run the whole test suite there anyway, but for now I just wanted to get that one running.

@alerque alerque merged commit b2970ee into sile-typesetter:master Jan 25, 2023
@khaledhosny khaledhosny deleted the issue-1182 branch January 25, 2023 15:16
@khaledhosny
Copy link
Contributor Author

khaledhosny commented Jan 25, 2023

Thanks @alerque for getting this across the finish line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Software improvement or feature request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Variable font support
3 participants