Support unicode-mathematics and handle all special characters

[alyssadelaine]

Special characters like greek letters can be copied and pasted into the editors, but we don't yet have a way to deal with these in the LaTeX serialization code

subtasks

• support unicode in the body of LaTeX doc and -> PDF pipeline
• support unicode-mathematics in LaTeX -> PDF pipeline
• enable unicode-mathematics in react-katex browser rendering

[guidorice]

There are a couple of different different places where special characters (unicode) is not being handled well.

1. the client side latex renderer, react-katex. Looks like it has an option for unicode in math expressions, it might just need to be enabled.
2. the whole latex -> pdf conversion is dropping unicode characters

[guidorice]

An update to clarify the scope of the issue. There is a long history of handling special characters in LaTeX I will read through that doc and get up to speed on best practices.

I am editing the issue title to add support unicode-mathematics and adjusting the subtask list above.

cc @olafveerman

[guidorice]

Goal: user should be able to enter either a LaTeX symbol, or a unicode character, and it should work either way, e.g.

\gamma \Gamma renders the same as

𝛾 Γ

cc @olafveerman

[guidorice]

In PR #252 I have added xelatex to produce pdfs instead of pdflatex. This is apparently the modern way to produce pdfs from latex, and support unicode characters. This does not solve any of the immediate unicode issues yet, but should set the stage for better unicode support across the board.

Some notes:

• inserting emojis is not a good test of unicode support anywhere in the atbd document or the pdf system. Not all fonts support all unicode symbols, especially emojis, which are being invented all the time. More important is math characters, accented/diacritic marks and other extended chars (non-ascii). Emoji is a "nice to have".

• unicode support needs to be matched with the latex pipeline, and the font being used, and also with the Maths packages being used.

• Therefore the ATBD.tex template needs to be updated with need to use \usepackage{unicode-math} and \setmathfont{something}, \setmainfont{something} and possibly other commands, according to the resources below:

Here are some good resources:

• XeLaTeX reference guide
• about XeTeX
• overview of pdftex vs xetex vs luatex
• math starter packs
• emoji using xelatex
• more emoji adventure

cc @danielfdsilva @olafveerman