Use a custom docker image for tests

[BoboTiG]

For each PR tests job, most of the time is taken by LateX installation. For instance, it takes about 2m40s to install it against 30s to run tests.

Maybe should we investigate the creation of a custom Docker image with LaTeX preinstalled. If so, I would be in favor of using a Debian-based light distribution, but I am open to any distribution as soon as tests are passing as-is (e.g: no modifications to be done on the source code).

[lasconic]

We could try to cache: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows

• /usr/bin
• /usr/share/texlive/
• /var/lib/tex-common/

With a custom docker image, where is Python running ? In the image or in the host ?

[BoboTiG]

We could try to cache: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows

That would be the easiest solution, we can give a try.

With a custom docker image, where is Python running ? In the image or in the host ?

Good question. I didn't think about it, maybe the setup-python action works out of the box with docker 🤔

[BoboTiG]

Another idead: an even simpler solution would be to cache the APT cache.

[BoboTiG]

I'll give a try to https://github.com/awalsh128/cache-apt-pkgs-action.

[lasconic]

We could also check if we really need the full texlive-fonts-extra ... It's 300MB and I guess it takes quite some time to install all the fonts.

[BoboTiG]

Well, texlive-fonts-extra is needed finally: https://github.com/BoboTiG/ebook-reader-dict/actions/runs/3472221189/jobs/5802756706

[lasconic]

Probably just a part of it, but which one ?

[BoboTiG]

Good question.

BTW the package is 507 Mb. It lists a lot of fonts. If latex could tell us which font is missing then it would be easier :)

[BoboTiG]

And selecting specific fonts may be problematic. For instance, AFAIK it's not possible to just install adfsymbols or xits fonts.

[lasconic]

Maybe it can ? #1096 (comment)

[BoboTiG]

Do you are aware of any webservice we could leverage? Like calling an URL with the formula as query. It would be awesome.

[BoboTiG]

I'll dig into https://en.wikipedia.org/api/rest_v1/#/Math.

[lasconic]

Wow it looks promising.

[BoboTiG]

I got something working, that's great.

I would go with SVG directly, rather than going from PNG > GIF. Do you have any concern?

[lasconic]

None but the ones already highlighted (compatibility with old devices, other formats)

[BoboTiG]

Arf, I do not like that SVG discussion, it seems we will not make progress anytime soon.

Anyway, here is a working POC:

from base64 import b64encode
from io import BytesIO

import requests
from PIL import Image

HEADERS = {"User-Agent": "https://github.com/BoboTiG/ebook-reader-dict"}
URL_BASE = "https://en.wikipedia.org/api/rest_v1"
URL_MATH_HASH = f"{URL_BASE}/media/math/check/{{type}}"
URL_MATH_RENDER = f"{URL_BASE}/media/math/render/{{format}}/{{hash}}"


def render_formula(
    formula: str, cat: str = "inline-tex", output_format: str = "png"
) -> str:
    """
    Details on https://en.wikipedia.org/api/rest_v1/#/Math
    """
    # 1. Get the formula hash (type can be tex, inline-tx, or chem)
    url_hash = URL_MATH_HASH.format(type=cat)
    with requests.post(url_hash, headers=HEADERS, json={"q": formula}) as req:
        res = req.json()
        assert res["success"]
        formula_hash = req.headers["x-resource-location"]

    # 2. Get the rendered formula (format can be svg, mml, or png)
    url_render = URL_MATH_RENDER.format(format=output_format, hash=formula_hash)
    with requests.get(url_render, headers=HEADERS) as req:
        raw_png = req.content

    # 3. Convert the PNG to GIF
    with BytesIO(raw_png) as buf, BytesIO() as im:
        Image.open(buf).save(im, format="gif", optimize=True)
        im.seek(0)
        raw_gif = im.read()

    return b64encode(raw_gif).decode()


formula = "V^n"
gif = render_formula(formula)

It's quite clean. Note the user agent, it needs to be set to something where Wikimedia can contact us, so using the repos seems correct.

I need to test a <chem> formula, and if results are good I'll open a PR. I may finish it by tomorrow.

[BoboTiG]

At the end, GIF is still an interesting format given its size:

[lasconic]

If the output is clearer on Kobo, we have the SVG, let's use it. If we get complains with other format, let's find a solution to export something else for them ?

[lasconic]

I believe we still had some errors with Math formula, I guess they will be sorted out.

[BoboTiG]

If the output is clearer on Kobo, we have the SVG, let's use it. If we get complains with other format, let's find a solution to export something else for them ?

Agreed.

[lasconic]

In english:

<math> ERROR with \begin{align}a_0 &+ a_1x + a_2x^2 + a_3x^3 + \cdots + a_nx^n \\ &= a_0 + x \bigg(a_1 + x \Big(a_2 + x \big(a_3 + \cdots + x(a_{n-1} + x \, a_n) \cdots \big) \Big) \bigg).\end{align} in [Horner's rule]
<math> ERROR with \begin{align}\frac{\pi}{2} & = \prod_{n=1}^{\infty} \frac{ 4n^2 }{ 4n^2 - 1 } = \prod_{n=1}^{\infty} \left(\frac{2n}{2n-1} \cdot \frac{2n}{2n+1}\right) \\[6pt]& = \Big(\frac{2}{1} \cdot \frac{2}{3}\Big) \cdot \Big(\frac{4}{3} \cdot \frac{4}{5}\Big) \cdot \Big(\frac{6}{5} \cdot \frac{6}{7}\Big) \cdot \Big(\frac{8}{7} \cdot \frac{8}{9}\Big) \cdot \; \cdots \\\end{align} in [Wallis product]
<chem> ERROR with ^-N=\overset{+}N=N^- in [azide]
<math> ERROR with \frac = \frac in [circle of Apollonius]
<math> ERROR with \begin{align}\rho(g, h) (0,x_1,\ldots,x_k) &= g(x_1,\ldots,x_k) \\\rho(g, h) (y+1,x_1,\ldots,x_k) &= h(y,\rho(g, h) (y,x_1,\ldots,x_k),x_1,\ldots,x_k)\,\end{align} in [primitive recursion]

[BoboTiG]

Yup, those cases were tested with success. Actually I'm fighting with the display size/CSS. The rendering itself is well done ,but the display on the Kobo is not.

I will open a PR to gather some help.