Fix part of #4044: Add KotliTeX integration (direct LaTeX rendering)

[BenHenning]

Explanation

Fix part of #4044

This PR introduces support for rendering raw LaTeX in the app using the KotliTeX (https://github.com/karino2/kotlitex) library which is necessary since users may input arbitrary math expressions that need to be pretty-rendered (using the math expression -> LaTeX generation support added in #4054).

This implementation is heavily based on #3194 and leverages a new fork (https://github.com/oppia/kotlitex) of KotliTeX based on @anandwana001's fork.

In order to keep the implementation simpler, this hangs the functionality onto the existing custom Oppia noninteractive math tag except it only renders LaTeX if there's no SVG filename to load (which will be the case for dynamically generated LaTeX). This approach has the advantage of the app always being able to fall back to LaTeX rendering in the unlikely scenario where content has a math tag without an SVG URL being present (or where the SVG fails to load which could allow potential fault tolerance when downloads support is fully implemented). These changes required several changes to KotliTeX itself which are outlined in more detail below.

Furthermore, it was observed that the rendering was noticeably slow on a Nexus 5X for repeated renders (think navigation back and forth between views with rendered LaTeX, or expanding/collapsing multiple answers). This didn't seem acceptable, so this PR also introduces a platform parameter for utilizing Glide to pre-render the LaTeX into a PNG bitmap so that it can cache it. This is not only more performant for repeated exposure and faster to render, but it also offloads the expensive rendering step to a background thread rather than blocking the main thread (see the videos in the UI section below for a before-and-after comparison).

Details on changes to KotliTeX

A custom fork of KotliTeX (https://github.com/oppia/kotlitex) was needed to include the following changes:

• Support to be built on Bazel
• Lower the min sdk required by the library to match the app's
• A slight hack to ensure KotliTeX doesn't auto-fail for larger square roots (the library actually doesn't support them, so this workaround just stretches the smaller square root rather than leading to error text)
• A hack to expose the outer bounds of drawing requires by KotliTeX for its span (see caching details below for why)

Note that KotliTeX only supports a subset of LaTeX, but it seems to support everything the app needs for now (and it should be extensible by porting other portions of KaTeX as needed).

Details on the rendering caching & positioning challenges

At a high-level, the approach to caching is to render the KotliTeX drawable to a bitmap that can be saved on-disk in a format that Glide can easily load on its own (PNG). While it's not ideal to perform the extra PNG conversion step, it's much more disk-friendly (and in the future, we could look into offloading just this part to another thread if needed). Each individual load is keyed on the exact raw LaTeX, the line height (up to 2 digits), and whether it's inline or block rendered (see later in this section for the distinction). If all three of these properties match, there should be a Glide cache hit and no need to re-render the LaTeX.

A couple of issues arose when trying to implement this:

• Despite us being able to expose the internal KotliTeX drawable (which wasn't by default), I was having a lot of difficulty getting the text alignment correct. Instead, it was much easier to rely on Android's internal text rendering by just rendering the span in a similar way to what TextView does.
• While the above worked, it led to another problem: figuring out how large the bitmap should be. The actual drawable bounds that KotliTeX computes is an underestimate (its ascent and descent go outside of these bounds), so an additional estimation is needed.
• Because we're now drawing the LaTeX as a bitmap in an ImageSpan, the text aligns the image either with its bottom line or its baseline. This introduces a vertical center alignment issue as shown below (the image with the red text shows one attempt at fixing this by eliminating unnecessary upper space by using a combination of vertical translation and negative bounds). Due to this quirk, we can't actually have whitespace below the text without modifying the text's font metrics (specifically the bottom/top/ascent/descent properties). Fix vertical alignment for locally rendered and cached LaTeX #4170 is tracking this work. Note also that this is really only an issue for inline rendering which we're unlikely to use in the medium-term since all user-submitted answers will be rendered in block style which seems to work well.

Caching is demonstrably much more performant for large amounts of repetition, and it doesn't result in UI lag like direct rendering does since all rendering is offloaded to a background thread (see the videos below for the comparison).

Details on testing

The tests are mainly focused around changes to the tag handler since the Glide portions can't be tested, and neither can the rendering routines (hence the exemptions). The fake image loader was updated to include support for the math-based rendering load requests.

An aside change was made to make the Robolectric library test-only (it's something that was noticed in passing during development, and isn't directly needed by this PR; it's just a nice-to-have).

The new interfaces are exempted from tests since they have no logic to verify (their implementations are generated by Dagger and verified at compile time). Furthermore, I didn't add any tests for the changes to the compute affected tests script since it's a bit difficult to test, and already quite an edge case. Please let me know if you have any concerns with this.

Details on the race condition fix

KotliTeX seems to initialized shared state when its parsing the LaTeX, and this with multiple Glide threads can cause a race condition where the LaTeX fails to render. This has been fixed by leveraging coroutines (where we still try to maximize efficiency and parallelization), but it required exposing new injector pathways for the coroutine dispatchers and ConsoleLogger.

Details on the ComputeAffectedTests fix

This PR exposes an issue with the ComputeAffectedTests script wherein it can sometimes exceed the system's maximum argument length. This is being worked around by partitioning values for longer Bazel queries into multiple calls and concatenating the results. I'm fairly certain this is functionally equivalent to the current implementation, and for the vast majority of future PRs only one partition will ever be used. There doesn't seem to be an alternative possible since this is a platform-specific Kernel restriction.

Essential Checklist

• The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".)
• Any changes to scripts/assets files have their rationale included in the PR explanation.
• The PR follows the style guide.
• The PR does not contain any unnecessary code changes from Android Studio (reference).
• The PR is made from a branch that's not called "develop" and is up-to-date with "develop".
• The PR is assigned to the appropriate reviewers (reference).

For UI-specific PRs only

For the most part, this PR doesn't actually affect users yet since this functionality isn't exposed (all LaTeX currently rendered in the app will utilize the existing SVG pipeline). The user-facing UI changes will be thoroughly documented in the PR that's planned for after this one since it'll be integrating all prior PRs along with the changes introduced here.

That being said, the "What is a Fraction?" test revision card has been updated to include rendered LaTeX text in order to demonstrate the functionality more closely. The following is how rendering looks (with bitmap caching):

Note that the above, when compared with non-cached rendering, demonstrates where some of the spacing issues mentioned above come in:

Here's the image showing the bounds when trying to force the drawable downward:

This video shows how long a bunch of LaTeX expressions takes to render with direct rendering:

slow_rendering.mp4

Versus the performance of caching for the same experience (note this occurred with a completely clean local app cache):

fast_cached_rendering.mp4

Notes on rendering approach

Note that rendering during cache time involves a three step process:

1. Approximate the outer bounding volume by tracking all draw calls needed for a given LaTeX expression (these bounds are used to create the bitmap that will store the cacheable render)
2. Render the LaTeX to a canvas with dimensions larger than the approximated bounding (since it's possible for the rendering to exceed these estimates--see notes below)
3. Crop the bitmap to the smallest bounds needed to encapsulate all of the pixels

(1) is needed because KotliTeX actually incorrect computes bounds (it sometimes over and underestimates). This isn't as apparent when directly rendering because Android will handle text expanding past the space KotliTeX expects since it's rendered in-line. I spent some time trying to debug this within KotliTeX but I couldn't make much headway. It may be worth looking into this in the future to see if we can reliably calculate the bounds within KotliTeX (since it will likely be much more performant than the solution used here). Seeing KotliTeX's debug lines makes this clearer (where it thinks the bounds are relative to glyphs; note that this screenshot is using rendering without caching):

(2) is needed because the methodology used for (1) is inexact. It's actually really difficult to estimate exactly where the final glyph pixels will be rendered for a given glyph since KotliTeX is directly handling positioning (rather than allowing Android to compute x/y coordinates based on nearby characters to account for tracking, kerning, etc.). The dimension approximation for (1) is much closer to correct than KotliTeX's internal bounds calculations, however it tends to slightly underapproximate the bounds. To counteract this, the app uses a bitmap that's 2x what's actually estimated to be needed and offsets rendering into that bitmap such that 50% of the estimated size is a margin around the drawing area. This provides a large buffer to account for incorrect bounds calculations. The app then crops this to the edges of filled pixels so that the smallest bitmap is used to represent the final rendered equation. While this uses a lot more memory than should actually be required for rendering, it guarantees perfect results. See:

Before (notice the over-sized LaTeX, and that some expressions are partially cut-off from the right and above, or have extra space below):

After (with size estimation & croppingl this also includes a fix in UrlImageParser to ensure that LaTeX images are never auto-resized when rendering in block mode since it distorts them, and that they are properly centered):

[oppiabot]

Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue.
If you are still working on this PR, please make a follow-up commit within 3 days (and submit it for review, if applicable). Please also let us know if you are stuck so we can help you!

[oppiabot]

Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue.
If you are still working on this PR, please make a follow-up commit within 3 days (and submit it for review, if applicable). Please also let us know if you are stuck so we can help you!