Fix part of #4044: Add math tokenizer & parameterized test support

[BenHenning]

Explanation

Fix part of #4044
Originally copied from #2173 when it was in proof-of-concept form

This PR principally introduces the lexical tokenizer (lexer) for math expressions, as defined by the formal grammar. The tokenizer converts a string into a sequence of Tokens which provide context to the characters available in the string. While the tokenizer is not yet hooked up, it will be leveraged in an upcoming PR to add support for parsing math expressions and equations. Note that special care was taken to ensure the parser is an LL(1) parser to simplify implementation, ease future grammar additions, and to allow for a table-based parsing method in the future if increased performance is necessary.

Some caveats of the tokenizer:

• It's implemented on top of a sequence with lazy iteration to ensure that no processing time is spent parsing more than necessary (i.e. if the parser short-circuits due to an error then no additional tokenization cost is paid)
• Due to a limitation in the grammar being LL(1), function name parsing is a bit more nuanced and limiting. The lexer has to assume that the start of a function name is actually a name and not a series of variables. To account for this, a special token exists for invalid function names. See the tokenizer's test suite for more details on valid and invalid character combinations.
• Care is taken to ensure that overflowed integers & doubles result in failing tokens
• The lexer takes on a bit of the actual parsing (for integers & reals) to simplify the parser, but the majority of parsing is not done in the lexer

This PR introduces a new Truth subject for the tokenizer's Token class, though similar to subjects introduced in previous PRs this one does not include tests. #4121 is tracking adding them in the future.

To enforce the LL(1) grammar, the tokenizer makes use of a PeekableIterator that doesn't allow for lookahead beyond one character. This iterator will also be used by the upcoming parser to keep the LL(1) property.

To enable better testing for the tokenizer, I decided to add parameterized tests. Unfortunately, there wasn't an obvious way on how to exactly do this. While parameterized JUnit tests work, there are some limitations:

• They won't work with instrumentation or Robolectric
• They don't support combining parameterized & non-parameterized tests

Instead, I decided to implement a custom parameterized test runner that addresses both of the points above. I used AndroidJUnit4 and ParameterizedRobolectricTestRunner as key references into understanding how to actually build this. While the implementation we're now using is quite different than both, they serve as parts of the basis. AndroidJUnit4 is final, so we needed to implement a similar switching mechanism to select Robolectric or Espresso test helpers based on the platform (this is controlled via a test class-level annotation). Further, Robolectric's runner was important to understand how to even set up a parameterized runner, and to better understand how to make Robolectric work in this environment.

The result seems to be a very clean test runner, but I'm keen to hear reviewer's thoughts on this. Note a couple things:

• While there's an Espresso-supported runner, I didn't actually verify that it works since we haven't yet gotten instrumentation tests working natively with Bazel (though I do expect that it'll work)
• I didn't test this on Gradle, but per the CI run it appears to work just fine
• There are no tests for any of the new runners or their utilities. The utilities are fairly trivial, and the runners are difficult to test. I decided not to test them or file a tracking issue as they'll likely be the last components we add tests for when we aim to raise the codebase's code coverage (I didn't feel the difficulty was worth overcoming here over vs. manual verification).
• The runner has very robust error detection to try and reduce potential errors, including enforcing parameter order consistency
• I ran into a little snag with ktlint--see Update ktlint #4122
• The previous iteration of this led to platform selection between Espresso & Robolectric based on Bazel dependencies, but I decided to change this to be an explicit annotation. This has a number of considerations:
  • It leads to automatic build fixes (i.e. you can't compile a reference to a class without the Bazel dependency being complete)
  • It allowed me to introduce a JUnit-specific test suite (after changing MathTokenizerTest to this, I saw an almost 40x decrease in runtime; I plan to use this for all future parameterized math tests since Robolectric isn't actually needed for these and is much slower)
  • It doesn't facilitate shared tests like AndroidJUnit4, but we could conceivably create a shared runner in the future that uses either the instrumentation or Robolectric runner based on the current platform (similar to how AndroidJUnit4 works--see the commit history for the old code that did this)
• Finally, see the KDoc for OppiaParameterizedTestRunner for much more details on the runner, including both a code example and suggestions on when to actually use the runner. Further, MathTokenizerTest demonstrates real uses of the runner.

The way that the runner works is it generates a test suite that combines one runner for all non-parameterized tests with a runner for each parameterized iteration (so the total number of tests in a suite is number_of_non_parameterized_tests + sum(parameterized_iterations_in_all_parameterized_tests). The iteration names are appended to parameterized test's names to provide both Bazel filtering support and direct error reporting when tests fail (so that the exact iteration is known & can be run/debug in isolation). Android Studio will run all iterations when filtering on a method (since that should match against all), but this might be environment-specific (I verified this with Bazel in Android Studio, but not with Gradle).

Script asset changes

Test file exemptions were added for the new JUnit rules & the TokenSubject class--see above for the rationale.

Regex content exemptions were added for ParameterizedMethod since it uses Locale and capitalize. It doesn't have access to MachineLocale (at least, currently), and is only running in tests (so localization correctness isn't as important). Finally, the chosen locale to use for capitalization is US which should more or less match field names in tests for the Oppia codebase.

Further, a new regex check was added to require all uses of the new test runner to require approval (i.e. by adding an exemption). This will help ensure it doesn't get abused/used too broadly since parameterization should be an exceptional case rather than the norm.

Essential Checklist

• The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".)
• Any changes to scripts/assets files have their rationale included in the PR explanation.
• The PR follows the style guide.
• The PR does not contain any unnecessary code changes from Android Studio (reference).
• The PR is made from a branch that's not called "develop" and is up-to-date with "develop".
• The PR is assigned to the appropriate reviewers (reference).

For UI-specific PRs only

N/A -- This PR introduces a non-user facing utility, and the utility isn't yet hooked up (it will be in a subsequent PR).

[BenHenning]

Hi. As of today, some main reviewers have taken time off for the next few weeks, so it may take a little while before we can look at this PR. We appreciate your patience while some of our team members recharge. We'll be fully returning on 4 January 2021.

[oppiabot]

Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue.
If you are still working on this PR, please make a follow-up commit within 3 days (and submit it for review, if applicable). Please also let us know if you are stuck so we can help you!

[oppiabot]

Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue.
If you are still working on this PR, please make a follow-up commit within 3 days (and submit it for review, if applicable). Please also let us know if you are stuck so we can help you!

[oppiabot]

Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue.
If you are still working on this PR, please make a follow-up commit within 3 days (and submit it for review, if applicable). Please also let us know if you are stuck so we can help you!