Unit Testing Framework Post-Discussion

[dsouzai]

In the OMR architecture meeting on Nov 3rd, we discussed the POC unit test framework described in eclipse-openj9/openj9#16223 as well as the general notion, subtleties, and steps forward. This issue contains the takeaways and next steps.

Some key discussion points:

1. Different parts of the compiler have different notions of what a unit is. In the optimizer, a better notion of a unit might be running a specific optimization, generating code, running the code, and validating the functionality. The codegen on the other hand, might only involve giving an evaluator a specific set of IL and validating the instruction stream it generates. This partially comes down to the next point.

2. Testing a compiler involves two different notions of unit testing: testing the API contract and testing the functional semantics of a transformation. There is going to be some kind of tradeoff between these two notions as one does not want to get too specific with the tests to avoid having to write unit tests for every single routine a developer writes, but at the same time, it would not be good to be in a situation where a functionally correct change to the compiler somewhere causes a significant number of tests to fail. This leads to the next point.

3. Need to make the code more testable. This is in line with the general principle of trying to reduce our technical debt, since it essentially boils down to refactoring and cleaning up code. This is something that will take time but can be done on an ongoing basis.

4. Need to make it easy to generate IL, perhaps via TRIL abstracted away with some kind of "Builder". This is to address the fact that in general, there are a lot of structures one has to create in order to run something like an optimization pass. In order to reduce the burden on the developer, there should be a framework in place to just create the IL in a much easier way. One such option could be TRIL. In OpenJ9, because the entire JVM needs to run some kind of java code to get the JIT initialized, it may be possible to just have TRIL be placed into the annotations such that it can be consumed and then translated into the IL that can be used for testing.

5. Need to have a better framework, such as in the TestCompiler or Gtest. This is minimize the maintenance of the actual framework, as well as to minimize the amount of work this whole endeavour will require.

Steps forward (initially by me, hopefully aided by others once there's something more substantial) :

1. Continue with the basic prototype I have and iterate upon it to improve it.
2. Start with abstracting away how the IL is generated
   • For now, manually create it, but hide away somewhere else so in the future it can be changed to use some auto mechanism
3. Start looking at easy to unit test things
   • Simplifier Handlers
   • VP Handlers
   • Evaluators
4. Look into how to use the TestCompiler or Gtest framework if at all possible rather than try to create a whole new framework.
5. Look into how to generate IL automagically.

@vijaysun-omr @mstoodle @0xdaryl @jdmpapin @Leonardo2718 Feel free to add anything else I may have missed.