feat(gnovm): test execution refactor + speedup

[thehowl]

This PR tackles a large amount of improvements in our current internal testing systems for the gnovm, as well as those for gno test. The primary target is the execution of filetests; however many improvements have been implemented to remove dead or duplicate code, and bring over some of the improvements that have been implemented in tests to filetests, when they come at little added "cost".

The biggest headline concerns the execution of filetests. I wrote the specific improvements undertaken in a blog post on "diary of a gnome", but here's a side-by-side comparison of the execution in this PR (left) and the execution on master (right).

• Fixes test files cannot contain a tests variable #1084
• Fixes ci: consider adding back the gnovm parallel testing #2826 (by addressing root cause of slowness)
• Fixes manage long and special tests effectively #588, only running _long filetests on master and generally speeding up test execution.

Impact

• Test context (tests and filetests)
  • Tests and filetests now share the same test.Store when running. Coupled with test.LoadImports, it allows us to "eagerly load" imports ahead of the test execution, and save it in the store. This is the primary performance improvement of this PR, as all pure packages can be run and preprocessed only once, whereas before it was once per package, and once per filetest. (More info in the blog post.)
    • One of the consequences of this is that package initialization happens outside of the test; so a filetest can no longer check the output of a println used in package initialization.
  • The default user no longer has 200 gnot by default. There are two mechanisms that make this unnecessary: // SEND:, and std.TestIssueCoin. Some test balances had to be updated.
• Filetests
  • Running filetests in gno test -v now also prints their output.
  • Realm tests are no longer executed in main.gno, but in a filename following the name of the filetest; this changed some // Realm: directives.
  • The sytem to read directives and update them in the filetests has been re-written, and should now be more resilient and with fewer "exceptions". This means that leading and trailing whitespace in output now is correctly considered as output, leading to the addition of many empty // in the tests.
    • // Error: directives are now treated the same as everything else; and are updated if the -update-golden-tests flag is passed.
  • Removed the imports metric from the runtime metrics, as it's now no longer straightforward to calculate (given that imports are loaded "eagerly").
  • Removed support for different "modes" of importing stdlibs; further removing support for gonative ([chore] Remove NativeValue and NativeType from the VM entirely #1361). The remaining gonative libraries are os, fmt, encoding/json, internal/os_test and math/big. This removes the -with-native-fallback flag from gno test.
    • Consequently, filetests ending with _native.gno have largely been removed, and those ending with _stdlibs.go have had their suffix removed.
    • Some files testing gonative types and functions which were only used in a small subset of these tests, have been removed.
• Tests
  • gno test, for testing packages, created a main_test.gno file that is then called directly from the machine on each test. This creates two identifiers, tests and runtest, which could come into conflict with those defined by the tests themselves. We now call testing.RunTest directly, without an intermediary.
  • Exports in the normal tests (ie. defined in pkg) are now visible in the tests of pkg_test. This is most useful for some standard library tests, where there often is an export_test.go with the sole scope of exporting some internal functions and methods for testing in the "real" testing file.
• gno lint, and other occasions where we use issueFromError (like when parsing errors in gno test), will now also print the column of the error if given.

Summary of internal changes

• pkg/gnolang
  • TestFiles is the new function running the filetests.
  • eval_test.go has been removed, moving some of its improvements over to TestFiles, and reducing the scope of the corresponding tests in debugger.go, to what it's actually supposed to test for.
  • As a consequence of removing all of type mappings in imports.go, I removed SetStrictGo2GnoMapping in favour of always being "strict", and not allowing defining native types of defined types any more.
    • The tests in gonative_test where primarily related to this, so I removed them. Similarly for preprocess_test.
  • TestFunc / TestMemPackage have been removed as redundant, including some of their features in cmd/go/test.go (like supporting exporting symbols in _test.gno files)
  • The Store no longer has ClearCache and SetStrictGo2GnoMapping; ClearCache now has a better substitute in BeginTransaction.
• the testing stdlib no longer caches matchPat and matchString as pure packages should not be able to modify their values during execution, and as a result of other changes this was failing.
• The tests/ directory has been removed / moved to gnovm/pkg/test. This directory should eventually contain the internal code for gno test as well; but for now I wanted to clean tests to ensure it is a directory just for integration tests, rather than code and testing systems.
  • TestMachine and TestStore have been renamed to Machine and Store, to avoid redundancy with the test package name.
• Removed plenty instructions in gnovm/Makefile as now most tests are just in pkg/gnolang.
• I removed MsgContext.Msg as unused. It can be re-added later if necessary.
• In the CI, tests are now run with -short, at least until we figure out how to make the VM efficient enough to run these tests. Aside from some filetests, this now excludes some stdlibs: bytes, strconv, regexp/syntax. I accept other proposals that could make us have fast tests on PR's, while still testing (mostly) everything.

---

[codecov]

Codecov Report

Attention: Patch coverage is 77.03524% with 189 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
gnovm/pkg/test/imports.go	59.78%	67 Missing and 7 partials ⚠️
gnovm/pkg/test/filetest.go	77.57%	50 Missing and 11 partials ⚠️
gnovm/pkg/test/test.go	82.69%	38 Missing and 12 partials ⚠️
gnovm/pkg/gnolang/gonative.go	0.00%	3 Missing ⚠️
gnovm/pkg/gnolang/store.go	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[thehowl]

Marking this as ready for review. Yes, there are a lot of changed files, but I'm thinking we can still do this in one go rather than doing multiple PRs because there are actually very few "tricky" changes (ie., those involving non-test code in pkg/gnolang are very minimal), and the benefits on our mental sanity are quite large (given that this reduces greatly the execution of both examples and tests/files by several minutes).

As usual for my large PRs, I tried updating OP to give it as much information and guidance on how to review the PR. 🥂

[mvertes]

Only minor not-blocking nitpick suggestions for this monster PR. Super nice work, congratulations @thehowl!

[ltzmaxwell]

this looks generally good after a quick glance, I'll need to postpone this to next week since there are some more prioritized. @thehowl .

[thehowl]

applied code review changes from @mvertes. thank you!