Endo archive performance (pre-compiled archives)

[kriskowal]

Analysis

At time of writing, we’re pretty sure that Endo archives are close to correct and many programs in Agoric-SDK work fine under archives using @agoric/bundle-source and @agoric/import-bundle. The new endoZipBase64 has behavior parity with nestedEvaluate, but not performance parity. The performance of a round-trip to and from an archive is slightly worse than half as fast / twice as slow. This isn’t good enough for us to switch the default bundle type.

I’ve used 0x to generate a flame graph of bundling and importing the Zoe contract facet in Agoric SDK. In agoric-sdk/packages/zoe, I’ve altered scripts/build-zcfBundle.js to both bundleSource and importBundle, then run npx 0x -- node -r esm scripts/build-zcfBundle.js to produce these flame graphs, once with nestedEvaluate as the bundle type, and endoZipBase64 for the other.

flame-graphs.zip

Flame graphs are statistical and always add up to 100% no matter how fast you run, so I also timed the processes. They clock in at about 3s for nestedEvaluate and 8s for endoZipBase64. Even the apples to apples portions of these processes show some slowdown, like time spent on ESM vs ESM and the evasive transform sources, which run in both cases, are slower with Endo. The time spent on Rollup is most closely related to what we do with SES, but the time is distributed differently between bundle-source and import-bundle.

The long pole appears to be the SES StaticModuleRecord constructor, which runs under importBundle and is responsible for both the analysis of the edges of the module graph, but also transforming an ESM source string to a JS program string.

I had a theory that asynchrony in the new module loader might contribute to the slowdown. A closer look at the graphs would suggest the module-to-program transform is a much larger concern.

Mitigation Proposal

Endo archives current defer an expensive module transform to runtime. This is necessary because the SES shim very carefully hides the implementation details of its internal conventions for dealing with lazy binding propagation. However, SES also has a third-party module API. So, we could move the module-to-program concern out of the runtime.

Apart from performance, we also need to restore CommonJS support #501. That involves bringing in a pure-JS JS-lexer and adapting it to do module-graph-analysis for CommonJS modules. It also involves doing some work on SES to make the third-party-module record interface more complete: dealing with exports in addition to imports.

Doing this work first is good because then we’d have a pure-JS JS-lexer to play with, which we could adapt to also analyze and transform ESM. This would be an alternative to the Babel-based transform and would not support live-binding, but would be faster and could be moved from runtime to buildtime.

We could also investigate using that lexer to do the evasive transforms for comments.

Once all transforms are shifted to build time, we might benefit from moving that job into a worker pool to run much faster. The Endo model treats individual sources as fully independent then stuffs them in a Zip file. This should be embarassingly parallel.

Assuming we can use Endo itself to archive the source-to-source transform, we could also have a reliable cache key for the transform, such that changes to the transform or input invalidate the cache key. Then, we could reliably amortize the cost of building archives over CI runs.

Currently, Agoric deployment scripts are applications that bundle other applications. We could also eliminate bundling in bundles by treating bundles as binary modules. The Endo bundler could drive the construction of bundles that are embedded in other bundles, deferring none of that work to runtime.

It may then be the case that the long pole in runtime is the async API for loading modules. With the current bundle format, the modules at runtime are coming from a Zip file in memory. We could make a fast path through SES importNow and importNowHook, or some other mechanism to emulate what XS already does for precompiled modules.

[kriskowal]

Agoric/agoric-sdk#3273 is no longer blocked on performance issues, since the refactor to remove Babel from the execution phase of archived applications.