Switch to pooling copy-on-write instantiation strategy for WASM

[koute]

polkadot companion: paritytech/polkadot#5337

Fixes #10244

Fixes #11275

This PR introduces the following changes:

• The default instantiation strategy when using the wasmtime executor (so when --wasm-execution=compiled) for normal runtime calls is now using wasmtime's native pooling strategy coupled with copy-on-write memory reuse. This is both faster and less hacky than what we're using now. (See the benchmarks below; old strategy is legacy_instance_reuse, new strategy is pooling_cow.)
• The default instantiation strategy for PVFs now also uses copy-on-write memory. (See the polkadot companion for changes, and the benchmarks below; old strategy is recreate_instance_vanilla, new strategy is recreate_instance_cow.)
• A new command-line argument is added (--wasmtime-instantiation-strategy) which makes it possible to override the desired instantiation strategy. This is useful for testing (you can check how different instantiation strategies affect performance) as well as can be used to quickly and easily revert to the old instantiation strategy (through --wasmtime-instantiation-strategy=legacy-instance-reuse) in case of any problems.

Caveats

The copy-on-write based memory reuse which the new instantiation strategy uses does require one of two things to work:

• you need to be running Linux,
• or you need to be running an Unix-like OS and you need to precompile the runtime and load it from the disk

This means that:

• On Linux this will improve the performance,
• On macOS this will degrade the performance but that can be fixed in a future PR (excluding PVFs, which are already loaded from disk so they won't become slower),
• On Windows this will degrade the performance and nothing can't be done about it without further work on wasmtime On Windows this will improve the performance as we won't have to do an expensive memset on the memory anymore. It still won't be as fast as Linux though.

When not supported the CoW-based strategy will gracefully degrade to the non-CoW variant.

Another caveat is that the pooling family of strategies in wasmtime requires us to specify hard limits on the underlying runtime. This includes a limit on the size of the runtime itself (I've currently set those to 2x to what the current Kusama runtime requires) as well as a limit on the number of concurrent runtime instantiations. (I believe I've picked a high enough number to not cause any problems; nevertheless we can "fix" this later by adding a soft-cap and a proper QoS + queue for runtime instantiations to guarantee that this limit will never be crossed.)

Since the old instantiation strategy is still available and easily accessible through a command-line argument none of these caveats should be critical at this point though, if they end up being problematic.

Benchmarks

These benchmarks are all for instantiating current Kusama runtime. X axis are the number of threads concurrently instantiating runtimes. Y axis is time. Lower is better.

Legend:

• pooling_cow - the new default instantiation strategy
• pooling_vanilla - the new default instantiation strategy when not running on Linux
• legacy_instance_reuse - the old default instantiation strategy
• recreate_instance_vanilla - the old instantiation strategy for PVFs
• recreate_instance_cow - the new instantiation strategy for PVFs
• interpreted - the wasmi-based executor

I've also done a quick and dirty benchmark where I've imported the first ~100k Polkadot blocks, and the new instantiation strategy saves around ~10% of the CPU time compared to the old one. (Although in this case it doesn't actually take less real time due to other bottlenecks; also, for more recent blocks the results will probably be different.)

Future work

(I'll later create issues for points which don't have any.)

• Remove the legacy instance reuse. - long-term we want to get rid of this instantiation mechanism since it's somewhat hacky, and it's slower than what wasmtime offers now anyway. Let's give the new instantiation scheme at least one release to marinate before we do this though, in case there will be any problems with it.
• Improve the performance on macOS. - on macOS the copy-on-write memory reuse requires the runtime to be loaded from the disk; we could automatically do this internally inside of the executor to also speed up the instantiation on macOS which should, in theory, make it as fast as on Linux.
• Pool tasks for sp_tasks::spawn and add execution limits #11227 - this API's just broken and how it works needs to be reworked; the pooling family of instantiation strategies makes it slightly worse due to their hard-cap on the maximum concurrent number of instantiations. Fortunately no one seems to be using this API so it shouldn't be a critical problem.
• Revisit the executor cache #8846 - since the wasmtime now natively caches and reuses the instances some assumptions that we've made when designing the initial executor cache no longer hold true. Besides refactoring this code there are also more improvements we can make here, including a proper queue + quality-of-service mechanisms for runtime requests: there should be a limit on how many instances can be instantiated concurrently (since we don't have infinite number of CPU cores to run them anyway), and some instantiations should be given priority over the others (e.g. any invocation related to the consensus should take priority over an RPC call).

[koute]

I've also ran the block production benchmark; looks like this improves it by ~10%:

Block production/5277 transfers (no proof)                                                                          
                        time:   [1.1800 s 1.1811 s 1.1820 s]
                        thrpt:  [4.4644 Kelem/s 4.4679 Kelem/s 4.4719 Kelem/s]
                 change:
                        time:   [-12.409% -12.140% -11.929%] (p = 0.00 < 0.05)
                        thrpt:  [+13.544% +13.818% +14.167%]
                        Performance has improved.

Block production/5277 transfers (with proof)                                                                          
                        time:   [1.2214 s 1.2228 s 1.2241 s]
                        thrpt:  [4.3108 Kelem/s 4.3156 Kelem/s 4.3205 Kelem/s]
                 change:
                        time:   [-11.395% -11.254% -11.120%] (p = 0.00 < 0.05)
                        thrpt:  [+12.511% +12.681% +12.860%]
                        Performance has improved.

[pepyakin]

Thanks for this work. I will review this PR more thoroughly in near time.

[pepyakin]

Alright, this looks good overall. There are some concerns to address after which I am optimistic we can merge this.

[koute]

@pepyakin Thanks for the review!

I think I did everything you've asked for; please do let me know if there's anything else you'd like changed!

[bkchr]

But don't we still need this for wasmi?

[koute]

Most likely yes (assuming we care about its performance here; even with the caching it's probably going to be too slow to use for anything else other than testing and/or as a backup executor in case of a disaster), although 1) for wasmi we just should just essentially add something like this deeper in the sc-executor-wasmi itself (it makes more sense for every backend to decide for itself as to what's the most efficient way to reuse it), and 2) the interface's probably going to change anyway.

[koute]

bot merge