cosmic-swingset support for module installation: agd publish

[warner]

What is the Problem Being Solved?

#9517 describes a desire to improve the "install contract bundle to chain" pathway. Currently we install each bundle in a single (signed) cosmos message, containing its entire contents, even though most new bundles have significant overlap with bundles that already exist on the chain (eg most of Endo, ERTP, Nat, a lot of support code). We could shrink the cosmos messages considerably if we could install only the portions that were not already on the chain.

#9522 adds swing-store support, and #9523 adds kernel support.

This ticket is about the cosmic-swingset support, both the chain-side and the agd client side.

Description of the Design

TBD

We need two new cosmos messages: one to install a module (including an alleged ModuleID), and a second to install a compartment map (including an alleged BundleID). These should continue to have fees based on the size of their uncompressed contents (although the messages themselves might be compressed).

We also need an RPC call which exposes enough information for a client to determine which modules are already installed, and/or which of their new bundle's modules are still missing. Some options:

• send a list of ModuleIDs to the node, get back a list of missing ones
• send nothing to the node, get back a list of all installed ModuleIDs
• send nothing to the node, get back a bloom filter (compressed+lossy set of ModuleIDs)
  • this introduces a small chance of the install-compartment-map call failing

Then, we need an agd publish or agd publish-bundle subcommand, which should be given the filename of a local .bundle file (JSON-encoded bundle object). It should:

• examine the bundle, assert the moduleFormat and that all the hashes are correct
• build a list of ModuleIDs
• use the RPC call to figure out which modules are not yet installed on the chain
• build cosmos messages to install those modules
• build a cosmos message to install the compartment map
• aggregate those messages into a minimal number of cosmos transactions (while keeping each message smaller than a configurable size limit)
• sign and submit the resulting messages
• watch the chain for success before returning

We also need a new proposal/core-eval generation process, to take advantage of this new subcommand. #9517 has some links.

Security Considerations

The chain must not accept corrupted modules. The kernel won't accept them (whether enforced by the kernel or swingstore), and cosmic-swingset should not retain any state about the availability of modules (it should defer to the kernel), so cosmic-swingset does not need to check any hashes.

This new mechanism makes it cheaper to install bundles, when they contain modules that overlap with already-installed bundles. We should make sure this doesn't make it cheaper to abuse the chain.

Scaling Considerations

This design does not address the problem of individual modules becoming too large to fit into a single cosmos txn. Hopefully that won't be a concern for a while (entire bundles were at the hairy edge, but the module sizes are strictly smaller). If it becomes a problem, we could consider additional cosmos messages, to submit fragments, and then install a module defined by a list of fragment IDs. Such a design must answer questions about how long the fragments will be retained, and ensuring that this doesn't create a new opportunity for abuse.

Test Plan

The new txn types should get golang unit tests, however I don't know how effectively we can test txns that result in JS/swingset calls.

The new agd subcommand should get units tests that behave like existing tests, I'm not sure how those work either.

Upgrade Considerations

just the usual ones: this will require a chain-halting upgrade, to install both the new golang RPC calls and the new kernel/swingstore support. That will trigger a #8089 swing-store SQL upgrade. The frontend agd subcommand will require clients to build/use a new version of agd, which will make new RPC queries, so it will only work against an upgraded chain.