plan for durable/non-durable/soft-durable tests in ERTP/zoe/contracts

[warner]

What is the Problem Being Solved?

@Chris-Hibbert and @erights have been enhancing the ERTP code to be "durable": storing its data in durable collections, and providing enough API surface to enable clients (e.g. Zoe and contracts) to involve ERTP in the upgrade process. When a vat that includes a copy of Zoe is upgraded, Zoe will need to be told (at the right time, during startVat/buildRootObject) that an upgrade is happening, so Zoe can re-attach its internal Kinds. Zoe uses ERTP (so Zoe is a "client" of ERTP), and during this call, Zoe is responsible for invoking an ERTP-provided "we're upgrading now" interface, so ERTP can also re-attach its own internal Kinds.

By converting ERTP's internal collections to be durable (e.g. makeScalarBigMapStore(label, { durable: true })), ERTP tightens a requirement on its API surface. Any objects passed into ERTP that happen to be stored in one of these collections are now required to be durable-compatible. This includes vat imports (Presences) and locally-hosted durable objects ("durable Representatives": Representatives of a durable Kind), but excludes Promises, locally-hosted non-durable objects ("virtual Represtatives: Representatives of a merely-virtual Kind), and locally-hosted ephemeral objects (Remotables). For the case of ERTP, the concern is the "Amounts" that are used in Purses and Payments, in particular when the denomination is a Set or other non-Nat: an Invitation, rather than 35 BLD tokens. These Amounts can include object references, so the durable-compatible-or-not requirement becomes relevant.

This new API requirement is imposed on downstream client who use ERTP (e.g. Zoe). Those clients have unit tests that want to focus attention on the logic of the client, not the requirements of ERTP. Many of those tests run in a non-swingset environment (packages/zoe/test/unitTests/), where a special library is imported that provides mock implementations of the virtual/durable collection types. Other tests (packages/zoe/test/swingsetTests/) are using a real swingset and real vats, however maybe not as many vats as a zoe-using production service would configure.

Both kinds of tests are likely to want to provide ephemeral or locally-hosted non-durable objects to the ERTP API, where a production environment would provide either locally-hosted durable objects, or host those objects in a different vat altogether (thus being "imports" from the ERTP-hosting vat's point of view, and thus durable-compatible). It would be a burden to rewrite those tests to create durable-compatible objects, when the point of the test is to exercise the e.g. Zoe logic.

In production environments, the durable-compatible requirement may be met by virtue of objects being defined in vat A, sent to vat B, and then being added to a durable collection in vat B. During testing, this boundary between vat A and B is a burden to establish (it requires a real kernel, not the mock environment used by most unit tests), and it limits the debuggability of a single shared memory space. The requirement for upgrade makes vat boundaries visible, and thus something the programmer must be aware of, and program against. But still it would be nice to be able to write tests that are unaware of upgrade, so they can focus on other logic.

So there's a desire for a way to reduce the burden. One approach is the "only pay for what you need" scheme. Here, we would split the ERTP API into two pieces. The first is the traditional non-durable one. Clients using only this API do not need to satisfy any additional requirements: they can feed whatever they like into the API, durable-compatible or not, and the API works as expected. The second API is used to get a durable IssuerKit. If you use it, you must provide durable-compatible Amounts to the API (plus extra facilities like baggage when you create the IssuerKit).

Downstream libraries like Zoe can remain blissfully ignorant of durability up until the point when it tries to benefit from durability. At that point, the Zoe authors have a choice: implement the same API split, or not. If Zoe splits, then downstream clients of Zoe can remain ignorant of durability until they start using the "durable zoe" API. If Zoe does not split, then all downstream clients must become aware of durability/etc as soon as they upgrade their dependency to require/admit the newer Zoe.

A second approach, perhaps complementary, is to introduce a flag to the testing environment that says "relax the durability requirements". This would be a pretty global flag, meant to be set in some unit tests but not in others. If set, it would change the definition of "durable-compatible" to admit locally-hosted ephemeral objects (Remotables) as well as locally-hosted merely-virtual objects. It would continue to reject Promises. Unlike the currently-implemented fakeDurable option to makeScalarBigMapStore(), this flag would be swingset-wide, rather than being set on some collections but not on others.

If ERTP continues to provide a unified API (using durability internally, even if the client didn't ask for it), then clients can set their test environments to relax the durability requirement, rather than changing their tests to meet it. However all downstream tests (e.g. contracts) must do the same: Zoe cannot hide its upstream obligations from its downstream clients.

The special library that provides mock virtual/durable types (used by most zoe unit tests) could be changed to set this relax-durable-compatible flag, which would take care of most unit tests in our tree. The test/swingsetTests/ would need to be changed, to add a new config flag to their swingset setup functions. I count about 14 of these in agoric-sdk, probably others in the dapp repos.

• question 1: split ERTP's API into durable/non-durable makeIssuerKit ?
• question 2: implement a swingset-global "relax durable-compatible" flag ?

Description of the Design

A split ERTP API would probably have three functions.

• non-durable: makeIssuerKit(stuff)
• durable:
  • makeDurableIssuerKit(stuff, baggage)
  • handleUpgrade(baggage)

Of course makeIssuerKit(stuff, { baggage }) could be used instead of a distinct function. And handleUpgrade is not a great name.

The handleUpgrade function needs to be called during buildRootObject by the client of ERTP (e.g. zoe), so ERTP can re-attach all the kinds stored in the baggage. handleUpgrade does not allocate a new Issuer: only the make* functions do that. makeDurableIssuerKit should use a provide-pattern (with baggage as the backing store) to manage the KindHandle. The stuff needs to include a distinct name that we know where to look within the baggage.

Clients who restrict themselves to the non-durable makeIssuerKit() get a merely-virtual Issuer that can accept non-durable-compatible Amount objects. Those clients do not need to do anything special during their unit tests, as long as all the other libraries they use also stick to the non-durable API.

The "relax durable-compatible" flag would be set as an option on the swingset config object. It would be propagated to all liveslots instances (exactly as enableFakeDurable is now; it would just be a renaming of the existing flag). Within liveslots/collectionManager.js, it would change the behavior of assertCorrectDurabilityFlags to allow Remotables and merely-virtual Represntatives, while still rejecting Promises. It should also be checked during upgrade (stopVat()), to complain if someone forgets and leaves it enabled in a test or other environment that also attempts to perform a vat upgrade (which is where most of the explosion would occur).

Security Considerations

Allowing non-durables into durable collections and then allowing an upgrade would probably allow the v2 vat to access things that it should not, hence the desire to crash during upgrade if the flag was set. We might consider having swingset print a stderr warning at startup if the config flag is set, to reduce the changes of production environments being launched with the relaxed rules.

Test Plan

[Chris-Hibbert]

The durable ZCF PR included a test of upgrading the coveredCall contract.