switch to new marshalling format everywhere #6822
Comments
|
Please consider also doing #6879 / endojs/endo#1478 if/when doing this. |
|
I just made #7013 to explore changing the board marshaller and hence smart wallet etc. As to the others...
I'm not sure about that one... I suspect it's not used. @michaelfig ?
That's unserialize-only; it will Just Work.
That one goes to chainStorage. I should add it to #7013 .
The output of that one is fed immediately to
adding that to #7013
That's where I started.
I think those are only used in the ag-solo wallet.
I suspect |
|
On one hand, smaller serialized data means fewer bytes published to IAVL, which will speed up validators (slightly) and speed up client access (slightly), which are good. On the other hand, asking off-chain developers to use a custom deserializer for the data they fetch from the chain is a drag. OT3H they pretty much have to do that for the old format too, since JSON doesn't even support BigInts (which are pervasive in the data we publish), but at least plain I think that, if it doesn't cause insurmountable problems for the front end, that we should make this change while we still can (i.e. before the release). |
This changes every `makeMarshal()` instance which performs object
graph serialization to add the `{serializeBodyFormat: 'smallcaps'}`
option. All emitted data will be in the "smallcaps" format, which is
more concise (less overhead). For example, `1n` (a BigInt) is encoded
as `#"+1"` instead of `{"@qclass":"bigint","digits":"1"}`.
The old format was JSON except with those special `@qclass` objects to
capture everything that JSON could not, including object
references ("slots"). The new format is a leading `#` prefix, plus
JSON, except that all the special cases are expressed as strings.
Most significantly, this changes the format of the data we publish to
`chainStorage`. The old format encoded simple data as mostly JSON, so
bespoke parsers mostly worked, but even BigInts required special
handling. Off-chain followers who perform RPC queries to the chain are
highly encouraged to use the `@endo/marshal` library (or equivalent)
to parse the data. This library can understand old-format data too,
such as chain data published before this upgrade.
Notable changes:
* Many ad-hoc serializers were replaced with a real `makeMarshal`,
which requires hardened inputs, so some new `harden()` calls were
added
* Some marshallers which only perform unserialization (`fromCapData`)
were not modified
* All unit tests which compared serialized data against
manually-constructed examples had their examples updated
* Previously, the `makeMockChainStorageRoot()` utility parsed
Far/Remotable objects into an object with `{ iface }`. Now, it is
unserialized into actual Remotables. Both `t.deepEqual` and
`t.snapshot` will correctly compare the `iface` values of two Far
objects, so tests should simply construct a `Far(iface)` and include
it in the specimen data. As a result, many `t.snapshot`-style tests
had their snapshot data changed.
closes #6822
Co-authored-by: Chip Morningstar <[email protected]>
This changes every `makeMarshal()` instance which performs object
graph serialization to add the `{serializeBodyFormat: 'smallcaps'}`
option. All emitted data will be in the "smallcaps" format, which is
more concise (less overhead). For example, `1n` (a BigInt) is encoded
as `#"+1"` instead of `{"@qclass":"bigint","digits":"1"}`.
The old format was JSON except with those special `@qclass` objects to
capture everything that JSON could not, including object
references ("slots"). The new format is a leading `#` prefix, plus
JSON, except that all the special cases are expressed as strings.
Most significantly, this changes the format of the data we publish to
`chainStorage`. The old format encoded simple data as mostly JSON, so
bespoke parsers mostly worked, but even BigInts required special
handling. Off-chain followers who perform RPC queries to the chain are
highly encouraged to use the `@endo/marshal` library (or equivalent)
to parse the data. This library can understand old-format data too,
such as chain data published before this upgrade.
Notable changes:
* Many ad-hoc serializers were replaced with a real `makeMarshal`,
which requires hardened inputs, so some new `harden()` calls were
added
* Some marshallers which only perform unserialization (`fromCapData`)
were not modified
* All unit tests which compared serialized data against
manually-constructed examples had their examples updated
* Previously, the `makeMockChainStorageRoot()` utility parsed
Far/Remotable objects into an object with `{ iface }`. Now, it is
unserialized into actual Remotables. Both `t.deepEqual` and
`t.snapshot` will correctly compare the `iface` values of two Far
objects, so tests should simply construct a `Far(iface)` and include
it in the specimen data. As a result, many `t.snapshot`-style tests
had their snapshot data changed.
refs #6822
Co-authored-by: Chip Morningstar <[email protected]>
This changes every `makeMarshal()` instance which performs object
graph serialization to add the `{serializeBodyFormat: 'smallcaps'}`
option. All emitted data will be in the "smallcaps" format, which is
more concise (less overhead). For example, `1n` (a BigInt) is encoded
as `#"+1"` instead of `{"@qclass":"bigint","digits":"1"}`.
The old format was JSON except with those special `@qclass` objects to
capture everything that JSON could not, including object
references ("slots"). The new format is a leading `#` prefix, plus
JSON, except that all the special cases are expressed as strings.
Most significantly, this changes the format of the data we publish to
`chainStorage`. The old format encoded simple data as mostly JSON, so
bespoke parsers mostly worked, but even BigInts required special
handling. Off-chain followers who perform RPC queries to the chain are
highly encouraged to use the `@endo/marshal` library (or equivalent)
to parse the data. This library can understand old-format data too,
such as chain data published before this upgrade.
Notable changes:
* Many ad-hoc serializers were replaced with a real `makeMarshal`,
which requires hardened inputs, so some new `harden()` calls were
added
* Some marshallers which only perform unserialization (`fromCapData`)
were not modified
* All unit tests which compared serialized data against
manually-constructed examples had their examples updated
* Previously, the `makeMockChainStorageRoot()` utility parsed
Far/Remotable objects into an object with `{ iface }`. Now, it is
unserialized into actual Remotables. Both `t.deepEqual` and
`t.snapshot` will correctly compare the `iface` values of two Far
objects, so tests should simply construct a `Far(iface)` and include
it in the specimen data. As a result, many `t.snapshot`-style tests
had their snapshot data changed.
refs #6822
Co-authored-by: Chip Morningstar <[email protected]>
|
I gather @warner and @FUDCo have done their bit and @samsiegart is checking the dapps. |
|
All done! |
What is the Problem Being Solved?
During today's show-and-tell presentation by @samsiegart , @erights noticed that the chain-storage published data was still using the old
@qclassserialization format. We switched SwingSet to use theserializeBodyFormat: 'smallcaps'format in f289ec0 (#6326), but there are other places in agoric-sdk which usemakeMarshalto serialize data, which are still using the old format. I found the following withrgjs -w makeMarshal |grep -v test |grep -v SwingSet | grep -v import:The basic fix is to add
serializeBodyFormat: 'smallcaps'to theoptionsbag, but since the chain already has IAVL-tree data in the old format (which won't go away during the bulldozer upgrade), we must make sure the client-side tools that follow that data will be able to handle both.makeMarshalis tolerant of both, but if we have any other code that thinks it knows about the serialization details, that code will need to change.We had originally planned to drop read-side support for the old format from
makeMarshalafter completing the transition of all clients. However the presence of old-format IAVL data on the chain may prevent us from doing that, or perhaps preventing the client code from upgrading to a newer version ofmakeMarshal.BTW,
makeMarshallives in the Endo project/repository.Test plan
Most of these should work automatically by the Casting abstraction.
The text was updated successfully, but these errors were encountered: