Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

anachrophobia strikes vat v18 on restarting agoric on mainnet #6588

Closed
shivlim opened this issue Nov 18, 2022 · 12 comments · Fixed by #6758
Closed

anachrophobia strikes vat v18 on restarting agoric on mainnet #6588

shivlim opened this issue Nov 18, 2022 · 12 comments · Fixed by #6758
Assignees
@mhofman
Labels
bug Something isn't working SwingSet package: SwingSet vaults_triage DO NOT USE xsnap the XS execution tool
Milestone
Vaults EVP

Comments

@shivlim
Copy link

shivlim commented Nov 18, 2022 •
edited by dckc
Loading

Update: patch available

see #6588 (comment)

Original report

Had to restart node in order to put gas prices and node keeps crashing because of the following error

2022-11-18T15:23:06.120Z SwingSet: xsnap: v14: UnhandledPromiseRejectionWarning: No quorum
2022-11-18T15:23:06.343Z SwingSet: xsnap: v14: UnhandledPromiseRejectionWarning: No quorum
2022-11-18T15:23:06.367Z SwingSet: xsnap: v14: UnhandledPromiseRejectionWarning: No quorum
2022-11-18T15:23:06.393Z SwingSet: xsnap: v14: UnhandledPromiseRejectionWarning: No quorum
2022-11-18T15:23:36.075Z SwingSet: kernel: anachrophobia strikes vat v18
2022-11-18T15:23:36.075Z SwingSet: kernel: expected: {"0":"vatstoreGet","1":"vc.1096.|schemata","length":2}
2022-11-18T15:23:36.075Z SwingSet: kernel: got     : {"0":"vatstoreGetAfter","1":"","2":"vc.1096.","3":"vc.1096.{","length":4}
portHandler threw (Error#1)
Error#1: historical inaccuracy in replay of v18
  at requireIdentical (.../swingset-vat/src/kernel/vat-loader/transcript.js:8:8)
  at Object.simulateSyscall (.../swingset-vat/src/kernel/vat-loader/transcript.js:63:22)
  at Object.syscallFromWorker (.../swingset-vat/src/kernel/vat-loader/manager-helper.js:249:30)
  at handleUpstream (.../swingset-vat/src/kernel/vat-loader/manager-subprocess-xsnap.js:86:11)
  at handleCommand (.../swingset-vat/src/kernel/vat-loader/manager-subprocess-xsnap.js:109:14)
  at runToIdle (packages/xsnap/src/xsnap.js:208:37)
  at runMicrotasks (<anonymous>)
Cannot initialize Controller Error: historical inaccuracy in replay of v18
agd.service: Main process exited, code=exited, status=1/FAILURE
agd.service: Failed with result 'exit-code'.
@shivlim shivlim added the bug Something isn't working label Nov 18, 2022
@dckc
Copy link
Member

dckc commented Nov 18, 2022 •
edited
Loading

work-around: resume from snapshot (thru 6 Dec?)

2 validators report getting online using shapshots

I presume they mean from https://polkachu.com/tendermint_snapshots/agoric

I downloaded the 2 snapshots there when @mhofman suggested that it would be useful to get v18 snapshots from validators who were not affected:

$ sha256sum agoric_74*
1121aae6bd9cdd61a0ec542731a30ad137d5f8ef52971aa8d68c0b37e56c2137  agoric_7461424.tar.lz4
3e7f56baeede00f5393c22aeb539cea6023c9cf6371689910cbc9b097f7e95ae  agoric_7474766.tar.lz4

I confirmed that agoric_7474766.tar.lz4 includes XS snapshots such as:

data/ag-cosmos-chain-state/xs-snapshots/0e3ef21a86cad9a12b5dcc5373f3505023bd832b6f2fe6317e86a50b236c4423.gz

(not sure which vat that's from)

Note also discord discussion

in agoric-3 #peer-support, including original problem report at 9:19am America/Chicago time.

@dckc dckc changed the title Issue on restarting agoric on mainnet anachrophobia strikes vat v18 on restarting agoric on mainnet Nov 18, 2022
@dckc dckc added the SwingSet package: SwingSet label Nov 18, 2022
This was referenced Nov 18, 2022
Open
Closed
@mhofman
Copy link
Member

mhofman commented Nov 19, 2022

I made a change to the vat replay logic to re-trigger snapshot making at the same interval as occurred during consensus execution, and nuked the vat18 snapshot reference from the kvStore, causing the vat to replay entirely from transcript on restart. When using a state snapshot from polkachu's service taken at block height 7474766, I ended up encountering an anacrophobia error at deliveryNum 32828 (out of 108002). This likely means the restart based divergence started much before.

I'm now working on getting a transcript from a node that never restarted.

@mhofman
Copy link
Member

mhofman commented Nov 22, 2022

I have managed to reproduce divergent runs and different XS snapshots of vat18 based on transcripts with different reloads schedules.

First I extracted a transcript from a state snapshot provided by polkachu's validator. As stated above it would hit an anachrophobia when trying to replay without reloads at delivery 32828. When adding a snapshot reload at delivery 32002, 39002 and 102002, the transcript replay would succeed. I suspect that other reloads actually happened on that node (since they provide a daily snapshot), but that those did not have an impact on the transcript.

I've also recovered a state dump of our own follower node used for telemetry and stats. While I'm unclear on the restart schedule of that node, a transcript replay does seem to execute without any reloads being needed.

The really interesting part comes when comparing the XS snapshot hashes. While they diverge around the time of the reloads, they converge back after. Even more suprising, in one case they diverge, converge, diverge, and converge again, all without a reload.

--- snapshot-activity-telemetry.jsonl   2022-11-21 19:17:11.518713371 -0800
+++ snapshot-activity-polkachu.jsonl    2022-11-21 19:17:14.899851024 -0800
@@ -33,12 +33,14 @@
 {"type":"save","transcriptNum":31002,"snapshotID":"4a646c59d68dab827626ad0c3712b9f18478623e8a75ff60cdd76e04f1781f6a"}
 {"type":"save","transcriptNum":32002,"snapshotID":"cddb4164b6519b69685adba58a14c73ef0dcc7703296fc5249020053d1231cda"}
-{"type":"save","transcriptNum":33002,"snapshotID":"33b4334340a3a636ec6bf07f2ed96481fd483920998061fa3a2d569aa7c55b97"}
+{"type":"load","snapshotID":"cddb4164b6519b69685adba58a14c73ef0dcc7703296fc5249020053d1231cda","loadSnapshotID":"cddb4164b6519b69685adba58a14c73ef0dcc7703296fc5249020053d1231cda"}
+{"type":"save","transcriptNum":33002,"snapshotID":"c49065f0c5db738073201917df815df4adad74fdf7e3b7fc63a78e99110b1d90"}
 {"type":"save","transcriptNum":34002,"snapshotID":"65ea1229e647bac86f84c0157f1fc0b841109faa768e6e450cb5e9103d7208ee"}
-{"type":"save","transcriptNum":35002,"snapshotID":"ed1ebbc28f0c3ea87ac7d7b60623eff093d191e79d9c1da2fd1502d4a0b54390"}
+{"type":"save","transcriptNum":35002,"snapshotID":"91f2004a62bec7677069ae738da53c8fb6fb64a1e6330dc749f702b7ce7c0fca"}
 {"type":"save","transcriptNum":36002,"snapshotID":"3b05f0845cd79f03ea0ad6ba32403fddc60d0bae761a9262594713f3153e31d2"}
 {"type":"save","transcriptNum":37002,"snapshotID":"065bd999e21ae80af53fa6efa1023aa818874fbd8b89289defd47c95a5eaf123"}
 {"type":"save","transcriptNum":38002,"snapshotID":"a01360975d496a1e7236836f46f66921a23599904fb51711a8fcd02e52f71672"}
 {"type":"save","transcriptNum":39002,"snapshotID":"9e9176fbd6c60ca817af710e3e519142fe50f6ff41fb75712e15a24df89135ad"}
-{"type":"save","transcriptNum":40002,"snapshotID":"c9aa81aa36269d1c830edda88b70ff9cbc28710349b2231828632d84833fdac4"}
+{"type":"load","snapshotID":"9e9176fbd6c60ca817af710e3e519142fe50f6ff41fb75712e15a24df89135ad","loadSnapshotID":"9e9176fbd6c60ca817af710e3e519142fe50f6ff41fb75712e15a24df89135ad"}
+{"type":"save","transcriptNum":40002,"snapshotID":"861dbba75777a50b309d9372d18651fa5ac33a401fdde2a822e553c045df3e9a"}
 {"type":"save","transcriptNum":41002,"snapshotID":"72bbd041c68913eec55f9a6b133a731e6ee114c1de892086b369d12f66eb4aa4"}
 {"type":"save","transcriptNum":42002,"snapshotID":"2629a56d6e8aea8b9bca6de1eeaa6950a080c106913341ebde63da852ddcbbdd"}
@@ -103,4 +105,5 @@
 {"type":"save","transcriptNum":101002,"snapshotID":"25724b234a5ef9415c16adef98232f058a01b1f1bcfb3437ab86fb04f665cd91"}
 {"type":"save","transcriptNum":102002,"snapshotID":"a869bd0e70ab10564eeb4066060f2d9d53ceeab9405e341642d61e37ddb46876"}
+{"type":"load","snapshotID":"a869bd0e70ab10564eeb4066060f2d9d53ceeab9405e341642d61e37ddb46876","loadSnapshotID":"a869bd0e70ab10564eeb4066060f2d9d53ceeab9405e341642d61e37ddb46876"}
 {"type":"save","transcriptNum":103002,"snapshotID":"876f7bb8e2834e9adb2708ca3165fcc406121b32399efb0267ea4df1347ad051"}
 {"type":"save","transcriptNum":104002,"snapshotID":"917e5f2c0cbff6ec48e4ccf3695e7bfb04663f8fe435d89c30ad86c4d186e30c"}
@@ -108,6 +111,12 @@
 {"type":"save","transcriptNum":106002,"snapshotID":"566fc3bb54b3ffeb9df0440743a941b64b74e40fc05fea8b5e97ba7f62670e18"}
 {"type":"save","transcriptNum":107002,"snapshotID":"3723367790bf20fed5b91099468f84ff85e0d4c87c86b980e8be52a66121e8d9"}
-{"type":"save","transcriptNum":108002,"snapshotID":"22c0a6c3000dd08c0e68f97aaf203c01963cb315e1085eae5a47cb2a840b4433"}
+{"type":"save","transcriptNum":108002,"snapshotID":"fa8c694faf74740f15b439a8ac877610c4128f3869c972e7667bfe31cad0687c"}
 {"type":"save","transcriptNum":109002,"snapshotID":"0b6ba6bce4e7f3af4ba2fc3965d45a5003ef83d1302606fa3deff8d49e907696"}
 {"type":"save","transcriptNum":110002,"snapshotID":"803f2d5e9a722914cfbfb6bdb6d7140570f051137c41c48dc86fdb481cf1dab1"}
-{"type":"save","transcriptNum":111002,"snapshotID":"080f6ef61b30c5b53cf659b29fe21747498914b3504455b31425852c56311b82"}
+{"type":"save","transcriptNum":111002,"snapshotID":"8910177bb6df4cd116335e4608c03ff47e383b6cb4ee290f4cfba8f82401c271"}

The raw diff of the dumps of these snapshots all seem rather large, but it feels like they're actually mostly shifted memory due to some extra/missing slots in one of the snapshots. The exception are the snapshots for delivery 111002 which only differ in order of 2 strings in memory.

To replay, use the replay-transcript.js tool from branch mhofman/6588-diagnose.

Transcripts (must gunzip):

Diff of the prettified transcripts

transcript-v18-polkachu-vs-telemetry.diff:

--- transcript-v18-polkachu-first-111796.psst	2022-11-21 17:26:12.723609002 -0800
+++ transcript-v18-telemetry.psst	2022-11-21 17:26:44.794633029 -0800
@@ -2352049,44 +2352049,40 @@
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "vatstoreSet",
         "idCounters",
         "{\"exportID\":1250,\"collectionID\":630,\"promiseID\":33453}"
       ],
       "response": [
         "ok",
         null
       ]
     }
   ]
 }
 {
-  "type": "heap-snapshot-load",
-  "snapshotID": "cddb4164b6519b69685adba58a14c73ef0dcc7703296fc5249020053d1231cda"
-}
-{
   "transcriptNum": 32003,
   "d": [
     "notify",
     [
       [
         "p+33451",
         false,
         {
           "body": "{\"body\":\"{\\\"brands\\\":[{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: Zoe Invitation brand\\\",\\\"index\\\":0},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"set\\\"},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: Zoe Invitation issuer\\\",\\\"index\\\":1},\\\"petname\\\":\\\"invitations\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDC_axl brand\\\",\\\"index\\\":2},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDC_axl issuer\\\",\\\"index\\\":3},\\\"petname\\\":\\\"USDC_axl\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDC_grv brand\\\",\\\"index\\\":4},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDC_grv issuer\\\",\\\"index\\\":5},\\\"petname\\\":\\\"USDC_grv\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDT_axl brand\\\",\\\"index\\\":6},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDT_axl issuer\\\",\\\"index\\\":7},\\\"petname\\\":\\\"USDT_axl\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDT_grv brand\\\",\\\"index\\\":8},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDT_grv issuer\\\",\\\"index\\\":9},\\\"petname\\\":\\\"USDT_grv\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: BLD brand\\\",\\\"index\\\":10},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":11},\\\"petname\\\":\\\"BLD\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: IST brand\\\",\\\"index\\\":12},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: IST issuer\\\",\\\"index\\\":13},\\\"petname\\\":\\\"IST\\\"}],\\\"lastOfferId\\\":1667445912281,\\\"offerToUsedInvitation\\\":{},\\\"purses\\\":[{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":0},\\\"value\\\":[]},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":0}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":2},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"21090041\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":2}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":4},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"0\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":4}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":6},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"0\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":6}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":8},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"0\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":8}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":10},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"19878927\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":10}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":12},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"401000\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":12}}]}\",\"slots\":[\"board04016\",null,\"board0223\",\"board0592\",\"board0566\",\"board0425\",\"board0188\",\"board0257\",\"board02810\",\"board0639\",\"board04312\",null,\"board02314\",\"board05311\"]}",
           "slots": []
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "send",
         "o-4669",
         {
@@ -2412061,60 +2412057,40 @@
 {
   "transcriptNum": 32828,
   "d": [
     "notify",
     [
       [
         "p+34296",
         false,
         {
           "body": "{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDC_grv brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}",
           "slots": [
             "o-74"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.640.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.640.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.640.",
         "vc.640.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     }
   ]
 }
 {
   "transcriptNum": 32829,
   "d": [
     "notify",
     [
@@ -2412729,60 +2412705,40 @@
 {
   "transcriptNum": 32836,
   "d": [
     "notify",
     [
       [
         "p+34305",
         false,
         {
           "body": "{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDT_axl brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}",
           "slots": [
             "o-77"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.640.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.640.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.640.",
         "vc.640.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     }
   ]
 }
 {
   "transcriptNum": 32837,
   "d": [
     "notify",
     [
@@ -2414311,60 +2414267,40 @@
 {
   "transcriptNum": 32856,
   "d": [
     "notify",
     [
       [
         "p+34327",
         false,
         {
           "body": "{\"updateCount\":3,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: IST brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"250000\"}}}",
           "slots": [
             "o-57"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.640.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.640.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.640.",
         "vc.640.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-4832",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[3]]",
@@ -2557599,40 +2557535,80 @@
   "transcriptNum": 34808,
   "d": [
     "notify",
     [
       [
         "p+36357",
         false,
         {
           "body": "{\"@qclass\":\"slot\",\"iface\":\"Alleged: Virtual Purse\",\"index\":0}",
           "slots": [
             "o-5122"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "vatstoreGet",
+        "vc.678.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.678.|label"
+      ],
+      "response": [
+        "ok",
+        "brand purses"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.679.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.679.|label"
+      ],
+      "response": [
+        "ok",
+        "payments queues"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
         "vc.678.|o-77"
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "send",
         "o-77",
         {
           "methargs": {
             "body": "[\"getDisplayInfo\",[]]",
             "slots": []
           },
           "result": "p+36372"
         }
       ],
       "response": [
@@ -2558273,40 +2558249,80 @@
   "transcriptNum": 34816,
   "d": [
     "notify",
     [
       [
         "p+36365",
         false,
         {
           "body": "{\"@qclass\":\"slot\",\"iface\":\"Alleged: Virtual Purse\",\"index\":0}",
           "slots": [
             "o-5124"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "vatstoreGet",
+        "vc.678.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.678.|label"
+      ],
+      "response": [
+        "ok",
+        "brand purses"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.679.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.679.|label"
+      ],
+      "response": [
+        "ok",
+        "payments queues"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
         "vc.678.|o-80"
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "send",
         "o-80",
         {
           "methargs": {
             "body": "[\"getDisplayInfo\",[]]",
             "slots": []
           },
           "result": "p+36381"
         }
       ],
       "response": [
@@ -2559076,40 +2559092,60 @@
 {
   "transcriptNum": 34826,
   "d": [
     "notify",
     [
       [
         "p+36380",
         false,
         {
           "body": "{\"updateCount\":2,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDT_axl brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}}",
           "slots": [
             "o-77"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.680.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.680.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.680.",
         "vc.680.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-5125",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[2]]",
@@ -2560001,40 +2560037,60 @@
 {
   "transcriptNum": 34836,
   "d": [
     "notify",
     [
       [
         "p+36395",
         false,
         {
           "body": "{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: BLD brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"3857339\"}}",
           "slots": [
             "o-84"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.680.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.680.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.680.",
         "vc.680.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     }
   ]
 }
 {
   "transcriptNum": 34837,
   "d": [
     "notify",
     [
@@ -2560182,40 +2560238,60 @@
 {
   "transcriptNum": 34841,
   "d": [
     "notify",
     [
       [
         "p+36400",
         false,
         {
           "body": "{\"updateCount\":3,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: IST brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"250000\"}}}",
           "slots": [
             "o-57"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.680.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.680.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.680.",
         "vc.680.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-5130",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[3]]",
@@ -2567217,60 +2567293,40 @@
 {
   "transcriptNum": 34956,
   "d": [
     "notify",
     [
       [
         "p+3053",
         false,
         {
           "body": "{\"updateCount\":4,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: IST brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"50000\"}}}",
           "slots": [
             "o-57"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.76.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.76.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.76.",
         "vc.76.{"
       ],
       "response": [
         "ok",
         [
           "vc.76.fc27841b0c8805000",
           "{\"body\":\"{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: Zoe Invitation brand\\\",\\\"index\\\":0},\\\"value\\\":[{\\\"description\\\":\\\"Voter2\\\",\\\"handle\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: InvitationHandle\\\",\\\"index\\\":1},\\\"installation\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: BundleInstallation\\\",\\\"index\\\":2},\\\"instance\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: InstanceHandle\\\",\\\"index\\\":3}}]}\",\"slots\":[\"o-63\",\"o-517\",\"o-241\",\"o-242\"]}"
         ]
       ]
     },
     {
       "d": [
         "vatstoreGetAfter",
         "vc.76.fc27841b0c8805000",
         "vc.76.",
         "vc.76.{"
       ],
@@ -2869108,44 +2869164,40 @@
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "vatstoreSet",
         "idCounters",
         "{\"exportID\":1506,\"collectionID\":758,\"promiseID\":40737}"
       ],
       "response": [
         "ok",
         null
       ]
     }
   ]
 }
 {
-  "type": "heap-snapshot-load",
-  "snapshotID": "9e9176fbd6c60ca817af710e3e519142fe50f6ff41fb75712e15a24df89135ad"
-}
-{
   "transcriptNum": 39003,
   "d": [
     "notify",
     [
       [
         "p+40734",
         false,
         {
           "body": "{\"body\":\"{\\\"currentAmount\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDT_axl brand\\\",\\\"index\\\":0},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"0\\\"}},\\\"updated\\\":\\\"balance\\\"}\",\"slots\":[\"board0188\"]}",
           "slots": []
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "send",
         "o-5424",
         {
@@ -2899124,40 +2899176,80 @@
   "transcriptNum": 39398,
   "d": [
     "notify",
     [
       [
         "p+41135",
         false,
         {
           "body": "{\"@qclass\":\"slot\",\"iface\":\"Alleged: Virtual Purse\",\"index\":0}",
           "slots": [
             "o-5786"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "vatstoreGet",
+        "vc.762.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.762.|label"
+      ],
+      "response": [
+        "ok",
+        "brand purses"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.763.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.763.|label"
+      ],
+      "response": [
+        "ok",
+        "payments queues"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
         "vc.762.|o-77"
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "send",
         "o-77",
         {
           "methargs": {
             "body": "[\"getDisplayInfo\",[]]",
             "slots": []
           },
           "result": "p+41150"
         }
       ],
       "response": [
@@ -2900145,40 +2900237,60 @@
 {
   "transcriptNum": 39411,
   "d": [
     "notify",
     [
       [
         "p+41156",
         false,
         {
           "body": "{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDT_axl brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}",
           "slots": [
             "o-77"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.764.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.764.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.764.",
         "vc.764.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     }
   ]
 }
 {
   "transcriptNum": 39412,
   "d": [
     "notify",
     [
@@ -2906047,60 +2906159,40 @@
         "vatstoreGet",
         "vc.697.|schemata"
       ],
       "response": [
         "ok",
         "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
       ]
     },
     {
       "d": [
         "vatstoreGet",
         "vc.697.|label"
       ],
       "response": [
         "ok",
         "invitation makers"
       ]
     },
     {
       "d": [
-        "vatstoreGet",
-        "vc.696.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.696.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "send",
         "o-51",
         {
           "methargs": {
             "body": "[\"getPublicFacet\",[{\"@qclass\":\"slot\",\"iface\":\"Alleged: InstanceHandle\",\"index\":0}]]",
             "slots": [
               "o-220"
             ]
           },
           "result": "p+41253"
         }
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "subscribe",
@@ -2919728,40 +2919820,60 @@
 {
   "transcriptNum": 39709,
   "d": [
     "notify",
     [
       [
         "p+34156",
         false,
         {
           "body": "{\"updateCount\":3,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDC_axl brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"2337000000\"}}}",
           "slots": [
             "o-72"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.636.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.636.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.636.",
         "vc.636.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-4805",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[3]]",
@@ -2922616,40 +2922728,60 @@
 {
   "transcriptNum": 39755,
   "d": [
     "notify",
     [
       [
         "p+41504",
         false,
         {
           "body": "{\"updateCount\":20,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDC_axl brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}}",
           "slots": [
             "o-72"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.696.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.696.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.696.",
         "vc.696.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-5316",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[20]]",
@@ -2927404,60 +2927536,40 @@
   "transcriptNum": 39814,
   "d": [
     "notify",
     [
       [
         "p+41567",
         false,
         {
           "body": "{\"@qclass\":\"slot\",\"iface\":\"Alleged: Virtual Purse\",\"index\":0}",
           "slots": [
             "o-5839"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "vatstoreGet",
-        "vc.767.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.767.|label"
-      ],
-      "response": [
-        "ok",
-        "payments queues"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
         "vc.766.|o-74"
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "send",
         "o-74",
         {
           "methargs": {
             "body": "[\"getDisplayInfo\",[]]",
             "slots": []
           },
           "result": "p+41579"
         }
       ],
       "response": [
@@ -2928369,60 +2928481,40 @@
 {
   "transcriptNum": 39826,
   "d": [
     "notify",
     [
       [
         "p+41584",
         false,
         {
           "body": "{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDC_grv brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}",
           "slots": [
             "o-74"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.768.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.768.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.768.",
         "vc.768.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     }
   ]
 }
 {
   "transcriptNum": 39827,
   "d": [
     "notify",
     [
@@ -2929037,40 +2929129,60 @@
 {
   "transcriptNum": 39834,
   "d": [
     "notify",
     [
       [
         "p+41593",
         false,
         {
           "body": "{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDT_axl brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}",
           "slots": [
             "o-77"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.768.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.768.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.768.",
         "vc.768.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     }
   ]
 }
 {
   "transcriptNum": 39835,
   "d": [
     "notify",
     [
@@ -2929338,80 +2929450,40 @@
   "transcriptNum": 39837,
   "d": [
     "notify",
     [
       [
         "p+41588",
         false,
         {
           "body": "{\"@qclass\":\"slot\",\"iface\":\"Alleged: Virtual Purse\",\"index\":0}",
           "slots": [
             "o-5845"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "vatstoreGet",
-        "vc.766.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.766.|label"
-      ],
-      "response": [
-        "ok",
-        "brand purses"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.767.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.767.|label"
-      ],
-      "response": [
-        "ok",
-        "payments queues"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
         "vc.766.|o-84"
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "send",
         "o-84",
         {
           "methargs": {
             "body": "[\"getDisplayInfo\",[]]",
             "slots": []
           },
           "result": "p+41605"
         }
       ],
       "response": [
@@ -2935896,40 +2935968,60 @@
 {
   "transcriptNum": 39938,
   "d": [
     "notify",
     [
       [
         "p+41694",
         false,
         {
           "body": "{\"updateCount\":5,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: BLD brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"1108743\"}}}",
           "slots": [
             "o-84"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.768.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.768.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.768.",
         "vc.768.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-5848",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[5]]",
@@ -2937882,60 +2937974,40 @@
 {
   "transcriptNum": 39968,
   "d": [
     "notify",
     [
       [
         "p+41729",
         false,
         {
           "body": "{\"updateCount\":199,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: BLD brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"256247311\"}}}",
           "slots": [
             "o-84"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.508.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.508.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.508.",
         "vc.508.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-3510",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[199]]",
@@ -7387707,44 +7387779,40 @@
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "vatstoreSet",
         "idCounters",
         "{\"exportID\":2738,\"collectionID\":1374,\"promiseID\":105712}"
       ],
       "response": [
         "ok",
         null
       ]
     }
   ]
 }
 {
-  "type": "heap-snapshot-load",
-  "snapshotID": "a869bd0e70ab10564eeb4066060f2d9d53ceeab9405e341642d61e37ddb46876"
-}
-{
   "transcriptNum": 102003,
   "d": [
     "notify",
     [
       [
         "p+105710",
         false,
         {
           "body": "{\"body\":\"{\\\"brands\\\":[{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: Zoe Invitation brand\\\",\\\"index\\\":0},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"set\\\"},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: Zoe Invitation issuer\\\",\\\"index\\\":1},\\\"petname\\\":\\\"invitations\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDC_axl brand\\\",\\\"index\\\":2},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDC_axl issuer\\\",\\\"index\\\":3},\\\"petname\\\":\\\"USDC_axl\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDC_grv brand\\\",\\\"index\\\":4},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDC_grv issuer\\\",\\\"index\\\":5},\\\"petname\\\":\\\"USDC_grv\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDT_axl brand\\\",\\\"index\\\":6},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDT_axl issuer\\\",\\\"index\\\":7},\\\"petname\\\":\\\"USDT_axl\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDT_grv brand\\\",\\\"index\\\":8},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: USDT_grv issuer\\\",\\\"index\\\":9},\\\"petname\\\":\\\"USDT_grv\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: BLD brand\\\",\\\"index\\\":10},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":11},\\\"petname\\\":\\\"BLD\\\"},{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: IST brand\\\",\\\"index\\\":12},\\\"displayInfo\\\":{\\\"assetKind\\\":\\\"nat\\\",\\\"decimalPlaces\\\":6},\\\"issuer\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"iface\\\":\\\"Alleged: IST issuer\\\",\\\"index\\\":13},\\\"petname\\\":\\\"IST\\\"}],\\\"lastOfferId\\\":1668732147216,\\\"offerToUsedInvitation\\\":{},\\\"purses\\\":[{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":0},\\\"value\\\":[]},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":0}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":2},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"0\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":2}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":4},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"0\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":4}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":6},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"0\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":6}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":8},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"0\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":8}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":10},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"5509126\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":10}},{\\\"balance\\\":{\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":12},\\\"value\\\":{\\\"@qclass\\\":\\\"bigint\\\",\\\"digits\\\":\\\"290618\\\"}},\\\"brand\\\":{\\\"@qclass\\\":\\\"slot\\\",\\\"index\\\":12}}]}\",\"slots\":[\"board04016\",null,\"board0223\",\"board0592\",\"board0566\",\"board0425\",\"board0188\",\"board0257\",\"board02810\",\"board0639\",\"board04312\",null,\"board02314\",\"board05311\"]}",
           "slots": []
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "send",
         "o-9381",
         {
@@ -7439552,60 +7439620,40 @@
 {
   "transcriptNum": 102723,
   "d": [
     "notify",
     [
       [
         "p+106442",
         false,
         {
           "body": "{\"updateCount\":2,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: Zoe Invitation brand\",\"index\":0},\"value\":[]}}",
           "slots": [
             "o-63"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.1380.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.1380.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1380.",
         "vc.1380.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-13830",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[2]]",
@@ -7441805,60 +7441853,40 @@
   "transcriptNum": 102749,
   "d": [
     "notify",
     [
       [
         "p+106464",
         false,
         {
           "body": "{\"@qclass\":\"slot\",\"iface\":\"Alleged: Virtual Purse\",\"index\":0}",
           "slots": [
             "o-13836"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "vatstoreGet",
-        "vc.1378.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.1378.|label"
-      ],
-      "response": [
-        "ok",
-        "brand purses"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
         "vc.1379.|schemata"
       ],
       "response": [
         "ok",
         "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
       ]
     },
     {
       "d": [
         "vatstoreGet",
         "vc.1379.|label"
       ],
       "response": [
         "ok",
         "payments queues"
       ]
     },
     {
       "d": [
         "vatstoreGet",
@@ -7443633,60 +7443661,40 @@
 {
   "transcriptNum": 102769,
   "d": [
     "notify",
     [
       [
         "p+106494",
         false,
         {
           "body": "{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: BLD brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"38999367\"}}",
           "slots": [
             "o-84"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.1380.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.1380.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1380.",
         "vc.1380.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     }
   ]
 }
 {
   "transcriptNum": 102770,
   "d": [
     "notify",
     [
@@ -7443761,60 +7443769,40 @@
 {
   "transcriptNum": 102773,
   "d": [
     "notify",
     [
       [
         "p+106496",
         false,
         {
           "body": "{\"updateCount\":2,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: BLD brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"38999367\"}}}",
           "slots": [
             "o-84"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.1380.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.1380.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1380.",
         "vc.1380.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-13841",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[2]]",
@@ -7450150,60 +7450138,40 @@
 {
   "transcriptNum": 102873,
   "d": [
     "notify",
     [
       [
         "p+106598",
         false,
         {
           "body": "{\"updateCount\":841,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: BLD brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"312942400\"}}}",
           "slots": [
             "o-84"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.508.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.508.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.508.",
         "vc.508.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-3510",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[841]]",
@@ -7452644,60 +7452612,40 @@
 {
   "transcriptNum": 102901,
   "d": [
     "notify",
     [
       [
         "p+106628",
         false,
         {
           "body": "{\"updateCount\":2,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: Zoe Invitation brand\",\"index\":0},\"value\":[]}}",
           "slots": [
             "o-63"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.1384.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.1384.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1384.",
         "vc.1384.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-13848",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[2]]",
@@ -7456007,40 +7455955,60 @@
 {
   "transcriptNum": 102940,
   "d": [
     "notify",
     [
       [
         "p+106672",
         false,
         {
           "body": "{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDT_grv brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}",
           "slots": [
             "o-80"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.1384.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.1384.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1384.",
         "vc.1384.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     }
   ]
 }
 {
   "transcriptNum": 102941,
   "d": [
     "notify",
     [
@@ -7723636,40 +7723604,60 @@
 {
   "transcriptNum": 106778,
   "d": [
     "notify",
     [
       [
         "p+110590",
         false,
         {
           "body": "{\"updateCount\":90,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDT_grv brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}}",
           "slots": [
             "o-80"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.1096.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.1096.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1096.",
         "vc.1096.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-9393",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[90]]",
@@ -7728929,60 +7728917,40 @@
 {
   "transcriptNum": 106857,
   "d": [
     "notify",
     [
       [
         "p+110546",
         false,
         {
           "body": "{\"updateCount\":92,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDC_axl brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"7338500000\"}}}",
           "slots": [
             "o-72"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.1096.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.1096.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1096.",
         "vc.1096.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-9386",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[92]]",
@@ -7992054,60 +7992022,40 @@
 {
   "transcriptNum": 110609,
   "d": [
     "notify",
     [
       [
         "p+111978",
         false,
         {
           "body": "{\"updateCount\":196,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: IST brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"477328\"}}}",
           "slots": [
             "o-57"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.520.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.520.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.520.",
         "vc.520.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-3593",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[196]]",
@@ -7992740,60 +7992688,40 @@
 {
   "transcriptNum": 110619,
   "d": [
     "notify",
     [
       [
         "p+114494",
         false,
         {
           "body": "{\"updateCount\":162,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: IST brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"0\"}}}",
           "slots": [
             "o-57"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.256.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.256.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.256.",
         "vc.256.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-1825",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[162]]",
@@ -7995266,40 +7995194,60 @@
 {
   "transcriptNum": 110657,
   "d": [
     "notify",
     [
       [
         "p+114521",
         false,
         {
           "body": "{\"updateCount\":163,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: IST brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"115326685\"}}}",
           "slots": [
             "o-57"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.256.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.256.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.256.",
         "vc.256.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-1825",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[163]]",
@@ -8014326,40 +8014274,80 @@
   "transcriptNum": 110932,
   "d": [
     "notify",
     [
       [
         "p+114831",
         false,
         {
           "body": "{\"@qclass\":\"slot\",\"iface\":\"Alleged: Virtual Purse\",\"index\":0}",
           "slots": [
             "o-14853"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
         "vatstoreGet",
+        "vc.1398.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.1398.|label"
+      ],
+      "response": [
+        "ok",
+        "brand purses"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.1399.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.1399.|label"
+      ],
+      "response": [
+        "ok",
+        "payments queues"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
         "vc.1398.|o-74"
       ],
       "response": [
         "ok",
         null
       ]
     },
     {
       "d": [
         "send",
         "o-74",
         {
           "methargs": {
             "body": "[\"getDisplayInfo\",[]]",
             "slots": []
           },
           "result": "p+114843"
         }
       ],
       "response": [
@@ -8017494,60 +8017482,40 @@
 {
   "transcriptNum": 110969,
   "d": [
     "notify",
     [
       [
         "p+114877",
         false,
         {
           "body": "{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: IST brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"250000\"}}",
           "slots": [
             "o-57"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.1400.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.1400.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1400.",
         "vc.1400.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     }
   ]
 }
 {
   "transcriptNum": 110970,
   "d": [
     "notify",
     [
@@ -8017568,60 +8017536,40 @@
 {
   "transcriptNum": 110971,
   "d": [
     "notify",
     [
       [
         "p+114876",
         false,
         {
           "body": "{\"updateCount\":2,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: BLD brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"2306178\"}}}",
           "slots": [
             "o-84"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
-        "vatstoreGet",
-        "vc.1400.|schemata"
-      ],
-      "response": [
-        "ok",
-        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
-      ]
-    },
-    {
-      "d": [
-        "vatstoreGet",
-        "vc.1400.|label"
-      ],
-      "response": [
-        "ok",
-        "invitation amounts"
-      ]
-    },
-    {
-      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1400.",
         "vc.1400.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-14862",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[2]]",
@@ -8058319,40 +8058267,60 @@
 {
   "transcriptNum": 111551,
   "d": [
     "notify",
     [
       [
         "p+115437",
         false,
         {
           "body": "{\"updateCount\":110,\"value\":{\"brand\":{\"@qclass\":\"slot\",\"iface\":\"Alleged: USDC_axl brand\",\"index\":0},\"value\":{\"@qclass\":\"bigint\",\"digits\":\"1000000000\"}}}",
           "slots": [
             "o-72"
           ]
         }
       ]
     ]
   ],
   "syscalls": [
     {
       "d": [
+        "vatstoreGet",
+        "vc.1096.|schemata"
+      ],
+      "response": [
+        "ok",
+        "{\"body\":\"[{\\\"@qclass\\\":\\\"tagged\\\",\\\"tag\\\":\\\"match:scalar\\\",\\\"payload\\\":{\\\"@qclass\\\":\\\"undefined\\\"}}]\",\"slots\":[]}"
+      ]
+    },
+    {
+      "d": [
+        "vatstoreGet",
+        "vc.1096.|label"
+      ],
+      "response": [
+        "ok",
+        "invitation amounts"
+      ]
+    },
+    {
+      "d": [
         "vatstoreGetAfter",
         "",
         "vc.1096.",
         "vc.1096.{"
       ],
       "response": [
         "ok",
         [
           null,
           null
         ]
       ]
     },
     {
       "d": [
         "send",
         "o-9386",
         {
           "methargs": {
             "body": "[\"getUpdateSince\",[110]]",

Bundles needed for transcript replay (must gunzip):

Details of transcript snapshot activity:

Snapshots used for reload:

Divergent snapshots:

Other potentially relevant snapshots:

@AlexDCryptoC
Copy link

I fetched into the same issue with stake2earn validator after restart. I had to re-sync from a snapshot.

@warner
Copy link
Member

warner commented Dec 6, 2022

We got some updates from Moddable, and they were able to observe a divergence in the number of slots released during an organic GC mark+sweep, somewhat earlier than the syscall divergence that triggers the anachrophobia error. Our best hypothesis so far is an unintentional sensitivity to memory layout (something doing pointer comparisons), such that e.g. ASLR triggers different behavior. @mhofman is trying to reproduce their example, and we may dive into things like changing malloc() to return (controllable) pseudorandom pointers to investigate further.

@mhofman
Copy link
Member

mhofman commented Dec 9, 2022 •
edited
Loading

Moddable identified an un-initialized memory issue which can happen in a certain edge case, and provided a tentative fix.

While I have verified that the fix removed the observed divergences in the deliveries and snapshots around 32k-40k, the issues for deliveries after 102k remain.

The remaining issue(s) seem similar, with apparently some objects being collected at a slightly different schedule.

Given that it takes over an hour to replay 100k deliveries (especially given the slowdown in execution we're seeing, some highlighted in the #6625 investigation), I worked on updating the replay tool to find the earliest snapshot from which reloading will cause this issue, and handle the execution divergences when matching syscalls to the transcript, which ends up making it easier to reproduce this case.

  • I manually generated an hybrid transcript which contains the combination of all syscalls needed for both divergent executions (basically adding vatstoreGet("vc.*.\|(schemata|label)") syscalls where needed), and I updated the replay tool to skip over these extra syscalls if the worker doesn't make them.
  • I also added the ability for the replay tool to keep multiple workers and make deliveries to them in parallel (but in lockstep). When loading from a snapshot, it can keep previous workers, allowing to more directly compare the executions.

The observed divergent executions are:

  • Behavior 1: when replaying from scratch or any snapshot up to and including the one generated at delivery 69002 (342145d31a9cd3d9e9f29f294eacbb0a7bedd287c0752109fb8ec2894e00014f)
  • Behavior 2: when replaying from any snapshot generated from delivery 70002 (b1d21f0c68f61fc4d9f0f3fbb9017942328935c5559405529f5ef13f2bc6d2f9) up to and including the one generated at delivery 86002 (5fb5aea2e244c2358073724b299cfd466bdadfe0a0afe777b0d3c00bdce5ea2d)
  • Behavior 3: when replaying from the snapshot generated at delivery 87002 (521fb8df7759cd582bd4ab55446ea6c7adcb57cc91266239a057b96016aa1006) or any snapshot generated after, such as the one generated at delivery 102002 (a869bd0e70ab10564eeb4066060f2d9d53ceeab9405e341642d61e37ddb46876), or all divergent snapshots generated at delivery 108002, 111002 or 114002.

The following table shows which behavior need to issue syscalls to page back in the virtual collection metadata that had likely been evicted from memory by a preceding GC. Since the transcript contains all possible syscalls for all these deliveries, the empty cells (no vc syscalls performed) results in "anachrophobia" errors printed for those workers, but the replay tool then skips over the extra syscalls recorded in the transcript, and proceeds forward.
For the divergent snapshots, the table shows which behavior generated which snapshot.

deliveryNum Behavior 1 (0 -> 69002) Behavior 2 (70002 -> 86002) Behavior 3 (87002 -> 117002+)
102723 1380
102749 1378
102769 1380
102773 1380
102873 508
102901 1384
102940 1384 1384
106778 1096 1096
106857 1096
108002 snapshot 22c0a6c3 snapshot 22c0a6c3 snapshot fa8c694f
110609 520 520
110657 256
110932 1398 & 1399
110969 1400 1400
110971 1400 1400
111002 snapshot 080f6ef6 snapshot 8910177b snapshot 8910177b
111551 1096
112666 1410 & 1411
112680 1412
112682 1412 1412
112683 1412
112743 1412
112763 1412
112811 696 696
112828 696
112936 1096
113435 1096
114002 snapshot 72bedc64 snapshot 148779ad snapshot 148779ad
114671 440 440
114706 440 440
114751 440
115002 snapshot a948e8f0 snapshot 86b90ec8 snapshot 86b90ec8

The behaviors then apparently stop diverging again until end of transcript at deliveryNum 117129

Needed files:

To cut down on replay time, since I've validated that replaying from scratch or from the snapshot at delivery 69002 results in identical behavior:

cat transcript-v18-hybrid-with-loads.sst | tail -n +69004 > transcript-v18-hybrid-with-loads-tail-69k.sst

Or to cut down replay time even further and focus on the divergences between behavior 2 and 3:

cat transcript-v18-hybrid-with-loads.sst | tail -n +86006 > transcript-v18-hybrid-with-loads-tail-86k.sst

Then use the replay-transcript.js tool from the updated mhofman/6588-diagnose branch.

node path/to/agoric-sdk/packages/SwingSet/misc-tools/replay-transcript.sh transcript-v18-hybrid-with-loads-tail-69k.sst

(output-69k-replay.log.gz)

Edit: I had originally only confirmed that execution from scratch and from snapshot for deliveryNum 86002 matched until delivery 108002 (the diverging snapshots). When checking if it would proceed further, I uncovered a further diverging execution. The post has now been updated to detail the 3 diverging executions.

Edit 2: I had also missed the further diverging snapshots, which might provide insight as to what the divergence cause might be.

Edit 3: A more up to date transcript of vat18: transcript-v18-b7822517.sst.gz

@mhofman mhofman mentioned this issue Dec 11, 2022
Open
@warner
Copy link
Member

warner commented Dec 12, 2022

Current plan:

  • we know of at least one uninitialized-memory bug in XS (with a fix), we're running fuzzers to find more
  • as far as we can tell, the only consequence of the bug is to change the behavior of GC, specifically the number of slots released (which changes the timing of subsequent organice GC sweeps)
    • more significant changes would probably affect syscall behavior in more drastic ways, which would probably cause other forms of consensus failure, and we haven't received any reports of those
  • the timing of GC can turn into a change in the timing of virtual-collection vatstoreGet syscalls, see syscall sensitivity to GC of virtual collection objects and timing of organic GC #6360
  • when the validator is rebooted, all workers are reloaded. (this also happens without a reboot if a vat got paged out and then back in again, but we don't yet have enough vats to cause page-outs)
  • when a worker is reloaded, it replays the "transcript", which is the list of deliveries made since the last heap snapshot was recorded
    • the transcript includes both deliveries and the syscalls made during each delivery
    • the replay code insists that the syscalls made during replay match those recorded during the original execution
    • if a syscall.vatstoreGet happens at the wrong time, the transcript comparison will fail: either the vatstoreGet happens too early (and doesn't match whatever other syscall occurred at that time), or too late (and the next syscall is flagged as not matching the expected vatstoreGet). Another failure mode is that no syscalls happened, when the transcript said to expect a vatstoreGet
  • @mhofman is implementing a workaround which relaxes the transcript-replay syscall-comparison rules:
    • if the syscall is a vatstoreGet which references the kinds of keys used by virtual collections, then allow it to not match the transcript, and execute it for real (against the current state of the DB)
    • one key will be vc.${collectionID}.|schemata, the other will be vc.${collectionID}.|label
    • the values are constant, created at the same time we created table
      • so it should be safe to read them directly from the DB, rather than a transcript record: they'll always return the same value
    • this will effectively hide the consequences of the syscall sensitivity to GC of virtual collection objects and timing of organic GC #6360 bug for the pismo release
  • we're planning to test this for a few days and then make a pismoB release as a "soft patch": one which can be applied (or not) by each validator independently (rather than a "hard patch" which must be applied by everyone at the same time, which would be triggered by a governance action)
  • we believe that using this pismoB release on a validator which is failing to restart (with the "anachrophobia error" described above) will allow those validators to relaunch successfully
    • note that these validators will not have exactly the same internal (RAM) state as everyone else, nor the same transcripts
    • but if the XS bug is indeed limited to affecting GC timing, then there should be no other visible consequences of the divergence

@mhofman mhofman mentioned this issue Dec 13, 2022
Merged
@mhofman
Copy link
Member

mhofman commented Dec 13, 2022

  • @mhofman is implementing a workaround which relaxes the transcript-replay syscall-comparison rules

PR #6664

  • note that these validators will not have exactly the same internal (RAM) state as everyone else, nor the same transcripts

This is already the case today for validators with different restart schedule, and I don't expect this hot fix to change that.

@mhofman mhofman mentioned this issue Dec 13, 2022
Closed
@mhofman
Copy link
Member

mhofman commented Dec 13, 2022 •
edited
Loading

I have now 2 different node operators that have confirmed the patch of the #6664 PR fixes their inability to restart due to this anachrophobia error.

For reference, in agoric-sdk:

curl https://patch-diff.githubusercontent.com/raw/Agoric/agoric-sdk/pull/6664.diff | patch -p1

@dckc dckc added the xsnap the XS execution tool label Dec 13, 2022
mhofman added a commit that referenced this issue Dec 13, 2022
@mhofman
fix(swingset): workaround XS garbage collection bugs
b2b336f 
DO NOT MERGE INTO MAIN BRANCH
Workaround for #6588

During transcript replay, handle divergent syscalls which retrieve
stable Virtual Collection metadata. These can happen at different times
than recorded during the transcript because of some bugs in the XS
engine making GC sensitive to reload from heap snapshots.
mhofman added a commit that referenced this issue Dec 14, 2022
@mhofman
fix(swingset): workaround XS garbage collection bugs (#6664)
12e97f1 
DO NOT MERGE INTO MAIN BRANCH
Workaround for #6588

During transcript replay, handle divergent syscalls which retrieve
stable Virtual Collection metadata. These can happen at different times
than recorded during the transcript because of some bugs in the XS
engine making GC sensitive to reload from heap snapshots.
@mhofman
Copy link
Member

mhofman commented Dec 15, 2022 •
edited
Loading

vat6 is also subject to a divergent execution observable as an anachrophobia error at deliveryNum 44497. Thankfully it seems that the divergence happens between delivery 43002 and 44002, so it only takes a few seconds to reach.

The following archive with the 43002 snapshot and the transcript slice for the deliveries until the anachrophobia can be used with the replay-transcript.js tool from the updated mhofman/6588-diagnose branch:

For reference, here is the full vat6 transcript (double compressed to satisfy GitHub restrictions):

This was referenced Dec 27, 2022
Merged
Draft
@rowgraus rowgraus added the vaults_triage DO NOT USE label Jan 3, 2023
@mhofman mhofman mentioned this issue Jan 6, 2023
Merged
@mhofman mhofman linked a pull request Jan 6, 2023 that will close this issue
fix(xsnap): GC divergence fixes #6758
Merged
@mhofman mhofman mentioned this issue Jan 6, 2023
Closed
@mhofman mhofman mentioned this issue Jan 13, 2023
Open
mhofman added a commit that referenced this issue Jan 25, 2023
@mhofman
fix(swingset): workaround XS garbage collection bugs (#6664)
9050681 
Workaround for #6588

During transcript replay, handle divergent syscalls which retrieve
stable Virtual Collection metadata. These can happen at different times
than recorded during the transcript because of some bugs in the XS
engine making GC sensitive to reload from heap snapshots.

cherry picked from commit 12e97f1
Merging into master after further consideration
@warner warner mentioned this issue Jan 30, 2023
Open
@mhofman mhofman mentioned this issue Feb 8, 2023
Closed
@mhofman
Copy link
Member

mhofman commented Mar 14, 2023 •
edited by dckc
Loading

This has now be released on mainnet as part of the agoric-upgrade-9 release aka pismoC aka 636c850

@mhofman mhofman closed this as completed Mar 14, 2023
@mhofman
Copy link
Member

mhofman commented Jun 12, 2023

I initially believed to have noticed more divergences in the mainnet transcript of v18, but they happen to have occurred before pismoC was deployed.

For future reference, pismoC kicked in at block height 8941749 on mainnet. The first delivery to v18 after that is deliveryNum 359736. Any divergence prior to this delivery is expected. No divergence is expected after, and none has been observed so far.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mhofman mhofman

Labels
bug Something isn't working SwingSet package: SwingSet vaults_triage DO NOT USE xsnap the XS execution tool
Projects
None yet
Milestone
Vaults EVP
Development

Successfully merging a pull request may close this issue.

fix(xsnap): GC divergence fixes Agoric/agoric-sdk
7 participants
@warner @mhofman @dckc @ivanlei @shivlim @AlexDCryptoC @rowgraus