nodeapi: Store raw oasis-core in serialized form #382
Conversation
analyzer/consensus/consensus.go
Outdated
| @@ -482,7 +482,7 @@ func (m *Main) queueTxEventInserts(batch *storage.QueryBatch, data *storage.Cons | |||
| eventData := m.extractEventData(event) | |||
| txAccounts = append(txAccounts, eventData.relatedAddresses...) | |||
| accounts := extractUniqueAddresses(eventData.relatedAddresses) | |||
| body, err := json.Marshal(eventData.body) | |||
| body, err := json.Marshal(eventData.rawBodyJSON) | |||
There was a problem hiding this comment.
we should either (a) change rawBodyJSON to []byte and remove json.Marshal or (b) rename it to rawBody. otherwise marshaling something that's already labeled as "JSON" seems kind of weird
There was a problem hiding this comment.
Well json.RawMessage is defined as []byte, so we kinda already had (a).
I went with (b) now.
common/types.go
Outdated
| func TryAsJSON(v interface{}) json.RawMessage { | ||
| encoded, err := json.Marshal(v) | ||
| if err != nil { | ||
| return json.RawMessage("null") |
There was a problem hiding this comment.
is this what would have happened in the db if pgx failed to marshal the event?
There was a problem hiding this comment.
I believe not exactly; in the old setup, if pgx had failed, we would have failed the whole block analysis.
However
- failure to serialize is very unlikely (we're just re-serializing plain structs that were previously cbor-serialized by the node)
- a missing value here is low impact (people just won't be able to see the raw struct of an event)
- all the verbose error handling would make the rest of the code harder to read
So I went with this.
There was a problem hiding this comment.
dang I wish we had globally accessible logging
mitjat
force-pushed
the
mitjat/raw-event-body-as-json
branch
from
2321773 to
4002e02
Compare
|
💀 #360 merged part of this. rebase the second commit |
mitjat
force-pushed
the
mitjat/raw-event-body-as-json
branch
from
4002e02 to
20de4d8
Compare
Background: The indexer-internal
nodeapi.Eventtype that represents an oasis-core consensus event has aBodyfield. It stores the raw body of the Event as received from oasis-core, but intentionally erases the type (by using theinterface{}type) so that the rest of the indexer doesn't need to support all the variants of Event types from all the oasis-core versions. We just dump the raw body into the DB as json, for anybody who might want to inspect it later.Problem: #360 is introducing a caching layer, as part of which all oasis-core responses (=
nodeapitypes) are (de)serialized to/from the disk. Theinterface{}type does not round-trip losslessly and throws errors when deserialzing back from on-disk JSON or CBOR.Solution / This PR: This PR changes the
nodeapirepresentation of the event to store a serialized version (JSON) of the oasis-core Event, not just a type-erased version (interface{}). Instead of JSON, we could choose CBOR; CBOR makes more sense in that it's used everywhere internally. But since the goal of this field is to store it in the DB (which offers native JSON columns) and then present it via HTTP API (again JSON), I'm going with JSON.Note on runtimes: The runtime part of
nodeapihas no such issue because the event types have been stable across all versions of the sdk, so thenodeapiEvent representation does not erase the types and uses the current oasis-sdk types. At some point when types change enough, we might well need to introduce a similar solution there.Testing: Indexed 100 blocks of consensus and compared DB table contents. No diffs.