Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Task description
The DB was originally designed with separate tables in mind for each successive mainnet chain (oasis-2, oasis-3, oasis-4, etc; and date(?)-based names in testnet). Later, we made the decision to maintain a single schema and conform all chains' data to it.
We're now bringing in oasis-2 data, and soon oasis-4. It doesn't make sense to call the DB schema "oasis_3", so let's call it mainnet.
From a technical perspective, the name is arbitrary (= not derived from some existing config or property), so it will need to be specified in the initial config, replacing the current `chain_id: oasis–3`
Also rename stats -> mainnet_stats.
interchain
schema: think about it.What this PR does
Reorganizes the DB as described in the ticket:
- renames the
oasis_3
schema tochain
- moves the
processed_geneses
table frommultichain
schema (which only contained this one table) to justchain
;multichain
doesn't make sense any more now that we're treating the sequence of oasis-3, oasis-4 etc as a single chainTesting
Analysis:
I ran emerald from block 1_003_298 to 1_029_386 and consensus from block 8_048_956 to 8_058_462. EVM token analyzer and aggregate_stats analyzer were running in parallel at a 10min interval. No
error
orwarning
level logs were emitted.API:
I ran e2e_regressions; consensus and emerald ran for 100 blocks. None of the tested URLs found any differences.
Statecheck:
Ran it with
OASIS_GENESIS_DUMP=$PWD/cache/[email protected] HEALTHCHECK_TEST_CONN_STRING='postgresql://rwuser:password@localhost:5432/indexer?sslmode=disable' HEALTHCHECK_TEST _CHAIN_CONTEXT=b11b369e0da5bb230b220127f5e7b242d385ef8c6f54906243f30af63c815535 HEALTHCHECK_TEST_NODE_RPC=unix:/tmp/node.sock OASIS_INDEXER_HEALTHCHECK=1 go test -v -run TestGenesisFull github.com/oasisprotocol/oasis-indexer/tests/ statecheck
. Test partially fails functionally (detects some node registry discrepancies, as is common) but the technical SQL-accessing part shows no errors.Discussion from slack:
About renaming oasis_3 to mainnet: I started doing that but also thinking what it means for testnet. Migrations all hardcode oasis_3, soon to be mainnet. So we'd need copy-pasted migrations for `testnet table), or a way to templatize migrations. I actually implemented 50% of a templating solution.
If we go the templatization route:
[con] What about tables that are not mainnet or testnet specific? Their migrations would run twice (once with the template instantiated for mainnet, once for testnet; but the resulting SQL for those tables would be identical), which might cause problems. OTOH it doesn't seem likely we'll have such tables. We don't currently. Still, we're kind-of reinventing another layer of namespacing. It feels testnet should just be a separate db.
[con?] When we do any sketchy upgrades etc, db backups might make more sense to do for just testnet or just mainnet.
[con?] Combined testnet+mainnet load on the same db. Shouldn't be a problem in practice though unless both are doing a full reindex simultaneously.
[pro] Implementing the templatization mentioned above does not prevent us from introducing a whole separate DB for testnet.
OTOH if we do have testnet in a separate DB:
[con] More ops work, for the devops team but also for us (helping get everything set up, maintenance, deployments).
[pro] We could simplify all our queries to use hardcoded schemas (e.g.
chain.blocks
or evenblocks
if we use the default schema), as opposed to%[1]s.blocks
Opinions? For now, I'm moving forward with templatization because a) it's in line with what we already discussed and b) I prefer to manage complexity on the Go side than on the ops side. Easier to understand, test, reproduce locally, etc.
Update:
@pro-wh and @Andrew7234 both weighed towards separate DBs, saying it shouldn't be too hard on the ops side (we do have separate prod/staging deploys already) and that templating messes up coding/IDEs. So I'm renaming
oasis_3
tochain
instead.