Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

db: schema oasis_3 -> chain #341

Merged
merged 4 commits into from
Mar 7, 2023
Merged

db: schema oasis_3 -> chain #341

merged 4 commits into from
Mar 7, 2023

Conversation

mitjat
Copy link
Contributor

@mitjat mitjat commented Mar 3, 2023

Task description

The DB was originally designed with separate tables in mind for each successive mainnet chain (oasis-2, oasis-3, oasis-4, etc; and date(?)-based names in testnet). Later, we made the decision to maintain a single schema and conform all chains' data to it.

We're now bringing in oasis-2 data, and soon oasis-4. It doesn't make sense to call the DB schema "oasis_3", so let's call it mainnet.

From a technical perspective, the name is arbitrary (= not derived from some existing config or property), so it will need to be specified in the initial config, replacing the current `chain_id: oasis–3`  

Also rename stats -> mainnet_stats. interchain schema: think about it.

What this PR does
Reorganizes the DB as described in the ticket:
- renames the oasis_3 schema to chain
- moves the processed_geneses table from multichain schema (which only contained this one table) to just chain; multichain doesn't make sense any more now that we're treating the sequence of oasis-3, oasis-4 etc as a single chain

Testing
Analysis:
I ran emerald from block 1_003_298 to 1_029_386 and consensus from block 8_048_956 to 8_058_462. EVM token analyzer and aggregate_stats analyzer were running in parallel at a 10min interval. No error or warning level logs were emitted.

API:
I ran e2e_regressions; consensus and emerald ran for 100 blocks. None of the tested URLs found any differences.

Statecheck:
Ran it with OASIS_GENESIS_DUMP=$PWD/cache/[email protected] HEALTHCHECK_TEST_CONN_STRING='postgresql://rwuser:password@localhost:5432/indexer?sslmode=disable' HEALTHCHECK_TEST _CHAIN_CONTEXT=b11b369e0da5bb230b220127f5e7b242d385ef8c6f54906243f30af63c815535 HEALTHCHECK_TEST_NODE_RPC=unix:/tmp/node.sock OASIS_INDEXER_HEALTHCHECK=1 go test -v -run TestGenesisFull github.com/oasisprotocol/oasis-indexer/tests/ statecheck. Test partially fails functionally (detects some node registry discrepancies, as is common) but the technical SQL-accessing part shows no errors.

Discussion from slack:

About renaming oasis_3 to mainnet: I started doing that but also thinking what it means for testnet. Migrations all hardcode oasis_3, soon to be mainnet. So we'd need copy-pasted migrations for `testnet table), or a way to templatize migrations. I actually implemented 50% of a templating solution.

If we go the templatization route:

  • [con] What about tables that are not mainnet or testnet specific? Their migrations would run twice (once with the template instantiated for mainnet, once for testnet; but the resulting SQL for those tables would be identical), which might cause problems. OTOH it doesn't seem likely we'll have such tables. We don't currently. Still, we're kind-of reinventing another layer of namespacing. It feels testnet should just be a separate db.

  • [con?] When we do any sketchy upgrades etc, db backups might make more sense to do for just testnet or just mainnet.

  • [con?] Combined testnet+mainnet load on the same db. Shouldn't be a problem in practice though unless both are doing a full reindex simultaneously.

  • [pro] Implementing the templatization mentioned above does not prevent us from introducing a whole separate DB for testnet.

OTOH if we do have testnet in a separate DB:

  • [con] More ops work, for the devops team but also for us (helping get everything set up, maintenance, deployments).

  • [pro] We could simplify all our queries to use hardcoded schemas (e.g. chain.blocks or even blocks if we use the default schema), as opposed to %[1]s.blocks

Opinions? For now, I'm moving forward with templatization because a) it's in line with what we already discussed and b) I prefer to manage complexity on the Go side than on the ops side. Easier to understand, test, reproduce locally, etc.

Update:

@pro-wh and @Andrew7234 both weighed towards separate DBs, saying it shouldn't be too hard on the ops side (we do have separate prod/staging deploys already) and that templating messes up coding/IDEs. So I'm renaming oasis_3 to chain instead.

@mitjat mitjat force-pushed the mitjat/db-schema-no-oasis3 branch from f780985 to 4873c5f Compare March 3, 2023 22:27
@mitjat mitjat force-pushed the mitjat/db-schema-no-oasis3 branch from 4873c5f to cfe536d Compare March 3, 2023 22:40
@mitjat mitjat changed the base branch from main to mitjat/db-refactor-prework March 3, 2023 22:44
@mitjat mitjat changed the title refactor: Remove QueryFactory. Rename DB schema oasis_3 -> chain db: schema oasis_3 -> chain Mar 3, 2023
storage/migrations/01_consensus.up.sql Show resolved Hide resolved
analyzer/consensus/genesis.go Show resolved Hide resolved
@mitjat mitjat force-pushed the mitjat/db-refactor-prework branch from fc9b47d to 81ccddd Compare March 7, 2023 18:54
@mitjat mitjat force-pushed the mitjat/db-schema-no-oasis3 branch from 6f288b5 to c9a1636 Compare March 7, 2023 18:54
Base automatically changed from mitjat/db-refactor-prework to main March 7, 2023 19:04
@mitjat mitjat force-pushed the mitjat/db-schema-no-oasis3 branch from c9a1636 to 77866be Compare March 7, 2023 19:08
@mitjat mitjat enabled auto-merge March 7, 2023 19:08
@mitjat mitjat merged commit 4dd56a7 into main Mar 7, 2023
@mitjat mitjat deleted the mitjat/db-schema-no-oasis3 branch March 7, 2023 19:13
@csillag csillag mentioned this pull request Dec 19, 2024
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants