Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consensus: use batch for genesis queries #630

Merged
merged 2 commits into from
May 6, 2024
Merged

Conversation

pro-wh
Copy link
Collaborator

@pro-wh pro-wh commented Feb 8, 2024

this kinda sucks

  • db slower ~13s -> ~17s on eden genesis file
  • can no longer print out queries into a migration-like file

also some of the ON CONFLICT is not exactly preserved, so might be some hidden bugs in network upgrades

measurements, very informal, only one run each

reuse
{"analyzer":"consensus","caller":"genesis.go:52","count":355437,"height":"genesis","level":"info","module":"analysis_service","msg":"generated genesis queries","ts":"2024-02-08T01:42:57.758442953Z"}
{"analyzer":"consensus","caller":"consensus.go:204","level":"info","module":"analysis_service","msg":"genesis document queries generated","process_duration":"32.732929903s","ts":"2024-02-08T01:42:57.758504654Z"}
{"analyzer":"consensus","caller":"consensus.go:228","level":"info","module":"analysis_service","msg":"genesis document processed","send_batch_duration":"17.026809672s","ts":"2024-02-08T01:43:14.785349655Z"}

control
{"analyzer":"consensus","caller":"genesis.go:56","count":369,"height":"genesis","level":"info","module":"analysis_service","msg":"generated genesis queries","ts":"2024-02-08T02:30:31.547038915Z"}
{"analyzer":"consensus","caller":"consensus.go:204","level":"info","module":"analysis_service","msg":"genesis document queries generated","process_duration":"43.796619503s","ts":"2024-02-08T02:30:31.547102908Z"}
{"analyzer":"consensus","caller":"consensus.go:229","level":"info","module":"analysis_service","msg":"genesis document processed","send_batch_duration":"13.060895665s","ts":"2024-02-08T02:30:44.608061096Z"}

@pro-wh pro-wh added the analysis-layer Analysis layer-related issues. label Feb 8, 2024
@pro-wh
Copy link
Collaborator Author

pro-wh commented Feb 8, 2024

oh and hide whitespace on the diff due to some indentation decrease

Copy link
Contributor

@mitjat mitjat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, I see the dilemma.

  • The 13->17s does suck for local development.
  • The dumpable queries ... I don't care for particularly. They can be post-processed pretty easily to show in a readable format, IMO. Sth like output the query template only if it differs from the previous query, otherwise only output the args. Not that we should implement that now; it's just that whoever needs to debug genesis queries that deeply (I haven't, in recent memory) can hack together some insight easily enough.
  • The code is so much cleaner, and more resilient to future DB changes.

I vote to check this in. I can wait the extra 4 seconds (times 30 iterations) next time I'm working on something that requires genesis processing.

And it can be sped up if needed:

  • The queries are absolutely dominated by account balances (~280k) and staking delegations (~70); there's <10k other rows. So we only really need to speed the first two, or even just the balances.
  • The data types involved are simple, so using the COPY protocol like so should be simple too.
    • The caveat is that COPY cannot do upserts. We do need upserts because we apply the genesis after fast sync. So you'd need to COPY into a temp table, and then do a INSERT FROM mytemptable INTO accounts ON CONFLICT ....
    • The exact implementation is a little messy too, but it's not super bad. We'd need to either pass the db connection into genesis.go (inversion of control from what we have now; I'm not a fan), or Process() in genesis.go would need to return a triple: the [][]interface{} for balances, the [][]interface{} for delegations, and the batch. The caller would then have to make both CopyFrom() calls plus exec the batch (which would contain commands for upserting from the temp tables).
  • Conveniently, those are also the only two queries you needed to newly add into queries.go. (That's mostly irrelevant, except emotionally :))

Whether you want to take a stab at that is up to you. Even if not, I think this PR is a step in the right direction. Nice cleanup, thank you.

analyzer/consensus/genesis.go Show resolved Hide resolved
analyzer/consensus/genesis.go Show resolved Hide resolved
Base automatically changed from pro-wh/bugfix/genesissep to main February 8, 2024 18:27
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch from 740b9f7 to 65296a8 Compare February 8, 2024 20:32
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch 2 times, most recently from ea61d0b to b0c1a87 Compare February 22, 2024 00:41
@pro-wh pro-wh changed the base branch from main to pro-wh/feature/roothash3 February 22, 2024 00:42
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch from b0c1a87 to 92aa008 Compare February 22, 2024 00:44
@pro-wh pro-wh force-pushed the pro-wh/feature/roothash3 branch from 2446d19 to fca8b83 Compare February 23, 2024 01:29
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch from 92aa008 to 7af6e38 Compare February 23, 2024 01:29
@pro-wh pro-wh force-pushed the pro-wh/feature/roothash3 branch from fca8b83 to 56baec8 Compare March 1, 2024 00:07
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch from 7af6e38 to faf3cbd Compare March 1, 2024 00:07
@pro-wh pro-wh force-pushed the pro-wh/feature/roothash3 branch from 56baec8 to c7b6c03 Compare March 4, 2024 21:15
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch 2 times, most recently from f67bc7b to 943a172 Compare March 6, 2024 00:36
@pro-wh pro-wh force-pushed the pro-wh/feature/roothash3 branch from c7b6c03 to 5c3324f Compare March 6, 2024 00:36
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch from 943a172 to 8457396 Compare March 12, 2024 23:34
@pro-wh pro-wh force-pushed the pro-wh/feature/roothash3 branch 2 times, most recently from fba7669 to 8b046c5 Compare March 15, 2024 22:22
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch 2 times, most recently from b44e84b to 2487e50 Compare March 21, 2024 22:53
@pro-wh pro-wh force-pushed the pro-wh/feature/roothash3 branch 2 times, most recently from 32eb535 to ff220a6 Compare March 26, 2024 23:00
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch 2 times, most recently from 944e7a8 to 24cbe24 Compare March 27, 2024 23:41
@pro-wh pro-wh force-pushed the pro-wh/feature/roothash3 branch 2 times, most recently from 65790ae to 12d77ef Compare March 29, 2024 22:53
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch 2 times, most recently from 8a8fbb1 to 2f89492 Compare April 5, 2024 21:39
@pro-wh pro-wh force-pushed the pro-wh/feature/roothash3 branch 2 times, most recently from cf1a164 to af96146 Compare April 10, 2024 22:42
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch from 2f89492 to b68a82a Compare April 10, 2024 22:42
@pro-wh pro-wh force-pushed the pro-wh/feature/roothash3 branch 3 times, most recently from 7cb42d7 to c79a2c2 Compare April 15, 2024 21:15
Base automatically changed from pro-wh/feature/roothash3 to main April 18, 2024 20:51
@pro-wh pro-wh force-pushed the pro-wh/feature/genesisbatch branch from b68a82a to 96eba3f Compare May 6, 2024 20:44
@pro-wh
Copy link
Collaborator Author

pro-wh commented May 6, 2024

let's go with how this is now

@pro-wh pro-wh merged commit b2637a3 into main May 6, 2024
14 checks passed
@pro-wh pro-wh deleted the pro-wh/feature/genesisbatch branch May 6, 2024 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
analysis-layer Analysis layer-related issues.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants