-
Notifications
You must be signed in to change notification settings - Fork 285
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf(core-database-postgres): faster count method #2832
Conversation
I am afraid that there is no silver bullet for the problem with the slow
|
@@ -103,8 +103,8 @@ export abstract class Repository implements Database.IRepository { | |||
return { rows, count: Math.max(count, rows.length), countIsEstimate: true }; | |||
} | |||
|
|||
const countRow = await this.find(selectQueryCount); | |||
const { count } = await this.db.one(`SELECT COUNT(*) FROM (${selectQueryCount.toString()}) AS count;`); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you are going to do SELECT COUNT(*) FROM (... subquery ...)
then you do not need a separate selectQueryCount
variable and can use selectQuery
for this. But I specifically did not do that when I introduced selectQueryCount
because the straight-forward SELECT COUNT(*) FROM table WHERE ...
is to be preferred over the convoluted SELECT COUNT(*) FROM (SELECT ... FROM table WHERE ...) AS count;
because with the latter you are asking the database to do more stuff and depend on it being smart enough to detect that both queries are the same and actually do the first one under the hood. In other words, the variant with the subquery could also be slower!
Codecov Report
@@ Coverage Diff @@
## master #2832 +/- ##
=======================================
Coverage 65.33% 65.33%
=======================================
Files 363 363
Lines 8193 8193
Branches 393 393
=======================================
Hits 5353 5353
Misses 2806 2806
Partials 34 34
Continue to review full report at Codecov.
|
Given that the impact varies greatly between nodes, I'll close this PR. |
Use a faster query that performs better against larger datasets like on mainnet while the benefit on devnet is still visible but a lot smaller as the database is 1/3 the size. This is obviously not a permanent solution but a nice little improvement for now.
EXPLAIN ANALYZE SELECT COUNT(id) FROM blocks;
EXPLAIN ANALYZE SELECT COUNT(*) FROM (SELECT id FROM blocks) AS count;
Tested on a 4 CPU / 16 GB RAM / 160GB NVME server so results will vary between servers.
What kind of change does this PR introduce?
Does this PR introduce a breaking change?
Does this PR release a new version?
If yes, please describe the impact and migration path for existing applications:
The PR fulfills these requirements:
develop
branch, not themaster
branchIf adding a new feature, the PR's description includes: