This repository has been archived by the owner on Feb 8, 2018. It is now read-only.
Homepage updater fixes #1605
Closed
Homepage updater fixes #1605
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The explain for the original query to get new participants:
QUERY PLAN
------------------------------------------------------------------------------------------------------------
Limit (cost=13791.88..13791.91 rows=12 width=19)
-> Sort (cost=13791.88..13843.58 rows=20683 width=19)
Sort Key: p.claimed_time
-> Unique (cost=13007.48..13110.89 rows=20683 width=19)
-> Sort (cost=13007.48..13059.18 rows=20683 width=19)
Sort Key: p.username
-> Hash Join (cost=1716.85..11524.90 rows=20683 width=19)
Hash Cond: (e.participant = p.username)
-> Seq Scan on elsewhere e (cost=0.00..8570.72 rows=48972 width=11)
-> Hash (cost=1336.11..1336.11 rows=20699 width=19)
-> Seq Scan on participants p (cost=0.00..1336.11 rows=20699 width=19)
Filter: ((claimed_time IS NOT NULL) AND (is_suspicious IS NOT TRUE))
The explain for the updated query with the added index:
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------
Limit (cost=0.00..27.86 rows=12 width=19)
-> Nested Loop Semi Join (cost=0.00..39941.83 rows=17203 width=19)
-> Index Scan using participants_claimed_time on participants p (cost=0.00..3981.65 rows=20699 width=19)
Index Cond: (claimed_time IS NOT NULL)
-> Index Scan using elsewhere_participant on elsewhere e (cost=0.00..1.87 rows=1 width=11)
Index Cond: (participant = p.username)
The new query is roughly 80x faster (under 10ms).
Each query by itself is less likely to hit statement_timeout. Adding .strip() to the queries enables searching through logs with 'grep postgres duration' to see which queries are over 50ms. First line of the query can be used for its identification.
In preparation to use trigger to cache the values for easy and fast lookup. It will eventually allow us to remove the homepage updater thread.
The new queries with the added indexes execute instantaneously. Need to add triggers when the participants are updated.
Merged
|
Reticketed as #1610. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I have removed one of the homepage_* tables, split the update query by statements and integrated sentry (ref. #1598, #1601, #1549).
EDIT: I have removed also the other two homepage_* tables. Instead giving and receiving on participants are used. These fields are updated with triggers on tips table. Triggers on update of participants are still missing.
EDIT: Do not run it yet on a production size db. The calculation of giving and receiving for everyone takes forever. Need to look into it more.