Optimise sql queries #161
Merged
Optimise sql queries #161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
January 2, 2017 13:09
By defaut, SQLAlchemy uses lazy loading, which means that displaying n bills will generate around n queries (to get the list of owers of each bill). Pre-load the list of owers to drastically decrease the number of SQL queries. Before this commit: 1004 SQL queries, 7535 ms elapsed time, 7536 ms CPU time, 530 ms SQL time After this commit: 5 SQL queries, 3342 ms elapsed time, 3393 ms CPU time, 15 ms SQL time Measured request: display the list of all bills for the project (without displaying the sidebar with balances) Test setup to measure performance improvement: - 5 users with various weights - 1000 bills, each paid by a random user, each involving all 5 users - laptop with Celeron [email protected] GHz, SSD Samsung 850 EVO - sqlite database on SSD, using sqlite 3.15.2 - python 2.7.13 - Flask-DebugToolbar 0.10.0 (to count SQL queries and loading time) Performance measurements (using Flask-DebugToolbar with the second request, to avoid measuring cold-cache performance): - number of SQL queries - elapsed time (from request to response) - total CPU time consumed by the server handling the request - total time spent on SQL queries (as reported by SQLAlchemy)
This avoids creating thousands of small SQL queries when computing the
balance of users. This significantly improves the performance of
displaying the main page of a project, since the balance of users is
displayed there:
Before this commit: 4004 SQL queries, 19793 ms elapsed time, 19753 ms CPU time, 2094 ms SQL time
After this commit: 12 SQL queries, 3688 ms elapsed time, 3753 ms CPU time, 50 ms SQL time
Measured request: display the sidebar with the balance of all users for the project (without displaying the list of bills)
This commit also greatly improves the performance of the "settle bills" page:
Before this commit: 8006 SQL queries, 39167 ms elapsed time, 39600 ms CPU time, 4141 ms SQL time
After this commit: 22 SQL queries, 7144 ms elapsed time, 7283 ms CPU time, 96 ms SQL time
Measured request: display the "Settle bills" page
Test setup to measure performance improvement:
- 5 users with various weights
- 1000 bills, each paid by a random user, each involving all 5 users
- laptop with Celeron [email protected] GHz, SSD Samsung 850 EVO
- sqlite database on SSD, using sqlite 3.15.2
- python 2.7.13
- Flask-DebugToolbar 0.10.0 (to count SQL queries and loading time)
Performance measurements (using Flask-DebugToolbar on the second request,
to avoid measuring cold-cache performance):
- number of SQL queries
- elapsed time (from request to response)
- total CPU time consumed by the server handling the request
- total time spent on SQL queries (as reported by SQLAlchemy)
|
Just tested your branch, with a database of 1000 bills. This is really good ! I was going to run some numbers to show the improvements, but I saw you already have a nice summary on your commit message. |
aavenel
approved these changes
Jan 2, 2017
|
Really good numbers here, thanks! |
|
@almet Does this PR has been deployed on ihatemoney.org ? The website is still a bit slow unfortunatly |
|
It wasn't deployed, but I just did it, thanks for the poke! |
Jojo144
pushed a commit
to Jojo144/ihatemoney
that referenced
this pull request
Mar 21, 2020
Optimise sql queries
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These two commits mostly fix issue #159.