Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce refresh_queries's memory usage #4303

Closed
rauchy opened this issue Oct 27, 2019 · 5 comments
Closed

Reduce refresh_queries's memory usage #4303

rauchy opened this issue Oct 27, 2019 · 5 comments
Labels
Backend

Comments

@rauchy
Copy link
Contributor

rauchy commented Oct 27, 2019

refresh_queries looks up all outdated queries and then schedules refresh tasks for each. The lookup can take a while and consume a lot of memory. We could probably do this in batches or yield for each outdated query.
@rauchy rauchy added the Backend label Oct 27, 2019
@arikfr
Copy link
Member

arikfr commented Oct 28, 2019

Seems like it might be as simple as calling yield_per on the SQLA query object.

@weekly-digest weekly-digest bot mentioned this issue Oct 28, 2019
Closed
@rauchy
Copy link
Contributor Author

rauchy commented Oct 31, 2019

I profiled refresh_queries with 5k outdated queries and saw that peak memory rose by a mere 10% (~82mb). Also, yield_per provided negligible difference here so unless I am missing out on something, I'm not sure we should pursuit any optimization here at the moment.

@arikfr
Copy link
Member

arikfr commented Oct 31, 2019

What if you try 25K queries?

@rauchy
Copy link
Contributor Author

rauchy commented Oct 31, 2019

It rises to by 16% (~135mb).

@arikfr
Copy link
Member

arikfr commented Nov 3, 2019

👍

@arikfr arikfr closed this as completed Nov 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Backend
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arikfr @rauchy