Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query dependency graph #510

Open
domoritz opened this issue Sep 1, 2024 · 0 comments
Open

Query dependency graph #510

domoritz opened this issue Sep 1, 2024 · 0 comments
Labels
projects Project ideas for Mosaic

Comments

@domoritz
Copy link
Member

domoritz commented Sep 1, 2024

This builds on #388. The idea here is that Mosaic sends queries to a backend for execution but can only run so many queries simultaneously. With fine grained dependencies, we could increase parallelism and potentially eliminate superfluous queries.

We have to be careful about dependencies so that visualizations don't show stale data. As an example, imagine a dashboard with three charts A, B, and C. The user brushes in A which updates B and C. The brush changes create three queries b_1, b_2, b_3 to update B and c_1, c_2, c_3 to update C. Imagine these come in as b_1, c_1, b_2, c_2, b_3, c_3 and the backend can only run two queries at a time. Right now, Mosaic only resolves queries in the same order they were queued even though b_1 and c_1 could resolve in any order without causing issues. b_2 and c_2 do not really need to be sent to the backend if we already know their results will be superseded by b_3 and c_3 so if we can keep track of that as well, we can basically drop the queries from the queue. There are many more dependencies such as queries that create new tables (such as indexes) which need to run in the correct order. For now (even with #475) exec queries never run in parallel since we don't know whether there is a dependency between them.

This could be related to #461 since we can queue up and drop superseded queries when the chart comes into view again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
projects Project ideas for Mosaic
Projects
None yet
Development

No branches or pull requests

1 participant