-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor distributed query #558
Conversation
@javuto Could you please take a look and give some feedback? It was tested in our dev cluster. All the core functions work well. However, there is a bug when using multiple tags: Currently, it uses union instead of intersection. i.e. If we use two tags: {env:dev, platform:ubuntu}, in the past, it should select nodes that have both tags, but now it will select all nodes that have either one of the tags. It will be fixed when we refactor the old logic and in the future, it will supports more query based on different tags. e.g. all nodes in dev environment but not running on the windows platform. I tried to make this PR as small as possible and only replaced the way of creating new queries. Here are the follow-up tasks after this PR: |
Let's merge this and continue the refactor, nice job! 👏 🫡 |
@javuto , This PR is not completed but shows the idea of how we move all the calculations from each node's distributed query request to the distributed query creation.
In this idea, we would create a new table that records which node should execute which query. In this case, we only iteration all nodes when we create the distributed query. When osquery sends the distributed query result, we only check this table and we don't need to go through the whole list of distributed queries.
I also preferred a small change:
status
column instead of several ones. It would be simplicity and readability. Also, it would be efficient when querying the status, we don't have to write a query for each column. A classic example would be"active = ? AND completed = ? AND deleted = ? AND expired = ? AND type = ? AND environment_id = ?",
In this case, we can get rid of several tables, we only need one table to track the status of all distributed queries.
TODO after this merged: