Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Osquerybeat: Rewrite queries scheduling with extensions for scheduling and logging #25641

Merged
merged 12 commits into from
May 14, 2021

Conversation

aleksmaus
Copy link
Contributor

What does this PR do?

  • Rewritten queries scheduling, leveraging custom plugins for osquery logging and configuration.
    1. Osquerybeat now registers the plugins for configuration and logging
    2. Osqueryd provides better scheduling for the queries, spreading the execution over time and state tracking.
    3. The configuration plugin is polled by osqueryd every 60 secs for the configuration changes (schedules queries). The interval for scheduled queries refresh is acceptable and can be increased to poll even less often.
    4. The queries are always scheduled as "snapshot" queries by osquerybeat configuration plugin.
    5. All osqueryd logs including the snapshot results are captured by the logging plugin.
  • Refactored and cleaned up the osqueryd launcher and osqueryd client implementation.

Why is it important?

Addresses https://github.com/elastic/security-team/issues/1061

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas

Related issues

Screenshots

The osquery_schedule is now populated with the scheduled queries stats

Screen Shot 2021-05-10 at 10 27 36 AM

The scheduled queries and ad-hoc queries are working as expected

Screen Shot 2021-05-10 at 10 20 30 AM

Screen Shot 2021-05-10 at 10 19 50 AM

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 10, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 10, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: aleksmaus commented: /test

  • Start Time: 2021-05-14T11:30:14.299+0000

  • Duration: 149 min 42 sec

  • Commit: de9f51b

Test stats 🧪

Test Results
Failed 0
Passed 929
Skipped 186
Total 1115

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 929
Skipped 186
Total 1115

Copy link
Contributor

@lykkin lykkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit: 🎉

@aleksmaus
Copy link
Contributor Author

/test

bt *osquerybeat
}

func (a *actionHandler) Name() string {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requires pointer? not important, just curious

Copy link
Contributor Author

@aleksmaus aleksmaus May 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is passed into the agent client as interface so will escape to the heap no matter what, and it's only allocated once

}

func actionDataFromRequest(req map[string]interface{}) (ad actionData, err error) {
if req == nil {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

len(req) == 0 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

exedir := filepath.Dir(exefp)

// Install osqueryd if needed
err = installOsqueryWithDir(ctx, exedir)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return installOsqueryWithDir(ctx, exedir) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

@aleksmaus
Copy link
Contributor Author

/test

@aleksmaus aleksmaus merged commit c1a3a8d into elastic:master May 14, 2021
mergify bot pushed a commit that referenced this pull request May 14, 2021
…g and logging (#25641)

* Osquerybeat: Rewrite queries scheduling  with extensions for scheduling and logging

* Make check happy

* Update notice, to make linter happy

* Fix Go 1.16 check failure, when calling t.Fatal from goroutine

* Fix windows build

* Add configuration plugin tests

* Add logger plugin unit tests

* Rollback gotest_test change

* Address code review comments

(cherry picked from commit c1a3a8d)

# Conflicts:
#	go.mod
aleksmaus added a commit that referenced this pull request May 17, 2021
…xtensions for scheduling and logging (#25722)

* Osquerybeat: Rewrite queries scheduling  with extensions for scheduling and logging (#25641)

* Osquerybeat: Rewrite queries scheduling  with extensions for scheduling and logging

* Make check happy

* Update notice, to make linter happy

* Fix windows build

* Add configuration plugin tests

* Add logger plugin unit tests

* Rollback gotest_test change

* Address code review comments

(cherry picked from commit c1a3a8d)

# Conflicts:
#	go.mod

* Resolve conflict

Co-authored-by: Aleksandr Maus <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants