Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New input for Office 365 audit logs #16244

Merged
merged 20 commits into from
Mar 5, 2020
Merged

New input for Office 365 audit logs #16244

merged 20 commits into from
Mar 5, 2020

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Feb 11, 2020 •
edited
Loading

What does this PR do?

This adds a new input, o365audit, to retrieve audit events from an Office 365 subscription using Microsoft's Office 365 Management API.

Why is it important?

This is a first step in developing a module to ingest O365 audit events.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • Add fields.yml. moved to module
  • Documentation.
  • Finish request error handling in contentblob/pagination.
  • Persist state (needs Input V2).
  • (Optional) Client secret-based authentication.

How to test this PR locally

Setup an Azure application in an existing O365 subscription. Follow the steps in this blog post: https://medium.com/@kiamatthews/office-365-management-api-connector-for-elk-b94fe4ed4a53

Related issues

@adriansr adriansr added enhancement in progress Pull request is currently in progress. review Filebeat Filebeat Team:SIEM labels Feb 11, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@adriansr adriansr mentioned this pull request Feb 11, 2020
Closed
18 tasks
@andrewkroh andrewkroh self-requested a review February 11, 2020 23:57
@adriansr adriansr mentioned this pull request Feb 18, 2020
Merged
5 tasks
@adriansr adriansr marked this pull request as ready for review February 21, 2020 17:47
andrewkroh
andrewkroh reviewed Feb 25, 2020
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job. The main execution loop is interesting.

x-pack/filebeat/docs/inputs/input-o365audit.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-o365audit.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-o365audit.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-o365audit.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/input/o365audit/auth/auth.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/o365audit/contentblob.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/o365audit/listblobs.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/o365audit/listblobs.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/o365audit/listblobs.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/o365audit/pagination.go Show resolved Hide resolved
@adriansr adriansr changed the title [Draft] New input for Office 365 audit logs New input for Office 365 audit logs Feb 26, 2020
@adriansr adriansr removed the in progress Pull request is currently in progress. label Mar 2, 2020
@adriansr
Copy link
Contributor Author

adriansr commented Mar 2, 2020

Removed the in progress label. I think it makes sense to merge without state storage instead of waiting for Input V2, because we can either add in a later PR or ship as-is.

@adriansr adriansr requested a review from andrewkroh March 2, 2020 12:23
@andrewkroh
Copy link
Member

I agree about merging before the state storage is available since it can work without it. We'll update this input after the new API becomes available.

There are three more comments from me hidden by GH. It's not that you have to make changes for them, but I'm not sure you saw them.

Screen Shot 2020-03-02 at 11 05 04 AM

adriansr added 11 commits March 3, 2020 10:18
@adriansr
[SIEM] New o365 input for Office 365 audit logs
5150c54 
This input uses Microsoft's Office 365 Management API to fetch audit
events.

Relates elastic#16196
@adriansr
Improve error handling
bdb0ddb
@adriansr
Fix initial query failure when queued for a long time.
1a7df99 
If the first query list_blobs(now-7d,now-6d) is queued for more than 1h
(because of service unavailable errors for example), when it finally
runs it falls outside the acceptable time-range for the server,
resulting in a AF20030 error.
@adriansr
Removed extra debug
90f62bc
@adriansr
Better handling of network errors
30ebd24
@adriansr
Add support for client secret authentication
3b6e221
@adriansr
Improved config error handling
d3ef9e9
@adriansr
mage fmt
bb5251b
@adriansr
Minor date change
487d540
@adriansr
Set document ID from audit record ID.
837012c
@adriansr
Make input less noisy when backend is down
44a5400
@adriansr adriansr force-pushed the feature_o365_input branch from e44a873 to 6079341 Compare March 3, 2020 09:19
andrewkroh
andrewkroh approved these changes Mar 3, 2020
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adriansr adriansr merged commit ed80900 into elastic:master Mar 5, 2020
@adriansr adriansr added the needs_backport PR is waiting to be backported to other branches. label Mar 19, 2020
@adriansr adriansr mentioned this pull request Mar 19, 2020
Merged
10 tasks
@adriansr adriansr added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 19, 2020
adriansr added a commit to adriansr/beats that referenced this pull request Mar 19, 2020
@adriansr
New input for Office 365 audit logs (elastic#16244)
12c5ab4 
This input uses Microsoft's Office 365 Management API to fetch audit
events.

Relates to elastic#16196

(cherry picked from commit ed80900)
adriansr added a commit that referenced this pull request Mar 20, 2020
@adriansr
Cherry-pick #16244 to 7.x: New input for Office 365 audit logs (#17109)
d35ca4f 
This input uses Microsoft's Office 365 Management API to fetch audit
events.

Relates to #16196

(cherry picked from commit ed80900)
@adriansr adriansr mentioned this pull request Mar 20, 2020
Merged
5 tasks
@andrewkroh andrewkroh mentioned this pull request Apr 30, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees
No one assigned
Labels
enhancement Filebeat Filebeat review v7.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@adriansr @elasticmachine @andrewkroh