Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Improve ECS categorization field mappings for azure module #19376

Merged
merged 4 commits into from
Jul 8, 2020

Conversation

leehinman
Copy link
Contributor

@leehinman leehinman commented Jun 24, 2020

What does this PR do?

Improve ECS categorization field mappings for azure module.
Specifically:

  • activitylogs
    • add azure.activitylogs.result_type
    • set default_field: false
    • populate event.outcome with allowed values
    • set event.action
    • populate event.category with allowed values
    • set event.kind
    • set event.type
    • add support tickets example
    • add geoip for source.ip
    • add AS info for source.ip
    • add user.name
    • add user.full_name
    • add user.domain
  • auditlogs
    • set default_field: false
    • add azure.auditlogs.category
    • populate event.outcome with allowed values
    • set event.action
    • set event.kind
  • signinlogs
    • set default_field: false
    • set event.action
    • populate event.category with allowed values
    • set event.type
    • populate event.outcome with allowed values
    • add azure.signinlogs.category
    • add azure.signinlogs.result_type
    • set user.name
    • set user.domain
    • set user.full_name
    • set user.id
    • add geoip for source.ip
    • add AS info for source.ip

Why is it important?

ECS categorization fields allow cross correlation between filesets.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

TESTING_FILEBEAT_MODULES=azure mage -v pythonIntegTest

Related issues

  • Closes 16155

@leehinman leehinman added enhancement Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Team:SIEM labels Jun 24, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 24, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 24, 2020
@leehinman leehinman requested a review from narph June 24, 2020 18:18
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 24, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19376 updated]

  • Start Time: 2020-07-07T21:08:03.894+0000

  • Duration: 56 min 5 sec

Test stats 🧪

Test Results
Failed 0
Passed 4213
Skipped 677
Total 4890

@leehinman leehinman changed the title [Filebeat] 16155 azure ecs 1.4 [Filebeat] Improve ECS categorization field mappings for azure module Jun 25, 2020
@andrewkroh
Copy link
Member

run tests

"event.dataset": "azure.signinlogs",
"event.duration": 0.0,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That type checking PR you opened is looking like a very good idea 😄 .

@leehinman leehinman force-pushed the 16155_azure_ecs_1.4 branch from d52bbe8 to f4ab282 Compare July 1, 2020 14:50
leehinman added 3 commits July 1, 2020 13:22
- activitylogs
  + convert pipeline to yml
- auditlogs
  + convert pipeline to yml
- signinlogs
  + convert pipeline to yml

Closes elastic#16155
- activitylogs
  + add azure.activitylogs.result_type
  + set default_field: false
  + populate event.outcome with allowed values
  + set event.action
  + populate event.category with allowed values
  + set event.kind
  + set event.type
  + add support tickets example
  + add geoip for source.ip
  + add AS info for source.ip
  + add user.name
  + add user.full_name
  + add user.domain
- auditlogs
  + set default_field: false
  + add azure.auditlogs.category
  + populate event.outcome with allowed values
  + set event.action
  + set event.kind
- signinlogs
  + set default_field: false
  + set event.action
  + populate event.category with allowed values
  + set event.type
  + populate event.outcome with allowed values
  + add azure.signinlogs.category
  + add azure.signinlogs.result_type
  + set user.name
  + set user.domain
  + set user.full_name
  + set user.id
  + add geoip for source.ip
  + add AS info for source.ip
@leehinman leehinman force-pushed the 16155_azure_ecs_1.4 branch from f4ab282 to 80e67c8 Compare July 1, 2020 18:22
Copy link
Contributor

@narph narph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might worth checking against the existing dashboards, I think they contain some of the fields that have been replaced

@leehinman
Copy link
Contributor Author

@narph tested on azure and with changes to the dashboards everything looks ok. anything else you can think of to check?

@narph
Copy link
Contributor

narph commented Jul 8, 2020

@narph tested on azure and with changes to the dashboards everything looks ok. anything else you can think of to check?

LGTM, thanks

@leehinman leehinman merged commit 00a274e into elastic:master Jul 8, 2020
leehinman added a commit to leehinman/beats that referenced this pull request Jul 8, 2020
…elastic#19376)

* Improve ECS categorization field mappings in azure module

- activitylogs
  + convert pipeline to yml
  + add azure.activitylogs.result_type
  + set default_field: false
  + populate event.outcome with allowed values
  + set event.action
  + populate event.category with allowed values
  + set event.kind
  + set event.type
  + add support tickets example
  + add geoip for source.ip
  + add AS info for source.ip
  + add user.name
  + add user.full_name
  + add user.domain
  + update dashboards
- auditlogs
  + convert pipeline to yml
  + set default_field: false
  + add azure.auditlogs.category
  + populate event.outcome with allowed values
  + set event.action
  + set event.kind
  + update dashboards
- signinlogs
  + convert pipeline to yml
  + set default_field: false
  + set event.action
  + populate event.category with allowed values
  + set event.type
  + populate event.outcome with allowed values
  + add azure.signinlogs.category
  + add azure.signinlogs.result_type
  + set user.name
  + set user.domain
  + set user.full_name
  + set user.id
  + add geoip for source.ip
  + add AS info for source.ip
  + update dashboards

Closes elastic#16155

(cherry picked from commit 00a274e)
leehinman added a commit to leehinman/beats that referenced this pull request Jul 8, 2020
…elastic#19376)

* Improve ECS categorization field mappings in azure module

- activitylogs
  + convert pipeline to yml
  + add azure.activitylogs.result_type
  + set default_field: false
  + populate event.outcome with allowed values
  + set event.action
  + populate event.category with allowed values
  + set event.kind
  + set event.type
  + add support tickets example
  + add geoip for source.ip
  + add AS info for source.ip
  + add user.name
  + add user.full_name
  + add user.domain
  + update dashboards
- auditlogs
  + convert pipeline to yml
  + set default_field: false
  + add azure.auditlogs.category
  + populate event.outcome with allowed values
  + set event.action
  + set event.kind
  + update dashboards
- signinlogs
  + convert pipeline to yml
  + set default_field: false
  + set event.action
  + populate event.category with allowed values
  + set event.type
  + populate event.outcome with allowed values
  + add azure.signinlogs.category
  + add azure.signinlogs.result_type
  + set user.name
  + set user.domain
  + set user.full_name
  + set user.id
  + add geoip for source.ip
  + add AS info for source.ip
  + update dashboards

Closes elastic#16155

(cherry picked from commit 00a274e)
@leehinman leehinman added v7.9.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jul 8, 2020
leehinman added a commit that referenced this pull request Jul 8, 2020
…#19376) (#19737)

* Improve ECS categorization field mappings in azure module

- activitylogs
  + convert pipeline to yml
  + add azure.activitylogs.result_type
  + set default_field: false
  + populate event.outcome with allowed values
  + set event.action
  + populate event.category with allowed values
  + set event.kind
  + set event.type
  + add support tickets example
  + add geoip for source.ip
  + add AS info for source.ip
  + add user.name
  + add user.full_name
  + add user.domain
  + update dashboards
- auditlogs
  + convert pipeline to yml
  + set default_field: false
  + add azure.auditlogs.category
  + populate event.outcome with allowed values
  + set event.action
  + set event.kind
  + update dashboards
- signinlogs
  + convert pipeline to yml
  + set default_field: false
  + set event.action
  + populate event.category with allowed values
  + set event.type
  + populate event.outcome with allowed values
  + add azure.signinlogs.category
  + add azure.signinlogs.result_type
  + set user.name
  + set user.domain
  + set user.full_name
  + set user.id
  + add geoip for source.ip
  + add AS info for source.ip
  + update dashboards

Closes #16155

(cherry picked from commit 00a274e)
@kevinserafin
Copy link

Pardon the question but I was attempting to integrate some of these pipeline changes in now and I am wondering if I am not understanding how the error_code set processors are supposed to function.

  • set:
    field: event.outcome
    value: success
    if: "ctx?.azure?.signinlogs?.properties?.status?.error_code == null || ctx.azure.signinlogs.properties.status.error_code == 0"
  • set:
    field: event.outcome
    value: failure
    if: "ctx?.azure?.signinlogs?.properties?.status?.error_code != null || ctx.azure.signinlogs.properties.status.error_code > 0"

So if I pass a zero for azure.signinlogs.properties.status.error_code, is it not evaluated as true for both the success and failure processor? All of my testing kept events showing up as failures so I started poking around, changing the failure processor to Boolean And seemed to fix it but I wanted to see if I was missing something first.

@leehinman
Copy link
Contributor Author

@kevinserafin you are correct that was a bug. PR #20254 should fix

@leehinman leehinman deleted the 16155_azure_ecs_1.4 branch October 5, 2020 19:12
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
…elastic#19376)

* Improve ECS categorization field mappings in azure module

- activitylogs
  + convert pipeline to yml
  + add azure.activitylogs.result_type
  + set default_field: false
  + populate event.outcome with allowed values
  + set event.action
  + populate event.category with allowed values
  + set event.kind
  + set event.type
  + add support tickets example
  + add geoip for source.ip
  + add AS info for source.ip
  + add user.name
  + add user.full_name
  + add user.domain
  + update dashboards
- auditlogs
  + convert pipeline to yml
  + set default_field: false
  + add azure.auditlogs.category
  + populate event.outcome with allowed values
  + set event.action
  + set event.kind
  + update dashboards
- signinlogs
  + convert pipeline to yml
  + set default_field: false
  + set event.action
  + populate event.category with allowed values
  + set event.type
  + populate event.outcome with allowed values
  + add azure.signinlogs.category
  + add azure.signinlogs.result_type
  + set user.name
  + set user.domain
  + set user.full_name
  + set user.id
  + add geoip for source.ip
  + add AS info for source.ip
  + update dashboards

Closes elastic#16155
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants