Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] add panos type and sub_type #20912

Merged
merged 4 commits into from
Sep 15, 2020
Merged

[Filebeat] add panos type and sub_type #20912

merged 4 commits into from
Sep 15, 2020

Conversation

leehinman
Copy link
Contributor

What does this PR do?

adds panw.panos.type & panw.panos.sub_type fields

Why is it important?

Original type & sub_type may be useful instead of event.category &
event.type that have ECS specified values.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

TESTING_FILEBEAT_MODULES=panw mage -v pythonIntegTest

@leehinman leehinman added enhancement Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Team:SIEM labels Sep 2, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 2, 2020
andrewkroh
andrewkroh reviewed Sep 2, 2020
View reviewed changes
x-pack/filebeat/module/panw/panos/config/input.yml Outdated
@@ -51,6 +51,8 @@ processors:
omit_empty: true
fail_on_error: false
mappings:
panw.panos.type: 3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like type=3 and subtype=4 are common across all of the log types in PAN-OS. So maybe the pipeline should set them for all log types by placing this around line 38?

@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 2, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20912 updated]

  • Start Time: 2020-09-15T16:06:40.820+0000

  • Duration: 51 min 20 sec

Test stats 🧪

Test Results
Failed 0
Passed 5594
Skipped 825
Total 6419

@gimmic
Copy link

gimmic commented Sep 2, 2020

Is there a reason this is destined to be panw.panos.* and not just panw.* ?

@leehinman
Copy link
Contributor Author

Is there a reason this is destined to be panw.panos.* and not just panw.* ?

@gimmic Do you know if type & sub_type are used in every panw product? If so I'll move them.

@gimmic
Copy link

gimmic commented Sep 3, 2020

Is there a reason this is destined to be panw.panos.* and not just panw.* ?

@gimmic Do you know if type & sub_type are used in every panw product? If so I'll move them.

While it could be assumed there are non PAN-OS based palo alto logging functions, I have not been able to find any documentation on them in my searching. I think any modern palo product will fit under pan-os, and instead if it isn't pan-os that outlier could be the additional demarcated field. (and having pan-os be the default, if that makes sense?)

panw.subtype and for non-panos, it could be panw.subproduct.fields

In the end it's probably inconsequential it is just from a human-analyst standpoint I get leery about unnecessary nesting in field names just from a memorization/typing standpoint.

@leehinman
add panos type and sub_type
2199050 
- add panw.panos.type
- add panw.panos.sub_type
@leehinman
add type & sub_type for all events
6fbfd5c
andrewkroh
andrewkroh approved these changes Sep 14, 2020
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@leehinman leehinman merged commit d14c6a1 into elastic:master Sep 15, 2020
leehinman added a commit to leehinman/beats that referenced this pull request Sep 15, 2020
@leehinman
[Filebeat] add panos type and sub_type (elastic#20912)
b174365 
* add panos type and sub_type

- add panw.panos.type
- add panw.panos.sub_type

(cherry picked from commit d14c6a1)
@leehinman leehinman mentioned this pull request Sep 15, 2020
Merged
4 tasks
@leehinman leehinman added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Sep 15, 2020
leehinman added a commit that referenced this pull request Sep 16, 2020
@leehinman
[Filebeat] add panos type and sub_type (#20912) (#21102)
111ad83 
* add panos type and sub_type

- add panw.panos.type
- add panw.panos.sub_type

(cherry picked from commit d14c6a1)
v1v added a commit to v1v/beats that referenced this pull request Sep 18, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
4c7e8da 
…ne-2.0

* upstream/master: (44 commits)
  Update users.asciidoc (elastic#20802) (elastic#21108)
  Fix docker provider builder. (elastic#21118)
  [Elastic Agent] Add docker composable dynamic provider. (elastic#20842)
  Add new modules/filesets from rsa2elk for 7.10 (elastic#20820)
  Fix broken links to external websites (elastic#21061)
  [docs] typo in the command line (elastic#20799)
  [Filebeat] add panos type and sub_type (elastic#20912)
  Move the `compute_vm_scalset` to  a light metricset and map the cloud metadata (elastic#21038)
  [Filebeat] Add support for Cloudtrail digest files (elastic#21086)
  Add metrics collection from cost explorer into aws/billing metricset (elastic#20527)
  Add vendoring to Google Cloud Functions again (elastic#21070)
  [Elastic Agent] Add fleet.host.id for sending to endpoint. (elastic#21042)
  Do not need Google credentials before using it (elastic#21072)
  [Filebeat][New Module] Zoom webhook module (elastic#20414)
  Add support for GMT timezone offset in decode_cef (elastic#20993)
  Filebeat: Fix random error on harvester close (elastic#21048)
  Add ingress controller dashboards (elastic#21052)
  Fix loggers in composable module. (elastic#21047)
  [Ingest Manager] Increase kibana client timeout to 5 minutes (elastic#21037)
  Add changelog. (elastic#21041)
  ...
@leehinman leehinman deleted the panos_type_subtype branch October 5, 2020 19:21
@jenback
Copy link

jenback commented Jan 12, 2021

Hi,
i'm not sure if that comment is correct under this topic.
We already use the latest Pipeline from here: https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/panw/panos/ingest/pipeline.yml and we saw, that the
event.kind-Field ist not always correct.
In PanOS the URL-Events are THREAT-Events too, so all URL-Events are also THREAT-Events.
I think the line 233 ( if: 'ctx?.panw?.panos?.type == "THREAT"' ) should be:
if: 'ctx?.panw?.panos?.type == "THREAT"' && ctx?.panw?.panos?.threat?.id != '9999'

Kind Regards
Jens Backs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees
No one assigned
Labels
enhancement Filebeat Filebeat v7.10.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@leehinman @elasticmachine @gimmic @jenback @andrewkroh