Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #23766 to 7.x: [Filebeat] Improve ASA/FTD Ingest Pipeline #24748

Merged
merged 1 commit into from
Mar 25, 2021

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Mar 24, 2021

Cherry-pick of PR #23766 to 7.x branch. Original message:

What does this PR do?

Fixes #21658

For messages 716002:

  • Changed to GROK; allows for better parsing of event.reason
  • Added field for cisco.webvpn.group_name
  • Added field for event.reason per cisco docs for why session was terminated
  • Added field for cisco.termination_user for the AAA username terminating the connection

For messages 722051:

  • Add angle brackets to dissect to properly dissect the message, per cisco docs
  • Added field for cisco.webvpn.group.name

For messages 305011:

  • Change to GROK; allows for variance in message format with identity firewall

For messages 302020:

  • Added GROK pattern to allows for variance in message format with identity firewall

For messages 302014/302016/302021:

  • Added patterns and modified order of patterns of GROK to better match teardown messages
  • Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match.
  • Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future.
  • Added icmp_type and icmp_code parsing for 302021 messages
  • Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME.

And:

  • Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor.
  • Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.
  • Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.

Why is it important?

Improved parsing of ASA/FTD messages mentioned above.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

cd x-pack/filebeat
PYTEST_ADDOPTS="-k cisco" TESTING_FILEBEAT_FILESETS=asa,ftd mage -v pythonIntegTest

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 24, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 24, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 24, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: andrewkroh commented: run tests

  • Start Time: 2021-03-25T11:32:54.615+0000

  • Duration: 61 min 49 sec

  • Commit: be87dc9

Test stats 🧪

Test Results
Failed 0
Passed 13130
Skipped 2257
Total 15387

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 13130
Skipped 2257
Total 15387

@andrewkroh
Copy link
Member Author

andrewkroh commented Mar 24, 2021

This is waiting on #24718 to be backported to 7.12 to fix the AMP tests.

Fixes elastic#21658

For messages 716002:
- Changed to GROK; allows for better parsing of event.reason
- Added field for cisco.webvpn.group_name
- Added field for event.reason per cisco docs for why session was terminated
- Added field for cisco.termination_user for the AAA username terminating the connection

For messages 722051:
- Add angle brackets to dissect to properly dissect the message, per cisco docs
- Added field for cisco.webvpn.group.name

For messages 305011:
- Change to GROK; allows for variance in message format with identity firewall

For messages 302020:
- Added GROK pattern to allows for variance in message format with identity firewall

For messages 302014/302016/302021:
- Added patterns and modified order of patterns of GROK to better match teardown messages
- Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match.
- Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future.
- Added icmp_type and icmp_code parsing for 302021 messages
- Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME.

And:

- Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor.
- Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.
- Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.

(cherry picked from commit b5e43fc)
@andrewkroh
Copy link
Member Author

run tests

@andrewkroh andrewkroh merged commit b476353 into elastic:7.x Mar 25, 2021
@zube zube bot removed the [zube]: Done label Jun 24, 2021
@andrewkroh andrewkroh deleted the backport_23766_7.x branch January 14, 2022 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants