Skip to content

Commit

Permalink
CEF CheckPoint: adjust fields for forward compatibility (elastic#17681)…
Browse files Browse the repository at this point in the history 
… (elastic#17712)

This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related elastic#16907 elastic#17682

(cherry picked from commit ddb92ca)
@adriansr
adriansr authored Apr 15, 2020
1 parent 06f1656 commit 9c9c7ee
Showing 7 changed files with 118 additions and 83 deletions.
55 changes: 23 additions & 32 deletions filebeat/docs/fields.asciidoc
View file
Original file line number Diff line number Diff line change
@@ -4894,7 +4894,7 @@ type: keyword
--
Confidence level determined.
type: keyword
type: integer
--
@@ -4988,15 +4988,6 @@ type: long
--
*`checkpoint.file_hash`*::
+
--
File hash (SHA1 or MD5).
type: keyword
--
*`checkpoint.frequency`*::
+
--
@@ -5051,6 +5042,15 @@ type: keyword
--
*`checkpoint.malware_family`*::
+
--
Malware family.
type: keyword
--
*`checkpoint.peer_gateway`*::
+
--
@@ -5065,7 +5065,7 @@ type: ip
--
Protection performance impact.
type: keyword
type: integer
--
@@ -5123,16 +5123,25 @@ type: keyword
--
*`checkpoint.malware_status`*::
*`checkpoint.spyware_name`*::
+
--
Malware status.
Spyware name.
type: keyword
--
*`checkpoint.subscription_expiration`*::
*`checkpoint.spyware_status`*::
+
--
Spyware status.
type: keyword
--
*`checkpoint.subs_exp`*::
+
--
The expiration date of the subscription.
@@ -5195,24 +5204,6 @@ type: keyword
--
*`checkpoint.malware_name`*::
+
--
Malware name.
type: keyword
--
*`checkpoint.malware_family`*::
+
--
Malware family.
type: keyword
--
*`checkpoint.voip_log_type`*::
+
--
16 changes: 8 additions & 8 deletions filebeat/docs/modules/cef.asciidoc
View file
Original file line number Diff line number Diff line change
@@ -70,17 +70,17 @@ Check Point CEF extensions are mapped as follows:
| deviceInboundInterface | - | observer.ingress.interface.name | - |
| deviceOutboundInterface | - | observer.egress.interface.name | - |
| externalId | - | - | checkpoint.uuid |
| fileHash | - | file.hash.{md5,sha1} | checkpoint.file_hash |
| fileHash | - | file.hash.{md5,sha1} | - |
| reason | - | - | checkpoint.termination_reason |
| checkrequestCookies | - | - | checkpoint.cookie |
| requestCookies | - | - | checkpoint.cookie |
| sourceNtDomain | - | dns.question.name | - |
| Signature | - | vulnerability.id | - |
| Recipient | - | destination.user.email | - |
| Sender | - | source.user.email | - |
| deviceCustomFloatingPoint1 | update version | observer.version | - |
| deviceCustomIPv6Address2 | source ipv6 address | source.ip | - |
| deviceCustomIPv6Address3 | destination ipv6 address | destination.ip | - |
.3+| deviceCustomNumber1 | elapsed time in seconds | host.uptime | - |
.3+| deviceCustomNumber1 | elapsed time in seconds | event.duration | - |
| email recipients number | - | checkpoint.email_recipients_num |
| payload | network.bytes | - |
.2+| deviceCustomNumber2 | icmp type | - | checkpoint.icmp_type |
@@ -100,9 +100,9 @@ Check Point CEF extensions are mapped as follows:
| update status | - | checkpoint.update_status |
| peer gateway | - | checkpoint.peer_gateway |
| categories | rule.category | - |
.4+| deviceCustomString6 | application name | process.name | - |
.4+| deviceCustomString6 | application name | network.application | - |
| virus name | - | checkpoint.virus_name |
| malware name | - | checkpoint.malware_name |
| malware name | - | checkpoint.spyware_name |
| malware family | - | checkpoint.malware_family |
.5+| deviceCustomString3 | user group | group.name | - |
| incident extension | - | checkpoint.incident_extension |
@@ -122,15 +122,15 @@ Check Point CEF extensions are mapped as follows:
| vlan id | network.vlan.id | - |
| authentication method | - | checkpoint.auth_method |
| email session id | - | checkpoint.email_session_id |
| deviceCustomDate2 | subscription expiration | - | checkpoint.subscription_expiration |
| deviceCustomDate2 | subscription expiration | - | checkpoint.subs_exp |
| deviceFlexNumber1 | confidence | - | checkpoint.confidence_level |
.2+| deviceFlexNumber2 | performance impact | - | checkpoint.performance_impact |
| destination phone number | - | checkpoint.dst_phone_number |
| flexString1 | application signature id | - | checkpoint.app_sig_id |
.2+| flexString2 | malware action | event.action | - |
.2+| flexString2 | malware action | rule.description | - |
| attack information | event.action | - |
| rule_uid | - | rule.uuid | - |
| ifname | - | observer.ingress.interface.name | - |
| ifname | - | observer.ingress.interface.name | - |
| inzone | - | observer.ingress.zone | - |
| outzone | - | observer.egress.zone | - |
| product | - | observer.product | - |
16 changes: 8 additions & 8 deletions x-pack/filebeat/module/cef/_meta/docs.asciidoc
View file
Original file line number Diff line number Diff line change
@@ -65,17 +65,17 @@ Check Point CEF extensions are mapped as follows:
| deviceInboundInterface | - | observer.ingress.interface.name | - |
| deviceOutboundInterface | - | observer.egress.interface.name | - |
| externalId | - | - | checkpoint.uuid |
| fileHash | - | file.hash.{md5,sha1} | checkpoint.file_hash |
| fileHash | - | file.hash.{md5,sha1} | - |
| reason | - | - | checkpoint.termination_reason |
| checkrequestCookies | - | - | checkpoint.cookie |
| requestCookies | - | - | checkpoint.cookie |
| sourceNtDomain | - | dns.question.name | - |
| Signature | - | vulnerability.id | - |
| Recipient | - | destination.user.email | - |
| Sender | - | source.user.email | - |
| deviceCustomFloatingPoint1 | update version | observer.version | - |
| deviceCustomIPv6Address2 | source ipv6 address | source.ip | - |
| deviceCustomIPv6Address3 | destination ipv6 address | destination.ip | - |
.3+| deviceCustomNumber1 | elapsed time in seconds | host.uptime | - |
.3+| deviceCustomNumber1 | elapsed time in seconds | event.duration | - |
| email recipients number | - | checkpoint.email_recipients_num |
| payload | network.bytes | - |
.2+| deviceCustomNumber2 | icmp type | - | checkpoint.icmp_type |
@@ -95,9 +95,9 @@ Check Point CEF extensions are mapped as follows:
| update status | - | checkpoint.update_status |
| peer gateway | - | checkpoint.peer_gateway |
| categories | rule.category | - |
.4+| deviceCustomString6 | application name | process.name | - |
.4+| deviceCustomString6 | application name | network.application | - |
| virus name | - | checkpoint.virus_name |
| malware name | - | checkpoint.malware_name |
| malware name | - | checkpoint.spyware_name |
| malware family | - | checkpoint.malware_family |
.5+| deviceCustomString3 | user group | group.name | - |
| incident extension | - | checkpoint.incident_extension |
@@ -117,15 +117,15 @@ Check Point CEF extensions are mapped as follows:
| vlan id | network.vlan.id | - |
| authentication method | - | checkpoint.auth_method |
| email session id | - | checkpoint.email_session_id |
| deviceCustomDate2 | subscription expiration | - | checkpoint.subscription_expiration |
| deviceCustomDate2 | subscription expiration | - | checkpoint.subs_exp |
| deviceFlexNumber1 | confidence | - | checkpoint.confidence_level |
.2+| deviceFlexNumber2 | performance impact | - | checkpoint.performance_impact |
| destination phone number | - | checkpoint.dst_phone_number |
| flexString1 | application signature id | - | checkpoint.app_sig_id |
.2+| flexString2 | malware action | event.action | - |
.2+| flexString2 | malware action | rule.description | - |
| attack information | event.action | - |
| rule_uid | - | rule.uuid | - |
| ifname | - | observer.ingress.interface.name | - |
| ifname | - | observer.ingress.interface.name | - |
| inzone | - | observer.ingress.zone | - |
| outzone | - | observer.egress.zone | - |
| product | - | observer.product | - |
2 changes: 1 addition & 1 deletion x-pack/filebeat/module/cef/fields.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

72 changes: 55 additions & 17 deletions x-pack/filebeat/module/cef/log/_meta/fields.yml
View file
Original file line number Diff line number Diff line change
@@ -18,170 +18,208 @@
fields:
- name: app_risk
type: keyword
overwrite: true
description: Application risk.

- name: app_severity
type: keyword
overwrite: true
description: Application threat severity.

- name: app_sig_id
type: keyword
overwrite: true
description: The signature ID which the application was detected by.

- name: auth_method
type: keyword
overwrite: true
description: Password authentication protocol used.

- name: category
type: keyword
overwrite: true
description: Category.

- name: confidence_level
type: keyword
type: integer
overwrite: true
description: Confidence level determined.

- name: connectivity_state
type: keyword
overwrite: true
description: Connectivity state.

- name: cookie
type: keyword
overwrite: true
description: IKE cookie.

- name: dst_phone_number
type: keyword
overwrite: true
description: Destination IP-Phone.

- name: email_control
type: keyword
overwrite: true
description: Engine name.

- name: email_id
type: keyword
overwrite: true
description: Internal email ID.

- name: email_recipients_num
type: long
overwrite: true
description: Number of recipients.

- name: email_session_id
type: keyword
overwrite: true
description: Internal email session ID.

- name: email_spool_id
overwrite: true
type: keyword

description: Internal email spool ID.

- name: email_subject
type: keyword
overwrite: true
description: Email subject.

- name: event_count
type: long
overwrite: true
description: Number of events associated with the log.

- name: file_hash
type: keyword
description: File hash (SHA1 or MD5).

- name: frequency
type: keyword
overwrite: true
description: Scan frequency.

- name: icmp_type
type: long
overwrite: true
description: ICMP type.

- name: icmp_code
type: long
overwrite: true
description: ICMP code.

- name: identity_type
type: keyword
overwrite: true
description: Identity type.

- name: incident_extension
type: keyword
overwrite: true
description: Format of original data.

- name: integrity_av_invoke_type
type: keyword
overwrite: true
description: Scan invoke type.

- name: malware_family
type: keyword
overwrite: true
description: Malware family.

- name: peer_gateway
type: ip
overwrite: true
description: Main IP of the peer Security Gateway.

- name: performance_impact
type: keyword
type: integer
overwrite: true
description: Protection performance impact.

- name: protection_id
type: keyword
overwrite: true
description: Protection malware ID.

- name: protection_name
type: keyword
overwrite: true
description: Specific signature name of the attack.

- name: protection_type
type: keyword
overwrite: true
description: Type of protection used to detect the attack.

- name: scan_result
type: keyword
overwrite: true
description: Scan result.

- name: sensor_mode
type: keyword
overwrite: true
description: Sensor mode.

- name: severity
type: keyword
overwrite: true
description: Threat severity.

- name: malware_status
- name: spyware_name
type: keyword
description: Malware status.
overwrite: true
description: Spyware name.

- name: subscription_expiration
- name: spyware_status
type: keyword
overwrite: true
description: Spyware status.

- name: subs_exp
type: date
overwrite: true
description: The expiration date of the subscription.

- name: tcp_flags
type: keyword
overwrite: true
description: TCP packet flags.

- name: termination_reason
type: keyword
overwrite: true
description: Termination reason.

- name: update_status
type: keyword
overwrite: true
description: Update status.

- name: user_status
type: keyword
overwrite: true
description: User response.

- name: uuid
type: keyword
overwrite: true
description: External ID.

- name: virus_name
type: keyword
overwrite: true
description: Virus name.

- name: malware_name
type: keyword
description: Malware name.

- name: malware_family
type: keyword
description: Malware family.

- name: voip_log_type
type: keyword
overwrite: true
description: VoIP log types.

- name: cef.extensions
36 changes: 22 additions & 14 deletions x-pack/filebeat/module/cef/log/ingest/cp-pipeline.yml
View file
Original file line number Diff line number Diff line change
@@ -76,7 +76,7 @@ processors:
- name: deviceExternalId
to: observer.type

# Product Family
# Product Family (override deviceExternalId if present).
- name: deviceFacility
to: observer.type
convert:
@@ -104,6 +104,10 @@ processors:
to: checkpoint.termination_reason

# Possibly an IKE cookie
- name: requestCookies
to: checkpoint.cookie

# Probably a typo in CP's CEF docs
- name: checkrequestCookies
to: checkpoint.cookie

@@ -136,7 +140,7 @@ processors:
- name: deviceCustomNumber1
labels:
payload: network.bytes
elapsed time in seconds: host.uptime
elapsed time in seconds: event.duration
email recipients number: checkpoint.email_recipients_num

- name: deviceCustomNumber2
@@ -172,9 +176,9 @@ processors:

- name: deviceCustomString6
labels:
application name: process.name
application name: network.application
virus name: checkpoint.virus_name
malware name: checkpoint.malware_name
malware name: checkpoint.spyware_name
malware family: checkpoint.malware_family

- name: deviceCustomString3
@@ -208,7 +212,7 @@ processors:

- name: deviceCustomDate2
labels:
subscription expiration: checkpoint.subscription_expiration
subscription expiration: checkpoint.subs_exp

- name: deviceFlexNumber1
labels:
@@ -225,7 +229,7 @@ processors:

- name: flexString2
labels:
malware action: event.action
malware action: rule.description
attack information: event.action

- name: rule_uid
@@ -295,15 +299,19 @@ processors:
field: event.duration
ignore_missing: true

# checkpoint.file_hash can be either MD5 or SHA1.
- set:
field: file.hash.md5
value: '{{checkpoint.file_hash}}'
# checkpoint.file_hash can be either MD5, SHA1 or SHA256.
- rename:
field: checkpoint.file_hash
target_field: file.hash.md5
if: 'ctx.checkpoint?.file_hash != null && ctx.checkpoint.file_hash.length()==32'
- set:
field: file.hash.sha1
value: '{{checkpoint.file_hash}}'
- rename:
field: checkpoint.file_hash
target_field: file.hash.sha1
if: 'ctx.checkpoint?.file_hash != null && ctx.checkpoint.file_hash.length()==40'
- rename:
field: checkpoint.file_hash
target_field: file.hash.sha256
if: 'ctx.checkpoint?.file_hash != null && ctx.checkpoint.file_hash.length()==64'

# Event kind is 'event' by default. 'alert' when a risk score and rule info
# is present.
@@ -324,7 +332,7 @@ processors:
- set:
field: event.category
value: malware
if: 'ctx.checkpoint?.protection_id != null || ctx.checkpoint?.malware_name != null || ctx.checkpoint?.malware_family != null || ctx.checkpoint?.spyware_status != null'
if: 'ctx.checkpoint?.protection_id != null || ctx.checkpoint?.spyware_name != null || ctx.checkpoint?.malware_family != null || ctx.checkpoint?.spyware_status != null'
- set:
field: event.category
value: intrusion_detection
4 changes: 1 addition & 3 deletions x-pack/filebeat/module/cef/log/test/checkpoint.log-expected.json
View file
Original file line number Diff line number Diff line change
@@ -116,8 +116,7 @@
"cef.severity": "Unknown",
"cef.version": "0",
"checkpoint.email_control": "SMTP Policy Restrictions",
"checkpoint.file_hash": "55f4a511e6f630a6b1319505414f114e7bcaf13d",
"checkpoint.subscription_expiration": "2020-04-11T10:42:13.000Z",
"checkpoint.subs_exp": "2020-04-11T10:42:13.000Z",
"destination.port": 25,
"event.action": "Bypass",
"event.code": "Log",
@@ -165,7 +164,6 @@
"cef.version": "0",
"checkpoint.app_risk": "High",
"checkpoint.event_count": "12",
"checkpoint.file_hash": "580a783c1cb2b20613323f715d231a69",
"checkpoint.severity": "Very-High",
"destination.ip": "::1",
"event.action": "Drop",

0 comments on commit 9c9c7ee

Please sign in to comment.