[sophos] Various improvements and log samples from 18.5 #3127

andrewkroh · 2022-04-19T19:02:35Z

What does this PR do?

Remove space from sophos.xg.trans_src_ip field name in mapping. Sync fix from [Filebeat] Remove space from field sophos.xg.trans_src_ ip beats#25250.
Do not modify event.original.
Populate url.* fields based on sophos.xg.url.
Rename sophos.xg.reason to event.reason (ECS).
Normalize all sophos.xg.* keys to lowercase.
Lowercase network.transport as per ECS.
Format source.mac and destination.mac as per ECS.
Add Sophos Firewall 18.5 log samples taken from documentation.
Set the event.code from the message ID (and remove sophos.xg.messageid).
Consolidate geoip enrichment
Consolidate related.ip processors
Consolidate lowercase network.transport/protocol
Consolidate related.user processing
Consolidate related.hash
Consolidate network.{bytes,packets} calculations
Add network.community_id
Add {url,source,destination}.domain to related.hosts
Set event.duration (ns) for WAF events using reponsetime (us)
Remove client/server mappings. This was a large amount of duplication causing increased event sizes.
Add reference link to xg syslog formats

Anti-Virus

Set destination.domain from sophos.xg.dstdomain in anti-virus log.
Use sophos.xg.dst_domainname to set destination.domain for Anti-Virus SMTP events.
Use sophos.xg.src_domainname to set source.domain for Anti-Virus SMTP events.
Set network.protocol for anti-virus logs.

Anti-Spam

Set network.protocol for anti-spam logs.

Content Filtering

Set network.protocol for Content Filtering. Use the URL scheme to derive the protocol.

Sandstorm

Set file.name for Sandstorm events (there was a typo in the if condition).
Set url.domain, destination.domain, and destination.ip for Sandstorm web events.
Handle sha256 Sandstorm file hashes. In 17.5 and earlier the sha1sum contained the sha1 checksum of the file being analyzed. In 18.0 and later the sha1sum contains the sha256 checksum of the file.

System Health

Convert sophos.xg.collisions, sophos.xg.receiveddrops, and sophos.xg.transmitteddrops to float in document _source.

Wireless Protection

Change sophos.xg.clients_conn_ssid to a long in the mapping.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

elasticmachine · 2022-04-19T19:05:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-04-21T15:53:11.842+0000
Duration: 16 min 51 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28
Skipped	0
Total	28

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

Sync fix from elastic/beats#25250.

Fix inconsistency in that only some keys were being lowercase. And the way it was done before was not good because it modified the event.original value.

Set the event.code from the message ID (and remove sophos.xg.messageid). Set destination.domain from dstdomain in anti-virus log. Fix event.original handling. The original message was being modified by the pipeline. Now the raw message is stored in event.original without modification. Lower case network.transport as per ECS. Set event.reason from sophos.xg.reason (and remove sophos.xg.reason).

Sandstorm Set file.name for sandstorm events (there was a typo in the if condition). Set url.domain, destination.domain, and destination.ip for sandstorm web events. System Health Convert sophos.xg.collisions, sophos.xg.receiveddrops, and sophos.xg.transmitteddrops to float in _source. Wireless Protection Change sophos.xg.clients_conn_ssid to long in mapping.

Use dst_domainname to set destination.domain for Anti-Virus SMTP events. Use src_domainname to set source.domain for Anti-Virus SMTP events.

In 17.5 and earlier the sha1sum contained the sha1 checksum of the file being analyzed. In 18.0 and later the sha1sum contains the sha256 checksum of the file.

Use the URL scheme to derive the protocol.

This was a large amount of duplication causing increased event sizes.

[git-generate] cd packages/sophos elastic-package format elastic-package build elastic-package test pipeline -g

elasticmachine · 2022-04-20T21:45:53Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

packages/sophos/data_stream/xg/elasticsearch/ingest_pipeline/sandstorm.yml

Replaces `ctx?.` with `ctx.` in all of the sophos.xg pipelines.

…ations into feature/labels-cloud-ephemeral-components * 'feature/labels-cloud-ephemeral-components' of github.com:v1v/integrations: (23 commits) use branch standardise labels/tags Update .ci/Jenkinsfile [ci][terraform][aws] tags with metadata Bump github.com/elastic/elastic-package from 0.46.0 to 0.47.0 (elastic#3182) Spring boot package [Memory - data stream] (elastic#2979) [cisco_ise] Add Cisco ISE package (elastic#2855) [apache_spark][executor] Add Apache Spark package with Executor data stream (elastic#2943) [apache_spark][driver] Add Apache Spark package with Driver data stream (elastic#2945) cisco_duo: simplify grok expression for handling ports (elastic#3170) cisco_duo: fix handling of IP addresses with port numbers (elastic#3117) [sophos] Various improvements and log samples from 18.5 (elastic#3127) [cisco_asa] fix visualizations (elastic#3146) [apache_spark][application] Add Apache Spark package with Application data stream (elastic#2941) Fix pagination bug that skipped events when more than one page is present. (elastic#3140) Cis k8s name migration (elastic#3113) remove exported fields; these will be added to Kibana docs instead (elastic#3093) Exclude ecosystem as owner of subdirectories of packages (elastic#3132) [awsfargate] bump package version (elastic#3130) add first csp rule template (elastic#3081) ...

andrewkroh added bug Something isn't working, use only for issues enhancement New feature or request Team:Security-External Integrations Integration:sophos Sophos labels Apr 19, 2022

andrewkroh force-pushed the sophos/fix-field-names-uri-parts branch from 41584c6 to ff17251 Compare April 19, 2022 19:03

andrewkroh force-pushed the sophos/fix-field-names-uri-parts branch from ff17251 to d52441c Compare April 20, 2022 16:09

andrewkroh added 2 commits April 20, 2022 12:29

Remove space from trans_src_ip

c790137

Sync fix from elastic/beats#25250.

Lowercase sophos.xg fields

06ea003

Fix inconsistency in that only some keys were being lowercase. And the way it was done before was not good because it modified the event.original value.

andrewkroh force-pushed the sophos/fix-field-names-uri-parts branch from d52441c to fdf6e8d Compare April 20, 2022 16:30

andrewkroh added 5 commits April 20, 2022 14:54

Set domain name

1a630d7

Use dst_domainname to set destination.domain for Anti-Virus SMTP events. Use src_domainname to set source.domain for Anti-Virus SMTP events.

Consolidate geoip enrichment

7e9e105

Consolidate lowercase network.transport/protocol

0fdc468

andrewkroh force-pushed the sophos/fix-field-names-uri-parts branch from fdf6e8d to 112e782 Compare April 20, 2022 18:57

andrewkroh added 13 commits April 20, 2022 15:07

Consolidate related.ip processors

1778832

Consolidate related.user processing

3d39467

Consolidate related.hash and handle sha256 sandstorm file hashes

31c8f7b

In 17.5 and earlier the sha1sum contained the sha1 checksum of the file being analyzed. In 18.0 and later the sha1sum contains the sha256 checksum of the file.

Consolidate network.bytes/packets, add community_id

76f2d7d

Add {url,source,destination}.domain to related.hosts

346e161

Set event.duration (ns) for WAF events using reponsetime (us)

906adc2

Set network.protocol for anti-spam logs

e48b1e0

Set network.protocol for anti-virus logs

bee6afb

Format MAC addresses as per ECS

d2743be

Add event.reason mapping

f4bd720

Set network.protocol for Content Filtering

251be59

Use the URL scheme to derive the protocol.

Remove client/server mappings

5aa2303

This was a large amount of duplication causing increased event sizes.

Add reference link to xg syslog formats

1a92089

andrewkroh force-pushed the sophos/fix-field-names-uri-parts branch from 112e782 to 2bcd82c Compare April 20, 2022 19:10

andrewkroh force-pushed the sophos/fix-field-names-uri-parts branch from 2bcd82c to ebe5cfd Compare April 20, 2022 19:49

andrewkroh changed the title ~~[sophos] Lowercase names, Add uri_parts, Fix trans_src_ip~~ [sophos] Various improvements and log samples from 18.5 Apr 20, 2022

andrewkroh force-pushed the sophos/fix-field-names-uri-parts branch 2 times, most recently from da09ecf to a11cd4b Compare April 20, 2022 20:13

andrewkroh added 3 commits April 20, 2022 17:15

Update xg sample_event.json

980029d

Update files

e36a8a5

[git-generate] cd packages/sophos elastic-package format elastic-package build elastic-package test pipeline -g

Add changelog

8216183

andrewkroh force-pushed the sophos/fix-field-names-uri-parts branch from a11cd4b to 8216183 Compare April 20, 2022 21:16

andrewkroh marked this pull request as ready for review April 20, 2022 21:45

andrewkroh requested a review from a team as a code owner April 20, 2022 21:45

efd6 approved these changes Apr 20, 2022

View reviewed changes

packages/sophos/data_stream/xg/elasticsearch/ingest_pipeline/sandstorm.yml Outdated Show resolved Hide resolved

andrewkroh added 2 commits April 21, 2022 11:23

Remove null-safe operator when accessing ctx

624f50e

Replaces `ctx?.` with `ctx.` in all of the sophos.xg pipelines.

Sort ecs.yml and fields.yml by name

ab1fe1c

andrewkroh force-pushed the sophos/fix-field-names-uri-parts branch from fbd17f1 to ab1fe1c Compare April 21, 2022 15:52

andrewkroh merged commit 919fe81 into elastic:main Apr 21, 2022

This was referenced Apr 21, 2022

[Filebeat] sophos.xg - Handle new log field names elastic/beats#31388

Merged

[Filebeat] Add sophos XG log samples containing new field names elastic/beats#31038

Closed

andrewkroh mentioned this pull request May 2, 2022

[production] Promote many packages from snapshot elastic/package-storage#4302

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sophos] Various improvements and log samples from 18.5 #3127

[sophos] Various improvements and log samples from 18.5 #3127

andrewkroh commented Apr 19, 2022 •

edited

Loading

elasticmachine commented Apr 19, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Apr 20, 2022

[sophos] Various improvements and log samples from 18.5 #3127

[sophos] Various improvements and log samples from 18.5 #3127

Conversation

andrewkroh commented Apr 19, 2022 • edited Loading

What does this PR do?

Checklist

elasticmachine commented Apr 19, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Apr 20, 2022

andrewkroh commented Apr 19, 2022 •

edited

Loading

elasticmachine commented Apr 19, 2022 •

edited

Loading