Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[sophos] Various improvements and log samples from 18.5 #3127

Merged
merged 25 commits into from
Apr 21, 2022
Merged

[sophos] Various improvements and log samples from 18.5 #3127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c790137
Remove space from trans_src_ip
andrewkroh Apr 19, 2022
06ea003
Lowercase sophos.xg fields
andrewkroh Apr 19, 2022
882c9d0
Add Sophos Firewall 18.5 samples
andrewkroh Apr 19, 2022
5b8713c
Various enhancements, fixes, test data
andrewkroh Apr 20, 2022
1a630d7
Set domain name
andrewkroh Apr 20, 2022
7e9e105
Consolidate geoip enrichment
andrewkroh Apr 20, 2022
0fdc468
Consolidate lowercase network.transport/protocol
andrewkroh Apr 20, 2022
1778832
Consolidate related.ip processors
andrewkroh Apr 20, 2022
3d39467
Consolidate related.user processing
andrewkroh Apr 20, 2022
31c8f7b
Consolidate related.hash and handle sha256 sandstorm file hashes
andrewkroh Apr 20, 2022
76f2d7d
Consolidate network.bytes/packets, add community_id
andrewkroh Apr 20, 2022
346e161
Add {url,source,destination}.domain to related.hosts
andrewkroh Apr 20, 2022
906adc2
Set event.duration (ns) for WAF events using reponsetime (us)
andrewkroh Apr 20, 2022
e48b1e0
Set network.protocol for anti-spam logs
andrewkroh Apr 20, 2022
bee6afb
Set network.protocol for anti-virus logs
andrewkroh Apr 20, 2022
d2743be
Format MAC addresses as per ECS
andrewkroh Apr 20, 2022
f4bd720
Add event.reason mapping
andrewkroh Apr 20, 2022
251be59
Set network.protocol for Content Filtering
andrewkroh Apr 20, 2022
5aa2303
Remove client/server mappings
andrewkroh Apr 20, 2022
1a92089
Add reference link to xg syslog formats
andrewkroh Apr 19, 2022
980029d
Update xg sample_event.json
andrewkroh Apr 20, 2022
e36a8a5
Update files
andrewkroh Apr 19, 2022
8216183
Add changelog
andrewkroh Apr 19, 2022
624f50e
Remove null-safe operator when accessing ctx
andrewkroh Apr 21, 2022
ab1fe1c
Sort ecs.yml and fields.yml by name
andrewkroh Apr 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions packages/sophos/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Currently it accepts logs in syslog format or from a file for the following devi

To configure a remote syslog destination, please reference the [SophosXG/SFOS Documentation](https://community.sophos.com/kb/en-us/123184).

The syslog format choosen should be `Default`.
The syslog format chosen should be `Default`.

## Compatibility

Expand All @@ -26,7 +26,9 @@ The `utm` dataset collects Astaro Security Gateway logs.

### XG log

This is the Sophos `xg` dataset.
This is the Sophos `xg` dataset. Reference information about the log formats
can be found in the [Sophos syslog guide](
https://docs.sophos.com/nsg/sophos-firewall/18.5/PDF/SF%20syslog%20guide%2018.5.pdf).

{{event "xg"}}

Expand Down
29 changes: 29 additions & 0 deletions packages/sophos/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,33 @@
# newer versions go on top
- version: "2.0.0"
changes:
- description: Remove space from sophos.xg.trans_src_ip field.
type: bugfix
link: https://github.com/elastic/integrations/pull/3127
- description: Do not modify event.original.
type: bugfix
link: https://github.com/elastic/integrations/pull/3127
- description: Populate `url.*` fields based on `sophos.xg.url`.
type: enhancement
link: https://github.com/elastic/integrations/pull/3127
- description: Rename `sophos.xg.reason` to `event.reason` (ECS).
type: enhancement
link: https://github.com/elastic/integrations/pull/3127
- description: Lowercase `network.transport` as per ECS.
type: bugfix
link: https://github.com/elastic/integrations/pull/3127
- description: Format `source.mac` and `destination.mac` as per ECS.
type: bugfix
link: https://github.com/elastic/integrations/pull/3127
- description: Set the `event.code` from the message ID (and remove `sophos.xg.message_id`).
type: enhancement
link: https://github.com/elastic/integrations/pull/3127
- description: Add `network.community_id`.
type: enhancement
link: https://github.com/elastic/integrations/pull/3127
- description: Reduce event size by removing `client` and `server` fields that are clones of `source` and `destination`, respectively.
type: breaking-change
link: https://github.com/elastic/integrations/pull/3127
- version: "1.2.3"
changes:
- description: Update pipelines to parse new fields
Expand Down
19 changes: 19 additions & 0 deletions packages/sophos/data_stream/xg/_dev/test/pipeline/test-sophos-18.5-anti-spam.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
device="SFW" date=2017-01-31 time=18:28:25 timezone="IST" device_name="CR750iNG-XP" device_id=C44313350024-P29PUA log_id=041107413001 log_type="Anti-Spam" log_component="SMTP" log_subtype="Spam" status="" priority=Warning fw_rule_id=0 user_name="gaurav" av_policy_name="Gaurav235" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="RPD Spam Test: Spam" mailid="c000000b-1485867502" mailsize=400 spamaction="DROP" reason="" src_domainname="iview.com" dst_domainname="" src_ip=10.198.47.71 src_country_code=R1 dst_ip=10.198.233.61 dst_country_code=R1 protocol="TCP" src_port=22258 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Spam"
device="SFW" date=2018-06-06 time=10:41:29 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041108413002 log_type="Anti-Spam" log_component="SMTP" log_subtype="Probable Spam" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="postman" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="[SPAM] RPD Spam test: Bulk" mailid="c0000006-1528261885" mailsize=438 spamaction="WARN" reason="Mail detected as PROBABLE SPAM." src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.16.204 dst_country_code=R1 protocol="TCP" src_port=56341 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Spam"
device="SFW" date=2017-01-31 time=18:36:22 timezone="IST" device_name="CR750iNG-XP" device_id=C44313350024-P29PUA log_id=041105613003 log_type="Anti-Spam" log_component="SMTP" log_subtype="Clean" status="" priority=Information fw_rule_id=0 user_name="gaurav" av_policy_name="None" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="EMAIL" mailid="<[email protected]>" mailsize=398 spamaction="Accept" reason="" src_domainname="iview.com" dst_domainname="" src_ip=10.198.47.71 src_country_code=R1 dst_ip=10.198.233.61 dst_country_code=R1 protocol="TCP" src_port=22477 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-06 time=11:08:08 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041108413004 log_type="Anti-Spam" log_component="SMTP" log_subtype="Probable Spam" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="postman" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="Test RBL email" mailid="c0000008-1528263488" mailsize=433 spamaction="DROP" reason="Sender IP address is blacklisted." src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.17.121 dst_country_code=R1 protocol="TCP" src_port=57854 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="RBL"
device="SFW" date=2017-01-31 time=18:34:41 timezone="IST" device_name="CR750iNG-XP" device_id=C44313350024-P29PUA log_id=041113413005 log_type="Anti-Spam" log_component="SMTP" log_subtype="Outbound Spam" status="" priority=Warning fw_rule_id=0 user_name="gaurav" av_policy_name="Gaurav123" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="RPD Spam Test: Spam" mailid="<[email protected]>" mailsize=405 spamaction="Accept" reason="" src_domainname="iview.com" dst_domainname="" src_ip=10.198.47.71 src_country_code=R1 dst_ip=10.198.233.61 dst_country_code=R1 protocol="TCP" src_port=22420 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Spam"
device="SFW" date=2018-06-06 time=11:10:11 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041114413006 log_type="Anti-Spam" log_component="SMTP" log_subtype="Outbound Probable Spam" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="rule 8" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="RPD Spam test: Bulk" mailid="<c63b1eb2-1c17-73ac-fcc3- [email protected]>" mailsize=439 spamaction="Drop" reason="Mail detected as OUTBOUND PROBABLE SPAM." src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.234.240 dst_country_code=R1 protocol="TCP" src_port=58043 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Spam"
device="SFW" date=2018-06-06 time=12:50:07 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041121613009 log_type="Anti-Spam" log_component="SMTP" log_subtype="DLP" status="" priority=Information fw_rule_id=0 user_name="" av_policy_name="postman" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="Fwd: TESt" mailid="c0000002-1528269606" mailsize=5041 spamaction="DROP" reason="Email containing confidential data detected. Relevant Data Protection Policy applied." src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.17.121 dst_country_code=R1 protocol="TCP" src_port=60134 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="DLP"
device="SFW" date=2018-06-06 time=12:51:34 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041122613010 log_type="Anti-Spam" log_component="SMTP" log_subtype="SPX" status="" priority=Information fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="[secure:pankhil]" mailid="c0000003-1528269693" mailsize=442 spamaction="Accept" reason="SPX Template of type Specified by Sender successfully applied on Email." src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.16.204 dst_country_code=R1 protocol="TCP" src_port=60298 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-06 time=12:52:49 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041122613011 log_type="Anti-Spam" log_component="SMTP" log_subtype="SPX" status="" priority=Information fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="Test failed" mailid="c0000004-1528269769" mailsize=431 spamaction="REJECT" reason="Email could not be SPX- encrypted because password was not found in the Email subject." src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.16.204 dst_country_code=R1 protocol="TCP" src_port=60305 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-06 time=12:53:39 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041123413012 log_type="Anti-Spam" log_component="SMTP" log_subtype="Dos" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="" to_email_address="" email_subject="" mailid="" mailsize=0 spamaction="TMPREJECT" reason="SMTP DoS" src_domainname="" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.17.121 dst_country_code=R1 protocol="TCP" src_port=60392 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2017-01-31 time=15:46:45 timezone="IST" device_name="CR750iNG-XP" device_id=C44313350024-P29PUA log_id=041101613013 log_type="Anti-Spam" log_component="SMTP" log_subtype="Allowed" status="" priority=Information fw_rule_id=0 user_name="gaurav" av_policy_name="Gaurav235" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="GP235" mailid="c000000a-1485857789" mailsize=391 spamaction="SANDSTORM ALLOW" reason="Mail is marked Clean by Sophos Sandstorm." src_domainname="iview.com" dst_domainname="" src_ip=10.198.47.71 src_country_code=R1 dst_ip=10.198.233.61 dst_country_code=R1 protocol="TCP" src_port=11255 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-06 time=12:56:53 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041102413014 log_type="Anti-Spam" log_component="SMTP" log_subtype="Denied" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="postman" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="Fwd: test sand" mailid="c0000008-1528270010" mailsize=419835 spamaction="DROP" reason="Email is marked Malicious by Sophos Sandstorm." src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.17.121 dst_country_code=R1 protocol="TCP" src_port=60608 dst_port=25 sent_bytes=0 recv_bytes=0
device="SFW" date=2017-01-31 time=18:31:11 timezone="IST" device_name="CR750iNG-XP" device_id=C44313350024-P29PUA log_id=041207414001 log_type="Anti-Spam" log_component="POP3" log_subtype="Spam" status="" priority=Warning fw_rule_id=0 user_name="gaurav" av_policy_name="GauravPatel" from_email_address="[email protected]" to_email_address="gaurav2@iview. com" email_subject="RPD Spam Test: Spam" mailid="<[email protected]>" mailsize=574 spamaction="Accept" reason="" src_domainname="iview.com" dst_domainname="iview.com" src_ip=10.198.47.71 src_country_code=R1 dst_ip=10.198.233.61 dst_country_code=R1 protocol="TCP" src_port=22333 dst_port=110 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-06 time=12:59:01 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=046108414002 log_type="Anti-Spam" log_component="POPS" log_subtype="Probable Spam" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="pop8" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="RPD Spam test: Bulk" mailid="<[email protected]>" mailsize=0 spamaction="Change Subject" reason="Mail detected as PROBABLE SPAM" src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code= dst_ip=10.198.234.240 dst_country_code= protocol="TCP" src_port=60742 dst_port=995 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-06 time=13:00:34 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=046105614003 log_type="Anti-Spam" log_component="POPS" log_subtype="Clean" status="" priority=Information fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="Test clean" mailid="<[email protected]>" mailsize=0 spamaction="Accept" reason="Mail is Clean" src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code= dst_ip=10.198.234.240 dst_country_code= protocol="TCP" src_port=60757 dst_port=995 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-06 time=13:01:42 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=046207415001 log_type="Anti-Spam" log_component="IMAPS" log_subtype="Spam" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="RPD Spam test: Spam" mailid="<[email protected]>" mailsize=0 spamaction="Accept" reason="Mail detected as SPAM" src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code= dst_ip=10.198.234.240 dst_country_code= protocol="TCP" src_port=58595 dst_port=993 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-06 time=13:02:54 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=046208415002 log_type="Anti-Spam" log_component="IMAPS" log_subtype="Probable Spam" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="RPD Spam test: Bulk" mailid="<[email protected]>" mailsize=0 spamaction="Accept" reason="Mail detected as PROBABLE SPAM" src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code= dst_ip=10.198.234.240 dst_country_code= protocol="TCP" src_port=58595 dst_port=993 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-06 time=13:03:58 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=046205615003 log_type="Anti-Spam" log_component="IMAPS" log_subtype="Clean" status="" priority=Information fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="Clean email" mailid="<[email protected]>" mailsize=0 spamaction="Accept" reason="Mail is Clean" src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code= dst_ip=10.198.234.240 dst_country_code= protocol="TCP" src_port=58595 dst_port=993 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
device="SFW" date=2018-06-05 time=19:11:26 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041101618035 log_type="Anti-Spam" log_component="SMTP" log_subtype="Allowed" status="" priority=Information fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="[email protected]" to_email_address="[email protected]" email_subject="dd" mailid="c0000005-1528206082" mailsize=421 spamaction="DELIVERED" reason="Email has been delivered to recipient(s)." src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.16.204 dst_country_code=R1 protocol="TCP" src_port=61636 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
Loading