[Rule Tuning] Timeline Templates For Windows and Linux #1892

terrancedejesus · 2022-03-28T15:50:28Z

Issues

Resolves #1879

Summary

The following rules have been chosen to test these timeline templates with.

Process Tracking

All information regarding steps taken can be found in this gdoc

terrancedejesus · 2022-03-28T15:56:08Z

Hosts File Modified

Replicated by moving hosts file located at C:\Windows\System32\drivers\etc\hosts to C:\Windows\System32\drivers\hosts.

Comprehensive Timeline Template Check

Timeline Added to Duplicate Rule

terrancedejesus · 2022-03-30T14:45:27Z

Rule Tuning

While testing spawn shells with python, I noticed Interactive Terminal Spawned via Python only accounts for python as a process name and not python3 which caused no alerts to generate. Updated in this branch since I am still tuning rules essentially.

Data

{
  "_index": ".ds-logs-endpoint.events.process-default-2022.03.28-000001",
  "_id": "Te9B238BDxJNEmTs_fNq",
  "_version": 1,
  "_score": 1,
  "_source": {
    "agent": {
      "id": "7bc5ce5c-de4a-4b79-81fa-45198c4d75b7",
      "type": "endpoint",
      "version": "8.2.0-SNAPSHOT"
    },
    "process": {
      "Ext": {
        "ancestry": [
          "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMjY0LTEzMjkzMTI0NDk5LjQwMDAwMDAw",
          "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMjYzLTEzMjkzMTI0NDk4Ljg5MDAwMDAwMA==",
          "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMTg5LTEzMjkzMTI0NDk2Ljk1MDAwMDAwMA==",
          "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTg5Mi0xMzI5MzEyMTcyMS4w",
          "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEtMTMyOTMxMjE3MDAuMA=="
        ]
      },
      "args": [
        "python3",
        "-c",
        "import pty; pty.spawn(\"/bin/sh\")"
      ],
      "parent": {
        "args": [
          "-bash"
        ],
        "name": "bash",
        "pid": 10264,
        "args_count": 0,
        "entity_id": "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMjY0LTEzMjkzMTI0NDk5LjQwMDAwMDAw",
        "command_line": "-bash",
        "executable": "/bin/bash"
      },
      "exit_code": 0,
      "name": "python3",
      "pid": 10285,
      "args_count": 0,
      "entity_id": "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMjg1LTEzMjkzMTI0NTkxLjUwMDAwMDAwMA==",
      "command_line": "python3 -c import pty; pty.spawn(\"/bin/sh\")",
      "executable": "/usr/bin/python3",
      "hash": {
        "sha1": "9b6d3f9d2129510da043a88f23c1d7dafdf8d104",
        "sha256": "2e833afd6114ff314d879486c01ec19d3d94ef6f8c808d193fb329c98c674097",
        "md5": "9822d4931aea8fb83f794d2d54c3c992"
      }
    },
    "message": "Endpoint process event",
    "@timestamp": "2022-03-30T14:36:38.2671779Z",
    "ecs": {
      "version": "1.11.0"
    },
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "endpoint.events.process"
    },
    "elastic": {
      "agent": {
        "id": "7bc5ce5c-de4a-4b79-81fa-45198c4d75b7"
      }
    },
    "host": {
      "hostname": "ubuntu-tdejesus",
      "os": {
        "Ext": {
          "variant": "Ubuntu"
        },
        "kernel": "5.13.0-1019-gcp #23~20.04.1-Ubuntu SMP Mon Mar 7 13:39:50 UTC 2022",
        "name": "Linux",
        "family": "ubuntu",
        "type": "linux",
        "version": "20.04.4",
        "platform": "ubuntu",
        "full": "Ubuntu 20.04.4"
      },
      "ip": [
        "127.0.0.1",
        "::1",
        "10.142.0.52",
        "fe80::4001:aff:fe8e:34"
      ],
      "name": "ubuntu-tdejesus",
      "id": "c78e287bbb0a5328e28b435c97de1cd0",
      "mac": [
        "42:01:0a:8e:00:34"
      ],
      "architecture": "x86_64"
    },
    "event": {
      "agent_id_status": "verified",
      "sequence": 62669,
      "ingested": "2022-03-30T14:36:49Z",
      "created": "2022-03-30T14:36:38.2671779Z",
      "kind": "event",
      "module": "endpoint",
      "action": "end",
      "id": "MYFVxCublyiVcrVQ++++01D+",
      "category": [
        "process"
      ],
      "type": [
        "end"
      ],
      "dataset": "endpoint.events.process"
    },
    "user": {
      "Ext": {
        "real": {
          "id": 1022
        }
      },
      "id": 1022
    },
    "group": {
      "Ext": {
        "real": {
          "id": 1023
        }
      },
      "id": 1023
    }
  },
  "fields": {
    "process.hash.md5": [
      "9822d4931aea8fb83f794d2d54c3c992"
    ],
    "host.os.full.text": [
      "Ubuntu 20.04.4"
    ],
    "process.command_line.caseless": [
      "python3 -c import pty; pty.spawn(\"/bin/sh\")"
    ],
    "event.category": [
      "process"
    ],
    "process.name.text": [
      "python3"
    ],
    "host.os.name.text": [
      "Linux"
    ],
    "host.os.full": [
      "Ubuntu 20.04.4"
    ],
    "process.parent.command_line": [
      "-bash"
    ],
    "process.parent.name": [
      "bash"
    ],
    "process.parent.pid": [
      10264
    ],
    "process.hash.sha256": [
      "2e833afd6114ff314d879486c01ec19d3d94ef6f8c808d193fb329c98c674097"
    ],
    "host.hostname": [
      "ubuntu-tdejesus"
    ],
    "process.pid": [
      10285
    ],
    "host.mac": [
      "42:01:0a:8e:00:34"
    ],
    "elastic.agent.id": [
      "7bc5ce5c-de4a-4b79-81fa-45198c4d75b7"
    ],
    "process.parent.entity_id": [
      "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMjY0LTEzMjkzMTI0NDk5LjQwMDAwMDAw"
    ],
    "host.os.version": [
      "20.04.4"
    ],
    "host.os.name": [
      "Linux"
    ],
    "host.name": [
      "ubuntu-tdejesus"
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "event.kind": [
      "event"
    ],
    "user.id": [
      "1022"
    ],
    "host.os.type": [
      "linux"
    ],
    "process.Ext.ancestry": [
      "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMjY0LTEzMjkzMTI0NDk5LjQwMDAwMDAw",
      "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMjYzLTEzMjkzMTI0NDk4Ljg5MDAwMDAwMA==",
      "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMTg5LTEzMjkzMTI0NDk2Ljk1MDAwMDAwMA==",
      "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTg5Mi0xMzI5MzEyMTcyMS4w",
      "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEtMTMyOTMxMjE3MDAuMA=="
    ],
    "user.Ext.real.id": [
      "1022"
    ],
    "data_stream.type": [
      "logs"
    ],
    "process.parent.args_count": [
      0
    ],
    "host.architecture": [
      "x86_64"
    ],
    "process.name": [
      "python3"
    ],
    "agent.id": [
      "7bc5ce5c-de4a-4b79-81fa-45198c4d75b7"
    ],
    "process.parent.executable.text": [
      "/bin/bash"
    ],
    "ecs.version": [
      "1.11.0"
    ],
    "event.created": [
      "2022-03-30T14:36:38.267Z"
    ],
    "agent.version": [
      "8.2.0-SNAPSHOT"
    ],
    "host.os.family": [
      "ubuntu"
    ],
    "process.command_line.text": [
      "python3 -c import pty; pty.spawn(\"/bin/sh\")"
    ],
    "group.id": [
      "1023"
    ],
    "process.entity_id": [
      "N2JjNWNlNWMtZGU0YS00Yjc5LTgxZmEtNDUxOThjNGQ3NWI3LTEwMjg1LTEzMjkzMTI0NTkxLjUwMDAwMDAwMA=="
    ],
    "process.parent.name.text": [
      "bash"
    ],
    "host.ip": [
      "127.0.0.1",
      "::1",
      "10.142.0.52",
      "fe80::4001:aff:fe8e:34"
    ],
    "event.sequence": [
      62669
    ],
    "process.executable.caseless": [
      "/usr/bin/python3"
    ],
    "agent.type": [
      "endpoint"
    ],
    "process.executable.text": [
      "/usr/bin/python3"
    ],
    "event.module": [
      "endpoint"
    ],
    "host.os.kernel": [
      "5.13.0-1019-gcp #23~20.04.1-Ubuntu SMP Mon Mar 7 13:39:50 UTC 2022"
    ],
    "host.os.full.caseless": [
      "ubuntu 20.04.4"
    ],
    "process.name.caseless": [
      "python3"
    ],
    "host.id": [
      "c78e287bbb0a5328e28b435c97de1cd0"
    ],
    "process.exit_code": [
      0
    ],
    "process.executable": [
      "/usr/bin/python3"
    ],
    "process.parent.name.caseless": [
      "bash"
    ],
    "process.parent.executable.caseless": [
      "/bin/bash"
    ],
    "process.parent.executable": [
      "/bin/bash"
    ],
    "process.parent.command_line.text": [
      "-bash"
    ],
    "process.args_count": [
      0
    ],
    "data_stream.namespace": [
      "default"
    ],
    "process.args": [
      "python3",
      "-c",
      "import pty; pty.spawn(\"/bin/sh\")"
    ],
    "message": [
      "Endpoint process event"
    ],
    "host.os.Ext.variant": [
      "Ubuntu"
    ],
    "process.parent.args": [
      "-bash"
    ],
    "group.Ext.real.id": [
      "1023"
    ],
    "event.action": [
      "end"
    ],
    "event.ingested": [
      "2022-03-30T14:36:49.000Z"
    ],
    "@timestamp": [
      "2022-03-30T14:36:38.267Z"
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "process.parent.command_line.caseless": [
      "-bash"
    ],
    "event.type": [
      "end"
    ],
    "data_stream.dataset": [
      "endpoint.events.process"
    ],
    "process.command_line": [
      "python3 -c import pty; pty.spawn(\"/bin/sh\")"
    ],
    "process.hash.sha1": [
      "9b6d3f9d2129510da043a88f23c1d7dafdf8d104"
    ],
    "event.id": [
      "MYFVxCublyiVcrVQ++++01D+"
    ],
    "host.os.name.caseless": [
      "linux"
    ],
    "event.dataset": [
      "endpoint.events.process"
    ]
  }
}

…via Python rule

terrancedejesus · 2022-03-30T15:14:29Z

Interactive Terminal Spawned via Python

Comprehensive Timeline Template Check

Timeline Template Added to Duplicate Rule

terrancedejesus · 2022-03-30T15:23:21Z

Findings/Notes/Questions

Build only checks for Generic

Looks like the builds fail if you try to use a comprehensive timeline template.

marshmallow.exceptions.ValidationError: {'rule': [ValidationError({'timeline_id': ['Must be one of: db366523-f1c6-4c1f-8731-6ce5ed9e5717, 91832785-286d-4ebe-b884-1a208d111a70, 76e52245-7519-4251-91ab-262fb1a1728c, 495ad7a7-316e-4544-8a0f-9c098daee76e.'], 'timeline_title': ['Must be one of: Generic Endpoint Timeline, Generic Network Timeline, Generic Process Timeline, Generic Threat Match Timeline.']}), ValidationError({'timeline_id': ['Must be one of: db366523-f1c6-4c1f-8731-6ce5ed9e5717, 91832785-286d-4ebe-b884-1a208d111a70, 76e52245-7519-4251-91ab-262fb1a1728c, 495ad7a7-316e-4544-8a0f-9c098daee76e.'], 'threshold': ['Missing data for required field.'], 'language': ['Must be one of: kuery, lucene.'], 'timeline_title': ['Must be one of: Generic Endpoint Timeline, Generic Network Timeline, Generic Process Timeline, Generic Threat Match Timeline.'], 'type': ['Must be equal to threshold.']}), ValidationError({'timeline_id': ['Must be one of: db366523-f1c6-4c1f-8731-6ce5ed9e5717, 91832785-286d-4ebe-b884-1a208d111a70, 76e52245-7519-4251-91ab-262fb1a1728c, 495ad7a7-316e-4544-8a0f-9c098daee76e.'], 'threat_index': ['Missing data for required field.'], 'language': ['Must be one of: kuery, lucene.'], 'threat_mapping': ['Missing data for required field.'], 'timeline_title': ['Must be one of: Generic Endpoint Timeline, Generic Network Timeline, Generic Process Timeline, Generic Threat Match Timeline.'], 'type': ['Must be equal to threat_match.']}), ValidationError({'timeline_id': ['Must be one of: db366523-f1c6-4c1f-8731-6ce5ed9e5717, 91832785-286d-4ebe-b884-1a208d111a70, 76e52245-7519-4251-91ab-262fb1a1728c, 495ad7a7-316e-4544-8a0f-9c098daee76e.'], 'machine_learning_job_id': ['Missing data for required field.'], 'timeline_title': ['Must be one of: Generic Endpoint Timeline, Generic Network Timeline, Generic Process Timeline, Generic Threat Match Timeline.'], 'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'index': ['Unknown field.'], 'query': ['Unknown field.'], 'language': ['Unknown field.']}), ValidationError({'timeline_id': ['Must be one of: db366523-f1c6-4c1f-8731-6ce5ed9e5717, 91832785-286d-4ebe-b884-1a208d111a70, 76e52245-7519-4251-91ab-262fb1a1728c, 495ad7a7-316e-4544-8a0f-9c098daee76e.'], 'language': ['Must be one of: kuery, lucene.'], 'timeline_title': ['Must be one of: Generic Endpoint Timeline, Generic Network Timeline, Generic Process Timeline, Generic Threat Match Timeline.'], 'type': ['Must be equal to query.']})]}

Missing Generic File and Registry Timeline Templates

Looks like we are missing a Generic File Timeline and Generic Registry Timeline template or maybe that was intentional?

How to check if template fits for the rule

The best way to verify this is to use Security > Timelines > Templates > Select your template and then add the additional parameters to the query to see if the document shows up.

Automated Generation of the Template References on Build

If these templates are going to stay this generic, then there we may be able to auto-generate the template_id and template_name values in the TOML files on build or at least to handle these in bulk. The logic would need to be determined, but takes the query type and query contents and compares them with what is expected in a template then assigns that ID and name. @brokensound77 thoughts? If so we may be able to bulk update these instead of one at a time.

Generic Threat Match Timeline Template

Not exactly sure where this data resides or how to check the list it is matching against. I assume this is a backlist of IoCs that we match against.

Generic Network Timeline Template

Currently this is based on source.ip and destination.ip. Could there be times when one or the other are NULL or the hostname is given instead of being an IP? All together, should this just match the others and look for event.category : (network or network_traffic) to account for any one offs.

terrancedejesus · 2022-03-30T15:57:48Z

Suspicious CertUtil Commands

Generic Process Timeline Template Check

terrancedejesus · 2022-03-30T16:09:30Z

RDP (Remote Desktop Protocol) from the Internet

Generic Network Timeline Template Check

terrancedejesus · 2022-03-30T16:12:45Z

LSASS Memory Dump Creation

Generic Endpoint Timeline Template Check

terrancedejesus · 2022-03-30T16:20:49Z

Adding Hidden File Attribute via Attrib

Generic Process Timeline Template Check

terrancedejesus · 2022-03-30T18:34:46Z

Telnet Port Activity

Generic Network Timeline Template Check

terrancedejesus · 2022-03-30T19:30:46Z

ImageLoad via Windows Update Auto Update Client

Generic Process Timeline Template Check

brokensound77 · 2022-03-31T03:16:48Z

Looks like the builds fail if you try to use a comprehensive timeline template

Just update this with the new comprehensive template IDs and names and it will pass tests

detection-rules/detection_rules/schemas/definitions.py

Lines 40 to 45 in a3d7427

    
           TIMELINE_TEMPLATES: Final[dict] = { 
        
               'db366523-f1c6-4c1f-8731-6ce5ed9e5717': 'Generic Endpoint Timeline', 
        
               '91832785-286d-4ebe-b884-1a208d111a70': 'Generic Network Timeline', 
        
               '76e52245-7519-4251-91ab-262fb1a1728c': 'Generic Process Timeline', 
        
               '495ad7a7-316e-4544-8a0f-9c098daee76e': 'Generic Threat Match Timeline' 
        
           }

… was updated

…ck comments and versions

…ows-and-linux

rules/linux/execution_python_tty_shell.toml

Co-authored-by: Justin Ibarra <[email protected]>

brokensound77

LGTM once passing (and pip files removed)

Removing pipfile

deleting pipfile.lock

rules/windows/execution_command_shell_started_by_svchost.toml

updating title

added comprehensive file timeline to Hosts File Modified rule

57ef02d

terrancedejesus added OS: Linux Rule: Tuning tweaking or tuning an existing rule OS: Windows windows related rules v8.2.0 labels Mar 28, 2022

terrancedejesus self-assigned this Mar 28, 2022

github-actions bot added the backport: auto label Mar 28, 2022

terrancedejesus requested review from brokensound77 and DefSecSentinel March 28, 2022 15:56

added Comprehensive Process Timeline to Interactive Terminal Spawned …

351537e

…via Python rule

botelastic bot added the Domain: Endpoint label Mar 30, 2022

updated rules to have generic instead of comprehensive

4d54761

terrancedejesus added 2 commits March 30, 2022 15:32

updated several rules with timeline ID and timeline title values

c6a41d6

changed updated_date for threat intel fleet integrations

2a6e8c3

added missing templates to timeline_templates dict in definitions.py

40f9acb

botelastic bot added python Internal python for the repository schema labels Mar 31, 2022

terrancedejesus added 2 commits March 31, 2022 10:33

added comprehensive timeline templates to alerts after definitions.py…

f501cfe

… was updated

updated rules with comprehensive timeline templates and added min sta…

03ba31d

…ck comments and versions

terrancedejesus and others added 2 commits March 31, 2022 12:43

removing timeline template changes which is tracked in #1904

9299171

Merge branch 'main' into 1879-rule-tuning-timeline-templates-for-wind…

403548a

…ows-and-linux

brokensound77 reviewed Apr 1, 2022

View reviewed changes

rules/linux/execution_python_tty_shell.toml Outdated Show resolved Hide resolved

Update rules/linux/execution_python_tty_shell.toml

b493be8

Co-authored-by: Justin Ibarra <[email protected]>

brokensound77 approved these changes Apr 1, 2022

View reviewed changes

terrancedejesus added 2 commits April 1, 2022 13:12

Delete Pipfile

d007e20

Removing pipfile

Delete Pipfile.lock

e37a0b6

deleting pipfile.lock

terrancedejesus commented Apr 1, 2022

View reviewed changes

rules/windows/execution_command_shell_started_by_svchost.toml Outdated Show resolved Hide resolved

Update rules/windows/execution_command_shell_started_by_svchost.toml

896387d

updating title

terrancedejesus requested review from Mikaayenson and w0rk3r April 1, 2022 17:22

w0rk3r approved these changes Apr 1, 2022

View reviewed changes

terrancedejesus merged commit 93edc44 into main Apr 1, 2022

terrancedejesus deleted the 1879-rule-tuning-timeline-templates-for-windows-and-linux branch April 1, 2022 17:44

brokensound77 mentioned this pull request Apr 13, 2022

7.13-8.1 branches in failing state #1924

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Rule Tuning] Timeline Templates For Windows and Linux #1892

[Rule Tuning] Timeline Templates For Windows and Linux #1892

terrancedejesus commented Mar 28, 2022 •

edited

Loading

terrancedejesus commented Mar 28, 2022 •

edited

Loading

terrancedejesus commented Mar 30, 2022 •

edited by brokensound77

Loading

terrancedejesus commented Mar 30, 2022 •

edited

Loading

terrancedejesus commented Mar 30, 2022 •

edited by brokensound77

Loading

terrancedejesus commented Mar 30, 2022

terrancedejesus commented Mar 30, 2022

terrancedejesus commented Mar 30, 2022

terrancedejesus commented Mar 30, 2022

terrancedejesus commented Mar 30, 2022 •

edited

Loading

terrancedejesus commented Mar 30, 2022

brokensound77 commented Mar 31, 2022 •

edited

Loading

brokensound77 left a comment

[Rule Tuning] Timeline Templates For Windows and Linux #1892

[Rule Tuning] Timeline Templates For Windows and Linux #1892

Conversation

terrancedejesus commented Mar 28, 2022 • edited Loading

Issues

Summary

Process Tracking

terrancedejesus commented Mar 28, 2022 • edited Loading

Hosts File Modified

Comprehensive Timeline Template Check

Timeline Added to Duplicate Rule

terrancedejesus commented Mar 30, 2022 • edited by brokensound77 Loading

Rule Tuning

terrancedejesus commented Mar 30, 2022 • edited Loading

Interactive Terminal Spawned via Python

Comprehensive Timeline Template Check

Timeline Template Added to Duplicate Rule

terrancedejesus commented Mar 30, 2022 • edited by brokensound77 Loading

Findings/Notes/Questions

Build only checks for Generic

Missing Generic File and Registry Timeline Templates

How to check if template fits for the rule

Automated Generation of the Template References on Build

Generic Threat Match Timeline Template

Generic Network Timeline Template

terrancedejesus commented Mar 30, 2022

Suspicious CertUtil Commands

Generic Process Timeline Template Check

terrancedejesus commented Mar 30, 2022

RDP (Remote Desktop Protocol) from the Internet

Generic Network Timeline Template Check

terrancedejesus commented Mar 30, 2022

LSASS Memory Dump Creation

Generic Endpoint Timeline Template Check

terrancedejesus commented Mar 30, 2022

Adding Hidden File Attribute via Attrib

Generic Process Timeline Template Check

terrancedejesus commented Mar 30, 2022 • edited Loading

Telnet Port Activity

Generic Network Timeline Template Check

terrancedejesus commented Mar 30, 2022

ImageLoad via Windows Update Auto Update Client

Generic Process Timeline Template Check

brokensound77 commented Mar 31, 2022 • edited Loading

brokensound77 left a comment

Choose a reason for hiding this comment

terrancedejesus commented Mar 28, 2022 •

edited

Loading

terrancedejesus commented Mar 28, 2022 •

edited

Loading

terrancedejesus commented Mar 30, 2022 •

edited by brokensound77

Loading

terrancedejesus commented Mar 30, 2022 •

edited

Loading

terrancedejesus commented Mar 30, 2022 •

edited by brokensound77

Loading

terrancedejesus commented Mar 30, 2022 •

edited

Loading

brokensound77 commented Mar 31, 2022 •

edited

Loading