x-pack/auditbeat/module/system/socket: stratify sockets and connections by PID #28713

efd6 · 2021-10-29T07:43:42Z

What does this PR do?

Previously it was possible for two processes to be assigned to the same socket and so flow due to socket sharing. This change prevents that by keying on PID as well as socket and network address.

Why is it important?

The previous state of the code made determination of the processes involved in network flows unreliable in some circumstances reducing its utility significantly in those cases.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Test locally as described below.

How to test this PR locally

Test the efficacy of the change by running heavy network traffic with the possibility of address collisions. I achieved this by running multiple (10s) concurrent instances of nmap doing repeated complete port scans while also running opensnitch. This usually results in a failure within a couple of minutes in the failing case.

Related issues

Closes auditbeat/socket returns incorrect process information #17165

Use cases

N/A

Screenshots

N/A

Logs

N/A

elasticmachine · 2021-10-29T07:43:44Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2021-10-29T07:43:46Z

This pull request does not have a backport label. Could you fix it @efd6? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

…ns by PID Previously it was possible for two processes to be assigned to the same socket and so flow due to socket sharing. This change prevents that by keying on PID as well as socket and network address.

elasticmachine · 2021-10-29T08:17:18Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-15T18:54:39.467+0000
Duration: 37 min 28 sec
Commit: 1917df5

Test stats 🧪

Test	Results
Failed	1
Passed	234
Skipped	33
Total	268

Test errors

Expand to view the tests failures

`Build&Test / x-pack/auditbeat-build / test_dns_long_request – x-pack.auditbeat.tests.system.test_system_socket.Test`

Expand to view the error details

 Failed: Timeout >90.0s

Expand to view the stacktrace

 self = <test_system_socket.Test testMethod=test_dns_long_request>
test = <test_system_socket.DNSTestCase object at 0x7f3203a76280>

    def execute(self, test):
        cleanup = self.noop
        if hasattr(test, 'cleanup'):
            cleanup = test.cleanup
    
        if hasattr(test, 'setup'):
            test.setup()
    
        try:
            test.run()
        except Exception:
            cleanup()
            raise
    
        cleanup()
    
        try:
            self.wait_until(lambda: self.output_lines() > 0, max_timeout=15)
        except Exception as e:
            raise Exception('No output received form Auditbeat').with_traceback(sys.exc_info()[2])
    
        expected = test.expected()
        found = False
        try:
>           self.wait_until(lambda: expected.match(self.flattened_output()), max_timeout=15)

tests/system/test_system_socket.py:187: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_system_socket.Test testMethod=test_dns_long_request>
cond = <function Test.execute.<locals>.<lambda> at 0x7f320287eee0>
max_timeout = 15, poll_interval = 0.1, name = 'cond'

    def wait_until(self, cond, max_timeout=10, poll_interval=0.1, name="cond"):
        """
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
>               raise TimeoutError("Timeout waiting for '{}' to be true. ".format(name) +
                                   "Waited {} seconds.".format(max_timeout))
E               beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 15 seconds.

../../libbeat/tests/system/beat/beat.py:362: TimeoutError

During handling of the above exception, another exception occurred:

self = <test_system_socket.Test testMethod=test_dns_long_request>
test = <test_system_socket.DNSTestCase object at 0x7f3203a76280>
extra_conf = {'socket.flow_inactive_timeout': '2s'}

    def with_runner(self, test, extra_conf=dict()):
        enable_ipv6_loopback()
        conf = {
            "socket.flow_inactive_timeout": "2s",
            "socket.flow_termination_timeout": "5s",
            "socket.development_mode": "true",
        }
        conf.update(extra_conf)
        self.render_config_template(modules=[{
            "name": "system",
            "datasets": ["socket"],
            "extras": conf,
        }])
        proc = self.start_beat()
        try:
            try:
                self.wait_until(lambda: self.log_contains('system/socket dataset is running.'),
                                max_timeout=60)
            except Exception as e:
                raise Exception('Auditbeat failed to start start').with_traceback(sys.exc_info()[2])
>           self.execute(test)

tests/system/test_system_socket.py:156: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_system_socket.Test testMethod=test_dns_long_request>
test = <test_system_socket.DNSTestCase object at 0x7f3203a76280>

    def execute(self, test):
        cleanup = self.noop
        if hasattr(test, 'cleanup'):
            cleanup = test.cleanup
    
        if hasattr(test, 'setup'):
            test.setup()
    
        try:
            test.run()
        except Exception:
            cleanup()
            raise
    
        cleanup()
    
        try:
            self.wait_until(lambda: self.output_lines() > 0, max_timeout=15)
        except Exception as e:
            raise Exception('No output received form Auditbeat').with_traceback(sys.exc_info()[2])
    
        expected = test.expected()
        found = False
        try:
            self.wait_until(lambda: expected.match(self.flattened_output()), max_timeout=15)
            found = True
        finally:
>           assert found, "The events in: {} don't match the condition: {}".format(
                pretty_print_json(list(self.flattened_output())),
                expected
            )
E           AssertionError: The events in: [
E                {
E                   "@timestamp": "2021-11-15T19:28:16.354Z",
E                   "agent.ephemeral_id": "00949a7f-58c2-49f4-a699-04da86aaebb5",
E                   "agent.id": "8b49382f-7736-4685-a51f-f1aead98a1eb",
E                   "agent.name": "30714023b394",
E                   "agent.type": "auditbeat",
E                   "agent.version": "8.1.0",
E                   "client.bytes": 64,
E                   "client.ip": "127.135.56.93",
E                   "client.packets": 1,
E                   "client.port": 39054,
E                   "destination.bytes": 72,
E                   "destination.ip": "127.26.157.180",
E                   "destination.packets": 1,
E                   "destination.port": 53,
E                   "ecs.version": "8.0.0",
E                   "event.action": "network_flow",
E                   "event.category": [
E                      "network",
E                      "network_traffic"
E                   ],
E                   "event.dataset": "socket",
E                   "event.duration": 53979,
E                   "event.end": "2021-11-15T19:28:13.396Z",
E                   "event.kind": "event",
E                   "event.module": "system",
E                   "event.start": "2021-11-15T19:28:13.396Z",
E                   "event.type": [
E                      "info",
E                      "connection"
E                   ],
E                   "flow.complete": false,
E                   "flow.final": true,
E                   "group.id": "0",
E                   "group.name": "root",
E                   "host.name": "30714023b394",
E                   "network.bytes": 136,
E                   "network.community_id": "1:Ni0XyulR6ESVtUh2y/be1AsnicE=",
E                   "network.direction": "ingress",
E                   "network.packets": 2,
E                   "network.transport": "udp",
E                   "network.type": "ipv4",
E                   "process.args": [
E                      "/go/src/github.com/elastic/beats/build/ve/docker/bin/python3",
E                      "/go/src/github.com/elastic/beats/build/ve/docker/bin/pytest",
E                      "--timeout=90",
E                      "--durations=20",
E                      "--junit-xml=build/TEST-python-integration.xml",
E                      "tests/system/test_exports.py",
E                      "tests/system/test_metricsets.py",
E                      "tests/system/test_system_socket.py"
E                   ],
E                   "process.created": "2021-11-15T19:27:38.130Z",
E                   "process.executable": "/usr/bin/python3.9",
E                   "process.name": "pytest",
E                   "process.pid": 55575,
E                   "related.ip": [
E                      "127.26.157.180",
E                      "127.135.56.93"
E                   ],
E                   "related.user": [
E                      "root"
E                   ],
E                   "server.bytes": 72,
E                   "server.ip": "127.26.157.180",
E                   "server.packets": 1,
E                   "server.port": 53,
E                   "service.type": "system",
E                   "source.bytes": 64,
E                   "source.ip": "127.135.56.93",
E                   "source.packets": 1,
E                   "source.port": 39054,
E                   "system.audit.socket.egid": 0,
E                   "system.audit.socket.euid": 0,
E                   "system.audit.socket.gid": 0,
E                   "system.audit.socket.kernel_sock_address": "0xffff9fb33a6dcc80",
E                   "system.audit.socket.uid": 0,
E                   "user.id": "0",
E                   "user.name": "root"
E                },
E                {
E                   "@timestamp": "2021-11-15T19:28:16.354Z",
E                   "agent.ephemeral_id": "00949a7f-58c2-49f4-a699-04da86aaebb5",
E                   "agent.id": "8b49382f-7736-4685-a51f-f1aead98a1eb",
E                   "agent.name": "30714023b394",
E                   "agent.type": "auditbeat",
E                   "agent.version": "8.1.0",
E                   "client.bytes": 56,
E                   "client.ip": "127.135.56.93",
E                   "client.packets": 1,
E                   "client.port": 39054,
E                   "destination.bytes": 80,
E                   "destination.ip": "127.26.157.180",
E                   "destination.packets": 1,
E                   "destination.port": 53,
E                   "ecs.version": "8.0.0",
E                   "event.action": "network_flow",
E                   "event.category": [
E                      "network",
E                      "network_traffic"
E                   ],
E                   "event.dataset": "socket",
E                   "event.duration": 228254,
E                   "event.end": "2021-11-15T19:28:13.396Z",
E                   "event.kind": "event",
E                   "event.module": "system",
E                   "event.start": "2021-11-15T19:28:13.396Z",
E                   "event.type": [
E                      "info",
E                      "connection"
E                   ],
E                   "flow.complete": false,
E                   "flow.final": true,
E                   "group.id": "0",
E                   "group.name": "root",
E                   "host.name": "30714023b394",
E                   "network.bytes": 136,
E                   "network.community_id": "1:Ni0XyulR6ESVtUh2y/be1AsnicE=",
E                   "network.direction": "egress",
E                   "network.packets": 2,
E                   "network.transport": "udp",
E                   "network.type": "ipv4",
E                   "process.args": [
E                      "/go/src/github.com/elastic/beats/build/ve/docker/bin/python3",
E                      "/go/src/github.com/elastic/beats/build/ve/docker/bin/pytest",
E                      "--timeout=90",
E                      "--durations=20",
E                      "--junit-xml=build/TEST-python-integration.xml",
E                      "tests/system/test_exports.py",
E                      "tests/system/test_metricsets.py",
E                      "tests/system/test_system_socket.py"
E                   ],
E                   "process.created": "2021-11-15T19:27:38.130Z",
E                   "process.executable": "/usr/bin/python3.9",
E                   "process.name": "pytest",
E                   "process.pid": 55575,
E                   "related.ip": [
E                      "127.135.56.93",
E                      "127.26.157.180"
E                   ],
E                   "related.user": [
E                      "root"
E                   ],
E                   "server.bytes": 80,
E                   "server.ip": "127.26.157.180",
E                   "server.packets": 1,
E                   "server.port": 53,
E                   "service.type": "system",
E                   "source.bytes": 56,
E                   "source.ip": "127.135.56.93",
E                   "source.packets": 1,
E                   "source.port": 39054,
E                   "system.audit.socket.egid": 0,
E                   "system.audit.socket.euid": 0,
E                   "system.audit.socket.gid": 0,
E                   "system.audit.socket.kernel_sock_address": "0xffff9fb33a6d9f80",
E                   "system.audit.socket.uid": 0,
E                   "user.id": "0",
E                   "user.name": "root"
E                },
E                {
E                   "@timestamp": "2021-11-15T19:28:16.354Z",
E                   "agent.ephemeral_id": "00949a7f-58c2-49f4-a699-04da86aaebb5",
E                   "agent.id": "8b49382f-7736-4685-a51f-f1aead98a1eb",
E                   "agent.name": "30714023b394",
E                   "agent.type": "auditbeat",
E                   "agent.version": "8.1.0",
E                   "client.bytes": 52,
E                   "client.ip": "10.224.0.43",
E                   "client.packets": 1,
E                   "client.port": 40162,
E                   "destination.bytes": 32,
E                   "destination.ip": "34.66.174.210",
E                   "destination.packets": 1,
E                   "destination.port": 5044,
E                   "ecs.version": "8.0.0",
E                   "event.action": "network_flow",
E                   "event.category": [
E                      "network",
E                      "network_traffic"
E                   ],
E                   "event.dataset": "socket",
E                   "event.duration": 1927860,
E                   "event.end": "2021-11-15T19:28:13.608Z",
E                   "event.kind": "event",
E                   "event.module": "system",
E                   "event.start": "2021-11-15T19:28:13.606Z",
E                   "event.type": [
E                      "info",
E                      "connection"
E                   ],
E                   "flow.complete": false,
E                   "flow.final": true,
E                   "host.name": "30714023b394",
E                   "network.bytes": 84,
E                   "network.community_id": "1:uOcaQ1dChB4yc0m8PhHb9qiriAI=",
E                   "network.direction": "unknown",
E                   "network.packets": 2,
E                   "network.transport": "tcp",
E                   "network.type": "ipv4",
E                   "related.ip": [
E                      "10.224.0.43",
E                      "34.66.174.210"
E                   ],
E                   "server.bytes": 32,
E                   "server.ip": "34.66.174.210",
E                   "server.packets": 1,
E                   "server.port": 5044,
E                   "service.type": "system",
E                   "source.bytes": 52,
E                   "source.ip": "10.224.0.43",
E                   "source.packets": 1,
E                   "source.port": 40162,
E                   "system.audit.socket.kernel_sock_address": "0xffff9fb5ffd9b480"
E                },
E                {
E                   "@timestamp": "2021-11-15T19:28:16.354Z",
E                   "agent.ephemeral_id": "00949a7f-58c2-49f4-a699-04da86aaebb5",
E                   "agent.id": "8b49382f-7736-4685-a51f-f1aead98a1eb",
E                   "agent.name": "30714023b394",
E                   "agent.type": "auditbeat",
E                   "agent.version": "8.1.0",
E                   "client.bytes": 308,
E                   "client.ip": "10.224.0.43",
E                   "client.packets": 2,
E                   "client.port": 51810,
E                   "destination.bytes": 3418,
E                   "destination.ip": "169.254.169.254",
E                   "destination.packets": 3,
E                   "destination.port": 80,
E                   "ecs.version": "8.0.0",
E                   "event.action": "network_flow",
E                   "event.category": [
E                      "network",
E                      "network_traffic"
E                   ],
E                   "event.dataset": "socket",
E                   "event.duration": 1151599,
E                   "event.end": "2021-11-15T19:28:14.098Z",
E                   "event.kind": "event",
E                   "event.module": "system",
E                   "event.start": "2021-11-15T19:28:14.097Z",
E                   "event.type": [
E                      "info",
E                      "connection"
E                   ],
E                   "flow.complete": false,
E                   "flow.final": true,
E                   "group.id": "0",
E                   "group.name": "root",
E                   "host.name": "30714023b394",
E                   "network.bytes": 3726,
E                   "network.community_id": "1:ghfE3cckp4kqnclJ+k1OU6VJMvM=",
E                   "network.direction": "unknown",
E                   "network.packets": 5,
E                   "network.transport": "tcp",
E                   "network.type": "ipv4",
E                   "process.args": [
E                      "/usr/bin/google_guest_agent"
E                   ],
E                   "process.created": "2021-11-15T19:11:11.830Z",
E                   "process.executable": "/usr/bin/google_guest_agent",
E                   "process.name": "google_guest_ag",
E                   "process.pid": 1832,
E                   "related.ip": [
E                      "10.224.0.43",
E                      "169.254.169.254"
E                   ],
E                   "related.user": [
E                      "root"
E                   ],
E                   "server.bytes": 3418,
E                   "server.ip": "169.254.169.254",
E                   "server.packets": 3,
E                   "server.port": 80,
E                   "service.type": "system",
E                   "source.bytes": 308,
E                   "source.ip": "10.224.0.43",
E                   "source.packets": 2,
E                   "source.port": 51810,
E                   "system.audit.socket.egid": 0,
E                   "system.audit.socket.euid": 0,
E                   "system.audit.socket.gid": 0,
E                   "system.audit.socket.kernel_sock_address": "0xffff9fb5fc9a3d40",
E                   "system.audit.socket.uid": 0,
E                   "user.id": "0",
E                   "user.name": "root"
E                },
E                {
E                   "@timestamp": "2021-11-15T19:28:17.354Z",
E                   "agent.ephemeral_id": "00949a7f-58c2-49f4-a699-04da86aaebb5",
E                   "agent.id": "8b49382f-7736-4685-a51f-f1aead98a1eb",
E                   "agent.name": "30714023b394",
E                   "agent.type": "auditbeat",
E                   "agent.version": "8.1.0",
E                   "client.bytes": 10494,
E                   "client.ip": "10.224.0.43",
E                   "client.packets": 5,
E                   "client.port": 34978,
E                   "destination.bytes": 198,
E                   "destination.ip": "35.184.73.24",
E                   "destination.packets": 4,
E                   "destination.port": 5000,
E                   "ecs.version": "8.0.0",
E                   "event.action": "network_flow",
E                   "event.category": [
E                      "network",
E                      "network_traffic"
E                   ],
E                   "event.dataset": "socket",
E                   "event.duration": 1456173583,
E                   "event.end": "2021-11-15T19:28:14.886Z",
E                   "event.kind": "event",
E                   "event.module": "system",
E                   "event.start": "2021-11-15T19:28:13.430Z",
E                   "event.type": [
E                      "info",
E                      "connection"
E                   ],
E                   "flow.complete": false,
E                   "flow.final": true,
E                   "group.id": "0",
E                   "group.name": "root",
E                   "host.name": "30714023b394",
E                   "network.bytes": 10692,
E                   "network.community_id": "1:vb5X/+Y/pqnDCJ+yHnlUk0Z46w4=",
E                   "network.direction": "unknown",
E                   "network.packets": 9,
E                   "network.transport": "tcp",
E                   "network.type": "ipv4",
E                   "process.args": [
E                      "/usr/local/bin/auditbeat",
E                      "-e",
E                      "-c",
E                      "/etc/auditbeat/auditbeat.yml"
E                   ],
E                   "process.created": "2021-11-15T19:11:14.200Z",
E                   "process.executable": "/opt/auditbeat-7.13.3-linux-x86_64/auditbeat",
E                   "process.name": "auditbeat",
E                   "process.pid": 2607,
E                   "related.ip": [
E                      "10.224.0.43",
E                      "35.184.73.24"
E                   ],
E                   "related.user": [
E                      "root"
E                   ],
E                   "server.bytes": 198,
E                   "server.ip": "35.184.73.24",
E                   "server.packets": 4,
E                   "server.port": 5000,
E                   "service.type": "system",
E                   "source.bytes": 10494,
E                   "source.ip": "10.224.0.43",
E                   "source.packets": 5,
E                   "source.port": 34978,
E                   "system.audit.socket.egid": 0,
E                   "system.audit.socket.euid": 0,
E                   "system.audit.socket.gid": 0,
E                   "system.audit.socket.kernel_sock_address": "0xffff9fb5cce1cec0",
E                   "system.audit.socket.uid": 0,
E                   "user.id": "0",
E                   "user.name": "root"
E                },
E                {
E                   "@timestamp": "2021-11-15T19:28:19.354Z",
E                   "agent.ephemeral_id": "00949a7f-58c2-49f4-a699-04da86aaebb5",
E                   "agent.id": "8b49382f-7736-4685-a51f-f1aead98a1eb",
E                   "agent.name": "30714023b394",
E                   "agent.type": "auditbeat",
E                   "agent.version": "8.1.0",
E                   "client.bytes": 52,
E                   "client.ip": "10.224.0.43",
E                   "client.packets": 1,
E                   "client.port": 47912,
E                   "destination.bytes": 32,
E                   "destination.ip": "172.217.212.95",
E                   "destination.packets": 1,
E                   "destination.port": 443,
E                   "ecs.version": "8.0.0",
E                   "event.action": "network_flow",
E                   "event.category": [
E                      "network",
E                      "network_traffic"
E                   ],
E                   "event.dataset": "socket",
E                   "event.duration": 1275442,
E                   "event.end": "2021-11-15T19:28:16.936Z",
E                   "event.kind": "event",
E                   "event.module": "system",
E                   "event.start": "2021-11-15T19:28:16.934Z",
E                   "event.type": [
E                      "info",
E                      "connection"
E                   ],
E                   "flow.complete": false,
E                   "flow.final": true,
E                   "host.name": "30714023b394",
E                   "network.bytes": 84,
E                   "network.community_id": "1:kRqdgglEYeKenI1BG2ca0lnQrUc=",
E                   "network.direction": "unknown",
E                   "network.packets": 2,
E                   "network.transport": "tcp",
E                   "network.type": "ipv4",
E                   "related.ip": [
E                      "10.224.0.43",
E                      "172.217.212.95"
E                   ],
E                   "server.bytes": 32,
E                   "server.ip": "172.217.212.95",
E                   "server.packets": 1,
E                   "server.port": 443,
E                   "service.type": "system",
E                   "source.bytes": 52,
E                   "source.ip": "10.224.0.43",
E                   "source.packets": 1,
E                   "source.port": 47912,
E                   "system.audit.socket.kernel_sock_address": "0xffff9fb5fcb788c0"
E                }
E             ] don't match the condition: the documents contain {
E                "agent.type": "auditbeat",
E                "client.bytes": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 30
E                },
E                "client.ip": "127.135.56.93",
E                "client.packets": 1,
E                "client.port": 39054,
E                "destination.bytes": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 30
E                },
E                "destination.ip": "127.26.157.180",
E                "destination.packets": 1,
E                "destination.port": 53,
E                "event.action": "network_flow",
E                "event.category": [
E                   "network",
E                   "network_traffic"
E                ],
E                "event.dataset": "socket",
E                "event.kind": "event",
E                "event.module": "system",
E                "network.bytes": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 60
E                },
E                "network.direction": "ingress",
E                "network.packets": 2,
E                "network.transport": "udp",
E                "network.type": "ipv4",
E                "process.pid": 55575,
E                "server.bytes": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 30
E                },
E                "server.ip": "127.26.157.180",
E                "server.packets": 1,
E                "server.port": 53,
E                "source.bytes": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 30
E                },
E                "source.ip": "127.135.56.93",
E                "source.packets": 1,
E                "source.port": 39054,
E                "user.id": "0"
E             },
E             {
E                "agent.type": "auditbeat",
E                "client.bytes": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 80
E                },
E                "client.ip": "127.53.165.224",
E                "client.packets": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 2
E                },
E                "client.port": 43039,
E                "destination.bytes": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 2
E                },
E                "destination.domain": "elastic.co",
E                "destination.ip": "127.128.206.158",
E                "destination.packets": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 2
E                },
E                "destination.port": 57617,
E                "event.action": "network_flow",
E                "event.category": [
E                   "network",
E                   "network_traffic"
E                ],
E                "event.dataset": "socket",
E                "event.kind": "event",
E                "event.module": "system",
E                "network.direction": "ingress",
E                "network.packets": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 5
E                },
E                "network.transport": "tcp",
E                "network.type": "ipv4",
E                "process.pid": 55575,
E                "server.bytes": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 2
E                },
E                "server.domain": "elastic.co",
E                "server.ip": "127.128.206.158",
E                "server.packets": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 2
E                },
E                "server.port": 57617,
E                "service.type": "system",
E                "source.bytes": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 80
E                },
E                "source.ip": "127.53.165.224",
E                "source.packets": {
E                   "operator": "<built-in function gt>",
E                   "type": "comparison",
E                   "value": 2
E                },
E                "source.port": 43039
E             }
E           assert False

tests/system/test_system_socket.py:190: AssertionError

During handling of the above exception, another exception occurred:

self = <test_system_socket.Test testMethod=test_dns_long_request>

    def test_dns_long_request(self):
        """
        test DNS enrichment of long request
        This test makes sure that DNS information is kept long after the
        DNS request has been performed, even if the internal DNS state
        is expired.
        """
>       self.with_runner(
            DNSTestCase(delay_seconds=10),
            extra_conf={
                'socket.flow_inactive_timeout': '2s'
            })

tests/system/test_system_socket.py:118: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_system_socket.py:158: in with_runner
    proc.check_kill_and_wait()
../../libbeat/tests/system/beat/beat.py:105: in check_kill_and_wait
    return self.check_wait(exit_code=exit_code)
../../libbeat/tests/system/beat/beat.py:92: in check_wait
    actual_exit_code = self.wait()
../../libbeat/tests/system/beat/beat.py:87: in wait
    return self.proc.wait()
/usr/lib/python3.9/subprocess.py:1189: in wait
    return self._wait(timeout=timeout)
/usr/lib/python3.9/subprocess.py:1919: in _wait
    (pid, sts) = self._try_wait(0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Popen: returncode: None args: ['/go/src/github.com/elastic/beats/x-pack/aud...>
wait_flags = 0

    def _try_wait(self, wait_flags):
        """All callers to this function MUST hold self._waitpid_lock."""
        try:
>           (pid, sts) = os.waitpid(self.pid, wait_flags)
E           Failed: Timeout >90.0s

/usr/lib/python3.9/subprocess.py:1877: Failed

Steps errors

Expand to view the steps failures

`x-pack/auditbeat-build - mage update build test`

Took 9 min 41 sec . View more details here
Description: mage update build test

`x-pack/auditbeat-build - mage update build test`

Took 5 min 47 sec . View more details here
Description: mage update build test

`x-pack/auditbeat-build - mage update build test`

Took 5 min 46 sec . View more details here
Description: mage update build test

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / x-pack/auditbeat-build / test_dns_long_request – x-pack.auditbeat.tests.system.test_system_socket.Test

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Entire collections of socket connections need to be identified when a socket is destroyed so we need to be able to get the actual socket. This approach uses the pid=0 socket as the marker for the socket as a resource as opposed to the pid≠0 which are the executable-socket bindings.

adriansr · 2021-11-02T11:23:07Z

/test

adriansr · 2021-11-02T11:23:45Z

Checking if the failed test is flaky as tests are passing locally

mergify · 2021-11-10T14:56:44Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b auditbeat/socketcollide upstream/auditbeat/socketcollide
git merge upstream/master
git push upstream auditbeat/socketcollide

andrewkroh · 2021-11-30T00:48:16Z

@adriansr @efd6, Checking in on this fix, are either of you debugging the tests? I saw #29166 so I think the answer is yes, but wanted to confirm.

efd6 · 2021-11-30T01:47:20Z

@adriansr Has done more work on debugging this and has a less invasive and more root cause fix (substantively in #29166). The change here will likely be abandoned in favour of that one or a derivative.

mergify · 2021-12-02T07:34:59Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b auditbeat/socketcollide upstream/auditbeat/socketcollide
git merge upstream/master
git push upstream auditbeat/socketcollide

efd6 · 2021-12-02T10:01:52Z

Abandoned

efd6 added bug Team:Security-External Integrations 8.1 candidate labels Oct 29, 2021

efd6 requested a review from a team as a code owner October 29, 2021 07:43

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 29, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 29, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 29, 2021

x-pack/auditbeat/module/system/socket: stratify sockets and connectio…

9b540d7

…ns by PID Previously it was possible for two processes to be assigned to the same socket and so flow due to socket sharing. This change prevents that by keying on PID as well as socket and network address.

efd6 force-pushed the auditbeat/socketcollide branch from 548037d to 9b540d7 Compare October 29, 2021 07:47

efd6 requested a review from adriansr October 29, 2021 07:47

efd6 marked this pull request as draft October 29, 2021 09:11

andrewkroh added backport-v7.16.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Nov 1, 2021

Merge branch 'master' into auditbeat/socketcollide

1917df5

efd6 closed this Dec 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x-pack/auditbeat/module/system/socket: stratify sockets and connections by PID #28713

x-pack/auditbeat/module/system/socket: stratify sockets and connections by PID #28713

efd6 commented Oct 29, 2021

elasticmachine commented Oct 29, 2021

mergify bot commented Oct 29, 2021

elasticmachine commented Oct 29, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

`Build&Test / x-pack/auditbeat-build / test_dns_long_request – x-pack.auditbeat.tests.system.test_system_socket.Test`

`x-pack/auditbeat-build - mage update build test`

`x-pack/auditbeat-build - mage update build test`

`x-pack/auditbeat-build - mage update build test`

`Error signal`

Genuine test errors

adriansr commented Nov 2, 2021

adriansr commented Nov 2, 2021

mergify bot commented Nov 10, 2021

andrewkroh commented Nov 30, 2021

efd6 commented Nov 30, 2021

mergify bot commented Dec 2, 2021

efd6 commented Dec 2, 2021

x-pack/auditbeat/module/system/socket: stratify sockets and connections by PID #28713

x-pack/auditbeat/module/system/socket: stratify sockets and connections by PID #28713

Conversation

efd6 commented Oct 29, 2021

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Oct 29, 2021

mergify bot commented Oct 29, 2021

elasticmachine commented Oct 29, 2021 • edited by jenkins-beats-ci bot Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Build&Test / x-pack/auditbeat-build / test_dns_long_request – x-pack.auditbeat.tests.system.test_system_socket.Test

Steps errors

x-pack/auditbeat-build - mage update build test

x-pack/auditbeat-build - mage update build test

x-pack/auditbeat-build - mage update build test

Error signal

🐛 Flaky test report

Genuine test errors

🤖 GitHub comments

adriansr commented Nov 2, 2021

adriansr commented Nov 2, 2021

mergify bot commented Nov 10, 2021

andrewkroh commented Nov 30, 2021

efd6 commented Nov 30, 2021

mergify bot commented Dec 2, 2021

efd6 commented Dec 2, 2021

elasticmachine commented Oct 29, 2021 •

edited by jenkins-beats-ci bot

Loading

`Build&Test / x-pack/auditbeat-build / test_dns_long_request – x-pack.auditbeat.tests.system.test_system_socket.Test`

`x-pack/auditbeat-build - mage update build test`

`x-pack/auditbeat-build - mage update build test`

`x-pack/auditbeat-build - mage update build test`

`Error signal`