Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

auditbeat/socket returns incorrect process information #17165

Closed
dcode opened this issue Mar 20, 2020 · 6 comments · Fixed by #29166
Closed

auditbeat/socket returns incorrect process information #17165

dcode opened this issue Mar 20, 2020 · 6 comments · Fixed by #29166
Assignees
@adriansr @efd6
Labels
8.1 candidate Auditbeat bug

Comments

@dcode
Copy link
Contributor

dcode commented Mar 20, 2020

Please include configurations and logs if available.

auditbeat info

  • Version: 7.6.1
  • Operating System: CentOS 7
  • relevant config:

auditbeat was started using systemd service file

system module configuration

- module: system
  datasets:
    - host    # General host information, e.g. uptime, IPs
    - login   # User logins, logouts, and system boots.
    - package # Installed, updated, and removed packages
    - process # Started and stopped processes
    - socket  # Opened and closed sockets
    - user    # User information

  # How often datasets send state updates with the
  # current state of the system (e.g. all currently
  # running processes, all open sockets).
  state.period: 12h

  # Enabled by default. Auditbeat will read password fields in
  # /etc/passwd and /etc/shadow and store a hash locally to
  # detect any changes.
  user.detect_password_changes: true

  # File patterns of the login record files.
  login.wtmp_file_pattern: /var/log/wtmp*
  login.btmp_file_pattern: /var/log/btmp*
  • Steps to Reproduce:

locally

ssh [email protected] -R23456:127.0.0.1:23456

remotely

# get pid for sshd user session
ps -u dcode | awk '/sshd/ { print $1 }'

# Connect to local port, doesn't actually go anywhere in this case
nc 127.0.0.1 23456

View logs in Kibana using the following filter, substituting in the pid printed above (it was 20785 in this test case).

agent.type: auditbeat and process.pid: (20785) and event.action: network_flow

snipped to just process object:

"process": {
      "pid": 20785,
      "name": "java",
      "args": [
        "java",
        "-Xmx1G",
        "-Xms1G",
        "-server",
        "-XX:+UseG1GC",
        "-XX:MaxGCPauseMillis=20",
        "-XX:InitiatingHeapOccupancyPercent=35",
        "-XX:+DisableExplicitGC",
        "-Djava.awt.headless=true",
        "-Xlog:gc:/var/log/kafka/kafkaServer-gc.log",
        "-Xlog:gc*",
        "-Dcom.sun.management.jmxremote",
        "-Dcom.sun.management.jmxremote.authenticate=false",
        "-Dcom.sun.management.jmxremote.ssl=false",
        "-Dkafka.logs.dir=/usr/share/kafka/bin/../logs",
        "-Dkafka.logs.dir=/var/log/kafka",
        "-Dlog4j.configuration=file:///etc/kafka/log4j.properties",
        "-cp",
        "/usr/share/java/kafka/activation-1.1.1.jar:/usr/share/java/kafka/jetty-io-9.4.18.v20190429.jar:/usr/share/java/kafka/aopalliance-repackaged-2.5.0.jar:/usr/share/java/kafka/argparse4j-0.7.0.jar:/usr/share/java/kafka/jsr305-3.0.2.jar:/usr/share/java/kafka/audience-annotations-0.5.0.jar:/usr/share/java/kafka/kafka_2.12-2.3.0.jar:/usr/share/java/kafka/commons-lang3-3.8.1.jar:/usr/share/java/kafka/log4j-1.2.17.jar:/usr/share/java/kafka/connect-api-2.3.0.jar:/usr/share/java/kafka/jetty-http-9.4.18.v20190429.jar:/usr/share/java/kafka/connect-basic-auth-extension-2.3.0.jar:/usr/share/java/kafka/lz4-java-1.6.0.jar:/usr/share/java/kafka/connect-file-2.3.0.jar:/usr/share/java/kafka/maven-artifact-3.6.1.jar:/usr/share/java/kafka/connect-json-2.3.0.jar:/usr/share/java/kafka/metrics-core-2.2.0.jar:/usr/share/java/kafka/connect-runtime-2.3.0.jar:/usr/share/java/kafka/jopt-simple-5.0.4.jar:/usr/share/java/kafka/connect-transforms-2.3.0.jar:/usr/share/java/kafka/zstd-jni-1.4.0-1.jar:/usr/share/java/kafka/guava-20.0.jar:/usr/share/java/kafka/zkclient-0.11.jar:/usr/share/java/kafka/hk2-api-2.5.0.jar:/usr/share/java/kafka/plexus-utils-3.2.0.jar:/usr/share/java/kafka/hk2-locator-2.5.0.jar:/usr/share/java/kafka/zookeeper-3.4.14.jar:/usr/share/java/kafka/hk2-utils-2.5.0.jar:/usr/share/java/kafka/kafka-clients-2.3.0.jar:/usr/share/java/kafka/jackson-annotations-2.9.9.jar:/usr/share/java/kafka/osgi-resource-locator-1.0.1.jar:/usr/share/java/kafka/jackson-core-2.9.9.jar:/usr/share/java/kafka/paranamer-2.8.jar:/usr/share/java/kafka/jackson-databind-2.9.9.jar:/usr/share/java/kafka/kafka-log4j-appender-2.3.0.jar:/usr/share/java/kafka/jackson-dataformat-csv-2.9.9.jar:/usr/share/java/kafka/kafka-streams-2.3.0.jar:/usr/share/java/kafka/jackson-datatype-jdk8-2.9.9.jar:/usr/share/java/kafka/kafka-streams-examples-2.3.0.jar:/usr/share/java/kafka/jackson-jaxrs-base-2.9.9.jar:/usr/share/java/kafka/jetty-client-9.4.18.v20190429.jar:/usr/share/java/kafka/jackson-jaxrs-json-provider-2.9.9.jar:/usr/share/java/kafka/jetty-continuation-9.4.18.v20190429.jar:/usr/share/java/kafka/jackson-module-jaxb-annotations-2.9.9.jar:/usr/share/java/kafka/kafka-streams-scala_2.12-2.3.0.jar:/usr/share/java/kafka/jackson-module-paranamer-2.9.9.jar:/usr/share/java/kafka/kafka-streams-test-utils-2.3.0.jar:/usr/share/java/kafka/jackson-module-scala_2.12-2.9.9.jar:/usr/share/java/kafka/kafka-tools-2.3.0.jar:/usr/share/java/kafka/jakarta.annotation-api-1.3.4.jar:/usr/share/java/kafka/reflections-0.9.11.jar:/usr/share/java/kafka/jakarta.inject-2.5.0.jar:/usr/share/java/kafka/rocksdbjni-5.18.3.jar:/usr/share/java/kafka/jakarta.ws.rs-api-2.1.5.jar:/usr/share/java/kafka/scala-library-2.12.8.jar:/usr/share/java/kafka/javassist-3.22.0-CR2.jar:/usr/share/java/kafka/slf4j-api-1.7.26.jar:/usr/share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/share/java/kafka/scala-logging_2.12-3.9.0.jar:/usr/share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/share/java/kafka/jaxb-api-2.3.0.jar:/usr/share/java/kafka/scala-reflect-2.12.8.jar:/usr/share/java/kafka/jersey-client-2.28.jar:/usr/share/java/kafka/slf4j-log4j12-1.7.26.jar:/usr/share/java/kafka/jersey-common-2.28.jar:/usr/share/java/kafka/jersey-container-servlet-2.28.jar:/usr/share/java/kafka/jetty-servlets-9.4.18.v20190429.jar:/usr/share/java/kafka/jersey-container-servlet-core-2.28.jar:/usr/share/java/kafka/jersey-hk2-2.28.jar:/usr/share/java/kafka/snappy-java-1.1.7.3.jar:/usr/share/java/kafka/jersey-media-jaxb-2.28.jar:/usr/share/java/kafka/spotbugs-annotations-3.1.9.jar:/usr/share/java/kafka/jersey-server-2.28.jar:/usr/share/java/kafka/jetty-security-9.4.18.v20190429.jar:/usr/share/java/kafka/jetty-server-9.4.18.v20190429.jar:/usr/share/java/kafka/jetty-servlet-9.4.18.v20190429.jar:/usr/share/java/kafka/jetty-util-9.4.18.v20190429.jar:/usr/share/java/kafka/validation-api-2.0.1.Final.jar",
        "kafka.Kafka",
        "/etc/kafka/server.properties"
      ],
      "executable": "/usr/lib/jvm/java-11-openjdk-11.0.6.10-1.el7_7.x86_64/bin/java",
      "created": "2020-03-18T03:14:02.380Z"
    },

In this case, the Kafka process above has a pid of 7268
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@adriansr adriansr self-assigned this Apr 2, 2020
@adriansr adriansr mentioned this issue Apr 2, 2020
Closed
3 tasks
@Rockso
Copy link

Rockso commented Mar 8, 2021

I'm running auditbeat 7.10.1 and have recently enabled the socket module and have also noticed that auditbeat reports incorrect process information for some socket flow events periodically. I've noticed that this is more noticeable on hosts with a high amount of network activity occurring.

Has this issue gained any traction since April of last year?

@adriansr adriansr changed the title auditbeat returns incorrect process information for sshd sockets auditbeat/socket returns incorrect process information Sep 8, 2021
@ayedem
Copy link

ayedem commented Sep 9, 2021 •
edited
Loading

This appears to be a pretty serious flaw in Auditbeat. If we can't trust the process information being logged into our SIEM, its really makes me question whether this data could be used in any official legal capacity.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@efd6
Copy link
Contributor

efd6 commented Oct 21, 2021 •
edited
Loading

I've just had a look at this with a view that it is probably a data aliasing issue. Initially I looked at building with the race detector on (go build -race) which fortuitously also turns on pointer checks. Building only with pointer checks (go build -gcflags=all=-d=checkptr) is enough to demonstrate a possible cause.

fatal error: checkptr: converted pointer straddles multiple allocations

goroutine 184 [running]:

runtime.throw({0x3b0cabd, 0x0})

.../go/src/runtime/panic.go:1198 +0x71 fp=0xc0008aed00 sp=0xc0008aecd0 pc=0x1376091

runtime.checkptrAlignment(0x33c5380, 0xc0008aedb8, 0x1)

.../go/src/runtime/checkptr.go:26 +0x6c fp=0xc0008aed20 sp=0xc0008aed00 pc=0x134722c

github.com/elastic/beats/v7/x-pack/auditbeat/tracing.(*structDecoder).Decode(0xc000451ac0, {0xc000167400, 0x3dc, 0x3dc}, {0x1499, 0x0, 0x0, 0x4f5a, 0x4f52, 0xdad})

.../src/github.com/elastic/beats/x-pack/auditbeat/tracing/decoder.go:344 +0x195 fp=0xc0008aee38 sp=0xc0008aed20 pc=0x2b0edd5

github.com/elastic/beats/v7/x-pack/auditbeat/tracing.(*PerfChannel).channelLoop(0xc000523e40)

.../src/github.com/elastic/beats/x-pack/auditbeat/tracing/perfevent.go:385 +0x295 fp=0xc0008aefc8 sp=0xc0008aee38 pc=0x2b11555

github.com/elastic/beats/v7/x-pack/auditbeat/tracing.(*PerfChannel).Run·dwrap·1()

.../src/github.com/elastic/beats/x-pack/auditbeat/tracing/perfevent.go:298 +0x26 fp=0xc0008aefe0 sp=0xc0008aefc8 pc=0x2b10c26

runtime.goexit()

.../go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0008aefe8 sp=0xc0008aefe0 pc=0x13aa5e1

created by github.com/elastic/beats/v7/x-pack/auditbeat/tracing.(*PerfChannel).Run

.../src/github.com/elastic/beats/x-pack/auditbeat/tracing/perfevent.go:298 +0x1a8


goroutine 1 [runnable]:

github.com/elastic/beats/v7/x-pack/auditbeat/module/system/socket/guess.guessOnce({0x4051f98, 0xc000546300}, {0x3fe7698, 0xc00005d9c0}, {{0x4011e68, 0xc000455ba0}, 0xc000def1d0, 0x0})

.../src/github.com/elastic/beats/x-pack/auditbeat/module/system/socket/guess/guess.go:211 +0xa36

github.com/elastic/beats/v7/x-pack/auditbeat/module/system/socket/guess.Guess({0x4051f98, 0xc000546300}, {0x3fe7698, 0xc00005d9c0}, {{0x4011e68, 0xc000455ba0}, 0xc000def1d0, 0xc00140f970})

.../src/github.com/elastic/beats/x-pack/auditbeat/module/system/socket/guess/guess.go:98 +0x129

github.com/elastic/beats/v7/x-pack/auditbeat/module/system/socket/guess.GuessAll({0x3fe7698, 0xc00005d9c0}, {{0x4011e68, 0xc000455ba0}, 0xc000def1d0, 0xc0005f68a0})

.../src/github.com/elastic/beats/x-pack/auditbeat/module/system/socket/guess/guess.go:279 +0x492

github.com/elastic/beats/v7/x-pack/auditbeat/module/system/socket.(*MetricSet).Setup(0xc000159dc0)

.../src/github.com/elastic/beats/x-pack/auditbeat/module/system/socket/socket_linux.go:394 +0xec5

github.com/elastic/beats/v7/x-pack/auditbeat/module/system/socket.newSocketMetricset({0x0, 0x1000, 0x80, 0x1, 0x7, 0x6fc23ac00, 0xdf8475800, 0x12a05f200, 0x5f5e100, 0x2540be400, ...}, ...)

.../src/github.com/elastic/beats/x-pack/auditbeat/module/system/socket/socket_linux.go:143 +0x3fb

github.com/elastic/beats/v7/x-pack/auditbeat/module/system/socket.New({{0xc000b76de0, 0x24}, {0xc00070ec40, 0x6}, {0x3fe7668, 0xc000b7b6b0}, {0x0, 0x0}, {{0x0, 0x0}, ...}, ...})

.../src/github.com/elastic/beats/x-pack/auditbeat/module/system/socket/socket_linux.go:120 +0x239

github.com/elastic/beats/v7/metricbeat/mb.initMetricSets(0xc00070ec20, {0x3fe7668, 0xc000b7b6b0})

.../src/github.com/elastic/beats/metricbeat/mb/builders.go:139 +0x515

github.com/elastic/beats/v7/metricbeat/mb.NewModule(0x0, 0x0)

.../src/github.com/elastic/beats/metricbeat/mb/builders.go:65 +0x16d

github.com/elastic/beats/v7/metricbeat/mb/module.(*Factory).Create(0xc0005301a0, {0x3fdf100, 0xc000546780}, 0xc000010040)

.../src/github.com/elastic/beats/metricbeat/mb/module/factory.go:44 +0x5b

github.com/elastic/beats/v7/metricbeat/beater.newMetricbeat(0xc00049e000, 0xc0000b7c80, {0xc00053a370, 0x1, 0xc000c4f018})

.../src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:169 +0x5be

github.com/elastic/beats/v7/metricbeat/beater.Creator.func1(0xc0000b7c80, 0x3a63019)

.../src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:81 +0x25

github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater(0xc00049e000, 0xc000542d00)

.../src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:392 +0x748

github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc00049e000, {{0x3a6c020, 0x9}, {0x3a6c020, 0x9}, {0x0, 0x0}, 0x1, 0x1, {{0x0, ...}, ...}, ...}, ...)

.../src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:456 +0x528

github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1({0x3a6c020, 0x9}, {0x3a6c020, 0x9}, {0x0, 0x0}, 0x1, 0xc000c4fd30, 0x0)

.../src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:211 +0x528

github.com/elastic/beats/v7/libbeat/cmd/instance.Run({{0x3a6c020, 0x9}, {0x3a6c020, 0x9}, {0x0, 0x0}, 0x1, 0x1, {{0x0, 0x0}, ...}, ...}, ...)

.../src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:212 +0x119

github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc00017cf00, {0x3a5e534, 0x3, 0x3})

.../src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x58

github.com/spf13/cobra.(*Command).execute(0xc00017cf00, {0xc0004f9380, 0x3, 0x3})

.../pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x5f8

github.com/spf13/cobra.(*Command).ExecuteC(0xc000698b00)

.../pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fc

github.com/spf13/cobra.(*Command).Execute(...)

.../pkg/mod/github.com/spf13/[email protected]/command.go:864

main.main()

.../src/github.com/elastic/beats/x-pack/auditbeat/main.go:21 +0x25


goroutine 13 [chan receive]:

github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0000a1e10)

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94

created by github.com/klauspost/compress/zstd.newBlockDec

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x167


goroutine 14 [chan receive]:

github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0000a1ee0)

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94

created by github.com/klauspost/compress/zstd.newBlockDec

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x167


goroutine 15 [chan receive]:

github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000290000)

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94

created by github.com/klauspost/compress/zstd.newBlockDec

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x167


goroutine 16 [chan receive]:

github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002900d0)

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94

created by github.com/klauspost/compress/zstd.newBlockDec

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x167


goroutine 66 [chan receive]:

github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002901a0)

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94

created by github.com/klauspost/compress/zstd.newBlockDec

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x167


goroutine 67 [chan receive]:

github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000290270)

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94

created by github.com/klauspost/compress/zstd.newBlockDec

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x167


goroutine 68 [chan receive]:

github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000290340)

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94

created by github.com/klauspost/compress/zstd.newBlockDec

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x167


goroutine 69 [chan receive]:

github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000290410)

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94

created by github.com/klauspost/compress/zstd.newBlockDec

.../pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x167


goroutine 70 [chan receive]:

k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)

.../pkg/mod/k8s.io/klog/[email protected]/klog.go:1164 +0x6a

created by k8s.io/klog/v2.init.0

.../pkg/mod/k8s.io/klog/[email protected]/klog.go:418 +0xfb


goroutine 72 [chan receive]:

github.com/golang/glog.(*loggingT).flushDaemon(0xc000541780)

.../pkg/mod/github.com/elastic/[email protected]/glog.go:890 +0x6a

created by github.com/golang/glog.init.0

.../pkg/mod/github.com/elastic/[email protected]/glog.go:404 +0x73


goroutine 98 [select]:

github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).run(0xc000178320)

.../src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:316 +0x17c

github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func1()

.../src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:176 +0x67

created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue

.../src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:174 +0x632


goroutine 99 [select]:

github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc000512460)

.../src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0xe6

github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2()

.../src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:180 +0x65

created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue

.../src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:178 +0x691


goroutine 100 [select]:

github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*consumer).Get(0xc0004c7780, 0x0)

.../src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/consume.go:65 +0x9e

github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc0000a3da0, {0x3fdf308, 0xc0004c7780})

.../src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:182 +0x198

github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1()

.../src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:86 +0x66

created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer

.../src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:84 +0x1a7


goroutine 101 [select]:

github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*retryer).loop(0xc00022a060)

.../src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:135 +0x234

created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newRetryer

.../src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:94 +0x1dd


goroutine 102 [select]:

github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run(0xc00022a300)

.../src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:127 +0x9d

created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker

.../src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:79 +0x2b2


goroutine 142 [chan receive, locked to thread]:

github.com/elastic/beats/v7/x-pack/auditbeat/module/system/socket/helper.NewFixedThreadExecutor.func1()

.../src/github.com/elastic/beats/x-pack/auditbeat/module/system/socket/helper/fixedthreadexecutor.go:64 +0xe5

created by github.com/elastic/beats/v7/x-pack/auditbeat/module/system/socket/helper.NewFixedThreadExecutor

.../src/github.com/elastic/beats/x-pack/auditbeat/module/system/socket/helper/fixedthreadexecutor.go:60 +0xd2


goroutine 161 [chan receive]:

github.com/elastic/go-perf.(*Event).poll(0xc000dc9d40)

.../pkg/mod/github.com/elastic/[email protected]/record.go:248 +0xbf

created by github.com/elastic/go-perf.(*Event).MapRingNumPages

.../pkg/mod/github.com/elastic/[email protected]/perf.go:284 +0x2c5


goroutine 160 [chan receive]:

github.com/elastic/go-perf.(*Event).poll(0xc000dc96c0)

.../pkg/mod/github.com/elastic/[email protected]/record.go:248 +0xbf

created by github.com/elastic/go-perf.(*Event).MapRingNumPages

.../pkg/mod/github.com/elastic/[email protected]/perf.go:284 +0x2c5


goroutine 183 [chan receive]:

github.com/elastic/go-perf.(*Event).poll(0xc000dfc340)

.../pkg/mod/github.com/elastic/[email protected]/record.go:248 +0xbf

created by github.com/elastic/go-perf.(*Event).MapRingNumPages

.../pkg/mod/github.com/elastic/[email protected]/perf.go:284 +0x2c5


goroutine 182 [chan receive]:

github.com/elastic/go-perf.(*Event).poll(0xc000dfc270)

.../pkg/mod/github.com/elastic/[email protected]/record.go:248 +0xbf

created by github.com/elastic/go-perf.(*Event).MapRingNumPages

.../pkg/mod/github.com/elastic/[email protected]/perf.go:284 +0x2c5


goroutine 181 [chan receive]:

github.com/elastic/go-perf.(*Event).poll(0xc000dfc1a0)

.../pkg/mod/github.com/elastic/[email protected]/record.go:248 +0xbf

created by github.com/elastic/go-perf.(*Event).MapRingNumPages

.../pkg/mod/github.com/elastic/[email protected]/perf.go:284 +0x2c5


goroutine 180 [chan receive]:

github.com/elastic/go-perf.(*Event).poll(0xc000dfc0d0)

.../pkg/mod/github.com/elastic/[email protected]/record.go:248 +0xbf

created by github.com/elastic/go-perf.(*Event).MapRingNumPages

.../pkg/mod/github.com/elastic/[email protected]/perf.go:284 +0x2c5


goroutine 179 [chan receive]:

github.com/elastic/go-perf.(*Event).poll(0xc000dfc000)

.../pkg/mod/github.com/elastic/[email protected]/record.go:248 +0xbf

created by github.com/elastic/go-perf.(*Event).MapRingNumPages

.../pkg/mod/github.com/elastic/[email protected]/perf.go:284 +0x2c5


goroutine 178 [chan receive]:

github.com/elastic/go-perf.(*Event).poll(0xc000dc9ee0)

.../pkg/mod/github.com/elastic/[email protected]/record.go:248 +0xbf

created by github.com/elastic/go-perf.(*Event).MapRingNumPages

.../pkg/mod/github.com/elastic/[email protected]/perf.go:284 +0x2c5

There is a bunch of unsafe shenanigans going on in tracing.(*structDecoder).Decode that could be the cause of this kind of data aliasing. I will look into it further.

This turns out to be unrelated.

I was able to reliably replicate the original issue by running a number of instances of nmap doing complete port scans of localhost on an otherwise reasonably quiet machine.

This was referenced Oct 21, 2021
Merged
Closed
@efd6 efd6 self-assigned this Oct 28, 2021
@efd6
Copy link
Contributor

efd6 commented Oct 29, 2021 •
edited
Loading

This is a result of sockets being shared between processes resulting in PIDs/processes colliding for a flow and then having a skewed update.

I have a partial fix that is reliable that is based on stratifying socket and flows by PID as well as socket address and network address. I'll clean it up and send it soon.

@efd6 efd6 mentioned this issue Oct 29, 2021
Closed
7 tasks
@adriansr adriansr mentioned this issue Dec 1, 2021
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adriansr adriansr

@efd6 efd6

Labels
8.1 candidate Auditbeat bug
Projects
None yet
Milestone
No milestone
8 participants
@ycombinator @dcode @adriansr @elasticmachine @Rockso @ayedem @jamiehynds @efd6