Filebeat kubernetes fatal error: concurrent map writes #17968

brummelbaer · 2020-04-24T14:49:38Z

Hi,

Created this issue after posting to: https://discuss.elastic.co/t/filebeat-kubernetes-fatal-error-concurrent-map-writes/229660

I have run into a problem with the autodiscovery module on kubernetes. Filebeat is being deployed as suggested in the documentation with the addition that I wanted to have additional cloud and host metadata. Therefore I enabled the add_cloud_metadata and the add_host_metadata processor. As soon as the add_cloud_metadata processor is activated and the beat is being deployed to the cluster it crashes on some on the nodes (see log below). If the add_cloud_metadata processor is disabled everything works fine. It might have something to do with elasticsearch itself, because the beat and elasticsearch are deployed in the same cluster.

Does anybody have any suggestions regarding this error?

Greetings

Samuel

Setup information

elastic kubernetes operator: 1.0.1
elastic version: 7.6.2
filebeat version: 7.6.2
master nodes: 3
data nodes: 2
kibana nodes: 1

Filebeat configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: elasticsearch
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
   
    processors:
      - add_cloud_metadata: ~
      - add_host_metadata:


    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      protocol: https

Debug log

2020-04-24T09:10:05.358Z        INFO    instance/beat.go:622    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-04-24T09:10:05.358Z        INFO    instance/beat.go:630    Beat ID: <beat-id>
2020-04-24T09:10:05.359Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-04-24T09:10:05.359Z        INFO    [beat]  instance/beat.go:958    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"},"type": "filebeat", "uuid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}}}
2020-04-24T09:10:05.359Z        INFO    [beat]  instance/beat.go:967    Build info      {"system_info": {"build": {"commit": "d57bcf8684602e15000d65b75afcd110e2b12b59", "libbeat": "7.6.2", "time": "2020-03-26T05:23:38.000Z", "version": "7.6.2"}}}
2020-04-24T09:10:05.359Z        INFO    [beat]  instance/beat.go:970    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.8"}}}
2020-04-24T09:10:05.360Z        INFO    [beat]  instance/beat.go:974    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-11T16:03:33Z","containerized":false,"name":"<hostname>","ip":["127.0.0.1/8","::1/128","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx","xxx.xxx.xxx.xxx/xx"],"kernel_version":"4.14.154-128.181.amzn2.x86_64","mac":["xx:xx:xx:xx:xx:xx","xx:xx:xx:xx:xx:xx","xx:xx:xx:xx:xx:xx","xx:xx:xx:xx:xx:xx"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-04-24T09:10:05.361Z        INFO    add_cloud_metadata/add_cloud_metadata.go:93     add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"xxxxxxxxxxx"},"availability_zone":"eu-central-1c","image":{"id":"ami-xxxxxxx"},"instance":{"id":"i-xxxxxxxxx"},"machine":{"type":"m5.large"},"provider":"aws","region":"eu-central-1"}
2020-04-24T09:10:05.363Z        INFO    [beat]  instance/beat.go:1003   Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-04-24T09:10:04.490Z"}}}
2020-04-24T09:10:05.363Z        INFO    instance/beat.go:298    Setup Beat: filebeat; Version: 7.6.2
2020-04-24T09:10:05.363Z        INFO    [index-management]      idxmgmt/std.go:182      Set output.elasticsearch.index to 'filebeat' as ILM is enabled.
2020-04-24T09:10:05.363Z        INFO    elasticsearch/client.go:174     Elasticsearch url: https://es.domain.tld:443
2020-04-24T09:10:05.363Z        INFO    elasticsearch/client.go:174     Elasticsearch url: https://es.domain.tld:443
2020-04-24T09:10:05.363Z        INFO    [publisher]     pipeline/module.go:110  Beat name: <beat-name>
2020-04-24T09:10:05.364Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-04-24T09:10:05.364Z        INFO    instance/beat.go:439    filebeat start running.
2020-04-24T09:10:05.364Z        INFO    registrar/registrar.go:145      Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-04-24T09:10:05.365Z        INFO    registrar/registrar.go:152      States Loaded from registrar: 60
2020-04-24T09:10:05.365Z        INFO    crawler/crawler.go:72   Loading Inputs: 0
2020-04-24T09:10:05.365Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 0
2020-04-24T09:10:05.365Z        WARN    [cfgwarn]       kubernetes/kubernetes.go:62     BETA: The kubernetes autodiscover is beta
2020-04-24T09:10:05.366Z        INFO    kubernetes/util.go:79   kubernetes: Using node <node-name> provided in the config
2020-04-24T09:10:05.366Z        INFO    [autodiscover]  autodiscover/autodiscover.go:104        Starting autodiscover manager
2020-04-24T09:10:05.468Z        INFO    log/input.go:152        Configured paths: [/var/log/containers/*<hash-1>.log]
2020-04-24T09:10:05.468Z        INFO    input/input.go:114      Starting input of type: container; ID: <id-1>
2020-04-24T09:10:05.469Z        INFO    log/harvester.go:297    Harvester started for file: /var/log/containers/<cluster-name>-es-data-1_default_elasticsearch-<hash-1>.log
2020-04-24T09:10:05.469Z        INFO    log/input.go:152        Configured paths: [/var/log/containers/*<hash-2>.log]
2020-04-24T09:10:05.469Z        INFO    input/input.go:114      Starting input of type: container; ID: <id-2>
2020-04-24T09:10:05.470Z        INFO    log/harvester.go:297    Harvester started for file: /var/log/containers/<cluster-name>-es-master-1_default_elasticsearch-<hash-2>.log
2020-04-24T09:10:05.471Z        INFO    log/input.go:152        Configured paths: [/var/log/containers/*<hash-3>.log]
2020-04-24T09:10:05.471Z        INFO    input/input.go:114      Starting input of type: container; ID: <id-3>
2020-04-24T09:10:05.473Z        INFO    log/input.go:152        Configured paths: [/var/log/containers/*<hash-4>.log]
2020-04-24T09:10:05.473Z        INFO    input/input.go:114      Starting input of type: container; ID: <id-4>
fatal error: concurrent map writes
fatal error: concurrent map writes

goroutine 160 [running]:
runtime.throw(0x330841d, 0x15)
        /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc000a3b7e0 sp=0xc000a3b7b0 pc=0x13e6082
runtime.mapassign_faststr(0x31f9be0, 0xc000397380, 0xc0000c08a0, 0xb, 0xc000a9e480)
        /usr/local/go/src/runtime/map_faststr.go:291 +0x3fe fp=0xc000a3b848 sp=0xc000a3b7e0 pc=0x13ca19e
github.com/elastic/beats/libbeat/common.MapStr.deepUpdateMap(0xc000397380, 0xc000b112f0, 0xc000878901)
        /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:90 +0x13e fp=0xc000a3b920 sp=0xc000a3b848 pc=0x18593be
github.com/elastic/beats/libbeat/common.deepUpdateValue(0x31f9be0, 0xc000397380, 0xc000b112f0, 0x1, 0xc0008789a8, 0xc0000c0888)
        /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:105 +0xb0 fp=0xc000a3b958 sp=0xc000a3b920 pc=0x1859750
github.com/elastic/beats/libbeat/common.MapStr.deepUpdateMap(0xc000b10fc0, 0xc000b112c0, 0xc000397301)
        /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:87 +0x25d fp=0xc000a3ba30 sp=0xc000a3b958 pc=0x18594dd
github.com/elastic/beats/libbeat/common.MapStr.DeepUpdate(...)
        /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:71
github.com/elastic/beats/libbeat/processors/actions.(*addFields).Run(0xc0001fab30, 0xc000db3a00, 0xc000db3a00, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/processors/actions/add_fields.go:81 +0x51 fp=0xc000a3ba58 sp=0xc000a3ba30 pc=0x1b0b7f1
github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000359f20, 0xc000db3a00, 0xc000db3a00, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94 fp=0xc000a3bb08 sp=0xc000a3ba58 pc=0x1b19594
github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000850600, 0xc000db3a00, 0xed6346277, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94 fp=0xc000a3bbb8 sp=0xc000a3bb08 pc=0x1b19594
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).publish(0xc0001a3200, 0x2282b626, 0xed634627d, 0x0, 0x0, 0xc000b10f30, 0x31fb340, 0xc000cdfe60, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:89 +0x571 fp=0xc000a3bde8 sp=0xc000a3bbb8 pc=0x1b1d851
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).Publish(0xc0001a3200, 0x2282b626, 0xed634627d, 0x0, 0x0, 0xc000b10f30, 0x31fb340, 0xc000cdfe60, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:68 +0xc2 fp=0xc000a3be80 sp=0xc000a3bde8 pc=0x1b1d282
github.com/elastic/beats/filebeat/channel.(*outlet).OnEvent(0xc0008507e0, 0x2282b626, 0xed634627d, 0x0, 0x0, 0xc000b10f30, 0x31fb340, 0xc000cdfe60, 0x0, 0x2000100000001)
        /go/src/github.com/elastic/beats/filebeat/channel/outlet.go:64 +0x7e fp=0xc000a3bed8 sp=0xc000a3be80 pc=0x269611e
github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc000521410, 0x37331e0, 0xc0008507e0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:45 +0x10d fp=0xc000a3bfc8 sp=0xc000a3bed8 pc=0x26967fd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000a3bfd0 sp=0xc000a3bfc8 pc=0x1416ec1
created by github.com/elastic/beats/filebeat/channel.SubOutlet
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8

goroutine 1 [chan receive]:
github.com/elastic/beats/filebeat/beater.(*signalWait).Wait(...)
        /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:45
github.com/elastic/beats/filebeat/beater.(*Filebeat).Run(0xc0004615c0, 0xc000500000, 0x0, 0x0)
        /go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:410 +0xef8
github.com/elastic/beats/libbeat/cmd/instance.(*Beat).launch(0xc000500000, 0x32eab08, 0x8, 0x32eab08, 0x8, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
        /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:445 +0x700
github.com/elastic/beats/libbeat/cmd/instance.Run.func1(0x32eab08, 0x8, 0x32eab08, 0x8, 0x0, 0x0, 0xc000947c58, 0x33f0e98, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:188 +0x56b
github.com/elastic/beats/libbeat/cmd/instance.Run(0x32eab08, 0x8, 0x32eab08, 0x8, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:189 +0x148
github.com/elastic/beats/libbeat/cmd.genRunCmd.func1(0xc0000d8280, 0xc000492630, 0x0, 0x3)
        /go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x9d
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).execute(0xc0000d8280, 0xc0000a2050, 0x3, 0x3, 0xc0000d8280, 0xc0000a2050)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:704 +0x285
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0000d8280, 0x0, 0xa6000, 0xc0000a4058)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:785 +0x2c9
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:738
main.main()
        /go/src/github.com/elastic/beats/x-pack/filebeat/main.go:22 +0x2f

goroutine 141 [running]:
        goroutine running on other thread; stack unavailable
created by github.com/elastic/beats/filebeat/channel.SubOutlet
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8

goroutine 132 [select]:
github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc00046ed00)
        /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c
created by github.com/elastic/beats/filebeat/input/log.NewInput
        /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x7e2

goroutine 26 [syscall]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 27 [chan receive]:
github.com/elastic/beats/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x5003900)
        /go/src/github.com/elastic/beats/vendor/k8s.io/klog/klog.go:1000 +0x8b
created by github.com/elastic/beats/vendor/k8s.io/klog.init.0
        /go/src/github.com/elastic/beats/vendor/k8s.io/klog/klog.go:404 +0x6c

goroutine 30 [select]:
github.com/elastic/beats/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0000f45f0)
        /go/src/github.com/elastic/beats/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
created by github.com/elastic/beats/vendor/go.opencensus.io/stats/view.init.0
        /go/src/github.com/elastic/beats/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57

goroutine 133 [select]:
github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc0002d60c0)
        /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x144
github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc0002f8670, 0xc0002d60c0)
        /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x65
created by github.com/elastic/beats/filebeat/input.(*Runner).Start
        /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x198

goroutine 182 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x37331e0, 0xc0009540c0, 0xc0000a5260)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 136 [select]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc000662df0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff
created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182

goroutine 144 [select]:
github.com/elastic/beats/filebeat/input/log.(*Harvester).monitorFileSize(0xc0003b4280)
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:362 +0x105
created by github.com/elastic/beats/filebeat/input/log.(*Harvester).Run
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:299 +0x29d

goroutine 143 [select]:
github.com/elastic/beats/filebeat/input/log.(*Harvester).Run.func2(0xc0003b4280, 0xc000313320, 0x81)
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:285 +0xca
created by github.com/elastic/beats/filebeat/input/log.(*Harvester).Run
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:278 +0x1f5

goroutine 185 [select]:
github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc000857ba0)
        /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c
created by github.com/elastic/beats/filebeat/input/log.NewInput
        /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x7e2

goroutine 178 [select]:
github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc0006622a0)
        /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x144
github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc0001ec6d0, 0xc0006622a0)
        /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x65
created by github.com/elastic/beats/filebeat/input.(*Runner).Start
        /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x198

goroutine 140 [select]:
github.com/elastic/beats/filebeat/channel.(*subOutlet).OnEvent(0xc0008bc840, 0xf513699, 0xed6346191, 0x0, 0x0, 0xc000b15e00, 0x31fb340, 0xc0008441b0, 0x0, 0xc000844100)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:84 +0x231
github.com/elastic/beats/filebeat/harvester.(*Forwarder).Send(0xc00047be20, 0xf513699, 0xed6346191, 0x0, 0x0, 0xc000b15e00, 0x31fb340, 0xc0008441b0, 0x0, 0xbfa0c91b5c0c4b5a, ...)
        /go/src/github.com/elastic/beats/filebeat/harvester/forwarder.go:50 +0x65
github.com/elastic/beats/filebeat/input/log.(*Harvester).onMessage(0xc0003b4280, 0xc00047be20, 0x0, 0x0, 0x0, 0x3765960, 0xc000796c30, 0xc000313320, 0x81, 0x5f5d9, ...)
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:457 +0x536
github.com/elastic/beats/filebeat/input/log.(*Harvester).Run(0xc0003b4280, 0x0, 0x0)
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:343 +0xb06
github.com/elastic/beats/filebeat/harvester.(*Registry).Start.func1(0xc0004e1840, 0x37333a0, 0xc0003b4280)
        /go/src/github.com/elastic/beats/filebeat/harvester/registry.go:96 +0x9a
created by github.com/elastic/beats/filebeat/harvester.(*Registry).Start
        /go/src/github.com/elastic/beats/filebeat/harvester/registry.go:90 +0x153

goroutine 145 [select]:
github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc000857a00)
        /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c
created by github.com/elastic/beats/filebeat/input/log.NewInput
        /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x7e2

goroutine 142 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x3733220, 0xc0008bc840, 0xc0000a5c20)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 139 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x3733220, 0xc0007cb290, 0xc0000a4de0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 93 [sync.Cond.Wait]:
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0001d7d00, 0x0)
        /usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0001d7cf0)
        /usr/local/go/src/sync/cond.go:56 +0x9d
github.com/elastic/beats/vendor/golang.org/x/net/http2.(*pipe).Read(0xc0001d7ce8, 0xc00018f000, 0x200, 0x200, 0x0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/vendor/golang.org/x/net/http2/pipe.go:64 +0xa6
github.com/elastic/beats/vendor/golang.org/x/net/http2.transportResponseBody.Read(0xc0001d7cc0, 0xc00018f000, 0x200, 0x200, 0x0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/vendor/golang.org/x/net/http2/transport.go:1959 +0xac
encoding/json.(*Decoder).refill(0xc00055ab00, 0xc00084da40, 0x1)
        /usr/local/go/src/encoding/json/stream.go:161 +0xeb
encoding/json.(*Decoder).readValue(0xc00055ab00, 0x0, 0x0, 0x2f28660)
        /usr/local/go/src/encoding/json/stream.go:136 +0x1dc
encoding/json.(*Decoder).Decode(0xc00055ab00, 0x2f93620, 0xc00084da40, 0x0, 0xc0004b4cf0)
        /usr/local/go/src/encoding/json/stream.go:63 +0x79
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000396e10, 0xc0005ec800, 0x400, 0x400, 0xc0004b4d60, 0x13c37b8, 0x38)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:150 +0x1a1
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0002373b0, 0x0, 0x3719ee0, 0xc00054e340, 0xc000465080, 0x4, 0xc000465080, 0x5003080, 0xc0004a6000)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
github.com/elastic/beats/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00084da20, 0xc0002d6cc0, 0xc000464f00, 0xc000464f88, 0xc000465038, 0xc0002d6d18, 0xc0000596d8)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x7c
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00054e300)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:104 +0x175
created by github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:71 +0xbe

goroutine 97 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x37331e0, 0xc0008507e0, 0xc0000a5260)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 137 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x37331e0, 0xc0007cb260, 0xc0000a5260)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 131 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x3733220, 0xc000850810, 0xc0000a4de0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 184 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x3733220, 0xc0009540f0, 0xc0000a4de0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 130 [chan receive]:
github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc000850810, 0x37331e0, 0xc0008507e0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150
created by github.com/elastic/beats/filebeat/channel.SubOutlet
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8

goroutine 181 [chan send]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*pipelineEventCB).pushMsg(...)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:226
github.com/elastic/beats/libbeat/publisher/pipeline.(*pipelineEventCB).onEvents(...)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:217
github.com/elastic/beats/libbeat/publisher/pipeline.(*eventDataACK).onACK(0xc0004fb840, 0x1, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:396 +0xeb
github.com/elastic/beats/libbeat/publisher/pipeline.(*boundGapCountACK).onACK(0xc00094e180, 0x1, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:332 +0x5a
github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc00094e190)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:165 +0x1a2
created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182

goroutine 138 [chan receive]:
github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc0007cb290, 0x37331e0, 0xc0007cb260)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150
created by github.com/elastic/beats/filebeat/channel.SubOutlet
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8

goroutine 183 [chan receive]:
github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc0009540f0, 0x37331e0, 0xc0009540c0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150
created by github.com/elastic/beats/filebeat/channel.SubOutlet
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8

goroutine 59 [runnable]:
github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).run(0xc00022d9a0)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:316 +0x1cf
github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker.func1(0xc000490a80, 0x3714620, 0xc00022d9a0)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:175 +0x67
created by github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:173 +0x373

goroutine 60 [select]:
github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc000502640)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0x107
github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker.func2(0xc000490a80, 0xc000502640)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:179 +0x61
created by github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:177 +0x3a2

goroutine 61 [select]:
github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*consumer).Get(0xc000461420, 0x0, 0x0, 0x0, 0xc000071e01, 0x1)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/consume.go:67 +0xe9
github.com/elastic/beats/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc000502730, 0x3714660, 0xc000461420)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:167 +0x411
created by github.com/elastic/beats/libbeat/publisher/pipeline.newEventConsumer
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:77 +0x154

goroutine 62 [select]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*retryer).loop(0xc000502780)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:139 +0x1fd
created by github.com/elastic/beats/libbeat/publisher/pipeline.newRetryer
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:91 +0x13d

goroutine 63 [chan receive]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc00054eb80)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:84 +0xc4
created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:48 +0xe0

goroutine 64 [chan receive]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc00054ebc0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:84 +0xc4
created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:48 +0xe0

goroutine 65 [select]:
github.com/elastic/beats/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0xc0005f31a0)
        /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:129 +0x410
github.com/elastic/beats/libbeat/monitoring/report/log.MakeReporter.func1(0xc0005f31a0)
        /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:107 +0x5d
created by github.com/elastic/beats/libbeat/monitoring/report/log.MakeReporter
        /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:105 +0x157

goroutine 99 [chan receive]:
github.com/elastic/beats/libbeat/service.HandleSignals.func1(0xc000098c60, 0xc000448f60, 0xc0003c8a60, 0xc000461600)
        /go/src/github.com/elastic/beats/libbeat/service/service.go:49 +0x41
created by github.com/elastic/beats/libbeat/service.HandleSignals
        /go/src/github.com/elastic/beats/libbeat/service/service.go:48 +0x115

goroutine 89 [select]:
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000795b80)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/shared_informer.go:570 +0x18d
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000491e90, 0xc00048e6f0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 101 [select]:
github.com/elastic/beats/filebeat/beater.(*registrarLogger).Published(0xc000449090, 0xc000d2d680, 0x1, 0x1)
        /go/src/github.com/elastic/beats/filebeat/beater/channels.go:53 +0xd3
github.com/elastic/beats/filebeat/beater.(*eventACKer).ackEvents(0xc0005f33b0, 0xc000854b80, 0x1, 0x1)
        /go/src/github.com/elastic/beats/filebeat/beater/acker.go:65 +0x3b7
github.com/elastic/beats/libbeat/publisher/pipeline.(*pipelineEventCB).reportEventsData(0xc00054ed80, 0xc000854b80, 0x1, 0x1, 0x1)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:319 +0x91
github.com/elastic/beats/libbeat/publisher/pipeline.(*pipelineEventCB).worker(0xc00054ed80)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:255 +0x264
created by github.com/elastic/beats/libbeat/publisher/pipeline.newPipelineEventCB
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:196 +0x15b

goroutine 102 [runnable]:
syscall.Syscall(0x4a, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.Fsync(0x8, 0xc000084c01, 0x1456357)
        /usr/local/go/src/syscall/zsyscall_linux_amd64.go:449 +0x40
internal/poll.(*FD).Fsync(0xc000b6bf80, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_fsync_posix.go:17 +0x8f
os.(*File).Sync(0xc000354980, 0x2a, 0x0)
        /usr/local/go/src/os/file_posix.go:113 +0x4c
github.com/elastic/beats/libbeat/common/file.SafeFileRotate(0xc0003cd240, 0x34, 0xc000053c80, 0x38, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/common/file/helper_other.go:44 +0x13c
github.com/elastic/beats/filebeat/registrar.(*Registrar).writeRegistry(0xc000098de0, 0x2c, 0xc000084e88)
        /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:398 +0xeb
github.com/elastic/beats/filebeat/registrar.(*Registrar).flushRegistry(0xc000098de0)
        /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:373 +0x2f
github.com/elastic/beats/filebeat/registrar.(*Registrar).Run(0xc000098de0)
        /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:302 +0x271
created by github.com/elastic/beats/filebeat/registrar.(*Registrar).Start
        /go/src/github.com/elastic/beats/filebeat/registrar/registrar.go:272 +0xe8

goroutine 105 [chan receive]:
github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000663aa0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xe0
created by github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x132

goroutine 106 [select]:
github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000491ea0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:215 +0x344
created by github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:56 +0x1d4

goroutine 107 [select]:
github.com/elastic/beats/libbeat/autodiscover.(*Autodiscover).worker(0xc000795c80)
        /go/src/github.com/elastic/beats/libbeat/autodiscover/autodiscover.go:123 +0x117
created by github.com/elastic/beats/libbeat/autodiscover.(*Autodiscover).Start
        /go/src/github.com/elastic/beats/libbeat/autodiscover/autodiscover.go:112 +0x10e

goroutine 108 [sync.Cond.Wait]:
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0003b7ba8, 0x1)
        /usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0003b7b98)
        /usr/local/go/src/sync/cond.go:56 +0x9d
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0003b7b80, 0xc00005c310, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:423 +0xa6
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000795d00)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/controller.go:150 +0x40
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0007bfe00)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007bfe00, 0x3b9aca00, 0x0, 0xc0007ae001, 0xc0000a54a0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*controller).Run(0xc000795d00, 0xc0000a54a0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/controller.go:124 +0x2fd
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0004e8ab0, 0xc0000a54a0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/shared_informer.go:280 +0x47f
created by github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start
        /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:146 +0x71

goroutine 151 [select]:
github.com/elastic/beats/filebeat/input/log.(*Input).stopWhenDone(0xc000857860)
        /go/src/github.com/elastic/beats/filebeat/input/log/input.go:753 +0x11c
created by github.com/elastic/beats/filebeat/input/log.NewInput
        /go/src/github.com/elastic/beats/filebeat/input/log/input.go:155 +0x7e2

goroutine 96 [select]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc0008ffdb0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:141 +0xff
created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182

goroutine 88 [chan receive]:
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1.1(0xc000625380, 0x0, 0xc000663ec0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/shared_informer.go:602 +0x54
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0x0, 0xc0005e0dd8, 0xc0003d6780, 0xc0007b61e0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:284 +0x51
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/shared_informer.go:601 +0x79
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0004b0f40)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0005e0f40, 0xdf8475800, 0x0, 0x13e5601, 0xc0000596e0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc000795b80)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/shared_informer.go:599 +0x9b
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000491e90, 0xc00048e6e0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 112 [chan receive]:
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000491e30, 0xc0000a5680)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/shared_informer.go:478 +0x49
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x2e
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00005b4f0, 0xc0007a8b00)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 113 [chan receive]:
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc0000a54a0, 0xc000795d00)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/controller.go:103 +0x34
created by github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*controller).Run
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0xae

goroutine 114 [select]:
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc00022dd60, 0x371a3a0, 0xc00054e300, 0xc00039fc00, 0xc0008ff620, 0xc0000a54a0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/reflector.go:329 +0x1ec
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00022dd60, 0xc0000a54a0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/reflector.go:300 +0x833
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/reflector.go:124 +0x33
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0004b16f0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00039fef0, 0x3b9aca00, 0x0, 0x1, 0xc0000a54a0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00022dd60, 0xc0000a54a0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/reflector.go:123 +0x169
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x2e
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00005b530, 0xc0007a8bc0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 91 [select]:
github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc00022dd60, 0xc0000a54a0, 0xc000b1f980, 0xc0008ff620)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/reflector.go:235 +0x178
created by github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/tools/cache/reflector.go:229 +0x260

goroutine 121 [sync.Cond.Wait]:
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc00004e510, 0x0)
        /usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc00004e500)
        /usr/local/go/src/sync/cond.go:56 +0x9d
github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue.(*Type).Get(0xc000663aa0, 0x0, 0x0, 0xc00034de00)
        /go/src/github.com/elastic/beats/vendor/k8s.io/client-go/util/workqueue/queue.go:145 +0xa0
github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process(0xc000491f10, 0x3751220, 0xc00004e540, 0x20202020200a3401)
        /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:184 +0x38
github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start.func1()
        /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:158 +0x48
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00005c010)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00005c010, 0x3b9aca00, 0x0, 0x2064756f6c632001, 0xc0000a54a0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00005c010, 0x3b9aca00, 0xc0000a54a0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start
        /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:157 +0x1d9

goroutine 122 [chan receive]:
github.com/elastic/beats/filebeat/beater.waitChannel.func1()
        /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:79 +0x33
github.com/elastic/beats/filebeat/beater.(*signalWait).Add.func1(0xc00005c020, 0xc000448fc0)
        /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:52 +0x27
created by github.com/elastic/beats/filebeat/beater.(*signalWait).Add
        /go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:51 +0x4c

goroutine 119 [IO wait]:
internal/poll.runtime_pollWait(0x7f629ae89c40, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0007c2118, 0x72, 0x5600, 0x562d, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007c2100, 0xc000810000, 0x562d, 0x562d, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0007c2100, 0xc000810000, 0x562d, 0x562d, 0x203000, 0x13c2f46, 0xc000237360)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000100d8, 0xc000810000, 0x562d, 0x562d, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc00084da00, 0xc000810000, 0x562d, 0x562d, 0x13fda7e, 0xc0002cc8c0, 0xc00047d970)
        /usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc0007c4258, 0x36e3540, 0xc00084da00, 0x13c0f25, 0x2f96c20, 0x321b680)
        /usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc0007c4000, 0x36e8880, 0xc0000100d8, 0x5, 0xc0000100d8, 0xc0001fa910)
        /usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0007c4000, 0x0, 0x0, 0x13f027e)
        /usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc0007c4000, 0xc0007e1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc0007b6c60, 0xc0003f6738, 0x9, 0x9, 0xc0007b6480, 0xc00047ddd8, 0xc00047dcf0)
        /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x36e33a0, 0xc0007b6c60, 0xc0003f6738, 0x9, 0x9, 0x9, 0x13bba75, 0xc0008ff680, 0xc00047ddd8)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
github.com/elastic/beats/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003f6738, 0x9, 0x9, 0x36e33a0, 0xc0007b6c60, 0x0, 0x0, 0xc000396cf0, 0x0)
        /go/src/github.com/elastic/beats/vendor/golang.org/x/net/http2/frame.go:237 +0x87
github.com/elastic/beats/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003f6700, 0xc000396d20, 0x0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
github.com/elastic/beats/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00047dfb8, 0x1ee415d, 0xc000099a40)
        /go/src/github.com/elastic/beats/vendor/golang.org/x/net/http2/transport.go:1686 +0x8c
github.com/elastic/beats/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc0007c0600)
        /go/src/github.com/elastic/beats/vendor/golang.org/x/net/http2/transport.go:1614 +0xa3
created by github.com/elastic/beats/vendor/golang.org/x/net/http2.(*Transport).newClientConn
        /go/src/github.com/elastic/beats/vendor/golang.org/x/net/http2/transport.go:673 +0x62f

goroutine 152 [select]:
github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc0008ff800)
        /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x144
github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc0003c9400, 0xc0008ff800)
        /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x65
created by github.com/elastic/beats/filebeat/input.(*Runner).Start
        /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x198

goroutine 155 [chan send]:
github.com/elastic/beats/libbeat/publisher/pipeline.(*pipelineEventCB).pushMsg(...)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:226
github.com/elastic/beats/libbeat/publisher/pipeline.(*pipelineEventCB).onEvents(...)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline_ack.go:217
github.com/elastic/beats/libbeat/publisher/pipeline.(*eventDataACK).onACK(0xc0004e1800, 0x1, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:396 +0xeb
github.com/elastic/beats/libbeat/publisher/pipeline.(*boundGapCountACK).onACK(0xc0002d6c60, 0x1, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:332 +0x5a
github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).ackLoop(0xc0002d6c70)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:165 +0x1a2
created by github.com/elastic/beats/libbeat/publisher/pipeline.(*gapCountACK).init
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/acker.go:129 +0x182

goroutine 156 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x37331e0, 0xc00089cba0, 0xc0000a5260)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 157 [chan receive]:
github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc00089cbd0, 0x37331e0, 0xc00089cba0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:44 +0x150
created by github.com/elastic/beats/filebeat/channel.SubOutlet
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8

goroutine 158 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x3733220, 0xc00089cbd0, 0xc0000a4de0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 159 [select]:
github.com/elastic/beats/filebeat/channel.(*subOutlet).OnEvent(0xc000521410, 0x2282b626, 0xed634627d, 0x0, 0x0, 0xc000b10f30, 0x31fb340, 0xc000cdfe60, 0x0, 0xc000cdfe00)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:84 +0x231
github.com/elastic/beats/filebeat/harvester.(*Forwarder).Send(0xc0003a1e20, 0x2282b626, 0xed634627d, 0x0, 0x0, 0xc000b10f30, 0x31fb340, 0xc000cdfe60, 0x0, 0xbfa0c91b5bf4549c, ...)
        /go/src/github.com/elastic/beats/filebeat/harvester/forwarder.go:50 +0x65
github.com/elastic/beats/filebeat/input/log.(*Harvester).onMessage(0xc0000d9180, 0xc0003a1e20, 0x0, 0x0, 0x0, 0x3765960, 0xc000170f70, 0xc0000bf100, 0x7f, 0x2b24ab, ...)
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:457 +0x536
github.com/elastic/beats/filebeat/input/log.(*Harvester).Run(0xc0000d9180, 0x0, 0x0)
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:343 +0xb06
github.com/elastic/beats/filebeat/harvester.(*Registry).Start.func1(0xc00054e900, 0x37333a0, 0xc0000d9180)
        /go/src/github.com/elastic/beats/filebeat/harvester/registry.go:96 +0x9a
created by github.com/elastic/beats/filebeat/harvester.(*Registry).Start
        /go/src/github.com/elastic/beats/filebeat/harvester/registry.go:90 +0x153

goroutine 161 [select]:
github.com/elastic/beats/filebeat/channel.CloseOnSignal.func1(0x3733220, 0xc000521410, 0xc0000589c0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:112 +0xb7
created by github.com/elastic/beats/filebeat/channel.CloseOnSignal
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:111 +0x76

goroutine 162 [select]:
github.com/elastic/beats/filebeat/input/log.(*Harvester).Run.func2(0xc0000d9180, 0xc0000bf100, 0x7f)
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:285 +0xca
created by github.com/elastic/beats/filebeat/input/log.(*Harvester).Run
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:278 +0x1f5

goroutine 163 [select]:
github.com/elastic/beats/filebeat/input/log.(*Harvester).monitorFileSize(0xc0000d9180)
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:362 +0x105
created by github.com/elastic/beats/filebeat/input/log.(*Harvester).Run
        /go/src/github.com/elastic/beats/filebeat/input/log/harvester.go:299 +0x29d

goroutine 186 [select]:
github.com/elastic/beats/filebeat/input.(*Runner).Run(0xc000663680)
        /go/src/github.com/elastic/beats/filebeat/input/input.go:147 +0x144
github.com/elastic/beats/filebeat/input.(*Runner).Start.func1(0xc0009407e0, 0xc000663680)
        /go/src/github.com/elastic/beats/filebeat/input/input.go:132 +0x65
created by github.com/elastic/beats/filebeat/input.(*Runner).Start
        /go/src/github.com/elastic/beats/filebeat/input/input.go:125 +0x198

goroutine 141 [running]:
runtime.throw(0x330841d, 0x15)
        /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc00047f7e0 sp=0xc00047f7b0 pc=0x13e6082
runtime.mapassign_faststr(0x31f9be0, 0xc000397380, 0xc0000c08a0, 0xb, 0xc000b3e000)
        /usr/local/go/src/runtime/map_faststr.go:291 +0x3fe fp=0xc00047f848 sp=0xc00047f7e0 pc=0x13ca19e
github.com/elastic/beats/libbeat/common.MapStr.deepUpdateMap(0xc000397380, 0xc000b06210, 0xc00084f001)
        /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:90 +0x13e fp=0xc00047f920 sp=0xc00047f848 pc=0x18593be
github.com/elastic/beats/libbeat/common.deepUpdateValue(0x31f9be0, 0xc000397380, 0xc000b06210, 0x1, 0xc00084f178, 0xc0000c0888)
        /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:105 +0xb0 fp=0xc00047f958 sp=0xc00047f920 pc=0x1859750
github.com/elastic/beats/libbeat/common.MapStr.deepUpdateMap(0xc000b15e90, 0xc000b061e0, 0xc000397301)
        /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:87 +0x25d fp=0xc00047fa30 sp=0xc00047f958 pc=0x18594dd
github.com/elastic/beats/libbeat/common.MapStr.DeepUpdate(...)
        /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:71
github.com/elastic/beats/libbeat/processors/actions.(*addFields).Run(0xc0001fab30, 0xc0008901c0, 0xc0008901c0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/processors/actions/add_fields.go:81 +0x51 fp=0xc00047fa58 sp=0xc00047fa30 pc=0x1b0b7f1
github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc000359f20, 0xc0008901c0, 0xc0008901c0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94 fp=0xc00047fb08 sp=0xc00047fa58 pc=0x1b19594
github.com/elastic/beats/libbeat/publisher/processing.(*group).Run(0xc00089cae0, 0xc0008901c0, 0xed634618c, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:104 +0x94 fp=0xc00047fbb8 sp=0xc00047fb08 pc=0x1b19594
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).publish(0xc000353e80, 0xf513699, 0xed6346191, 0x0, 0x0, 0xc000b15e00, 0x31fb340, 0xc0008441b0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:89 +0x571 fp=0xc00047fde8 sp=0xc00047fbb8 pc=0x1b1d851
github.com/elastic/beats/libbeat/publisher/pipeline.(*client).Publish(0xc000353e80, 0xf513699, 0xed6346191, 0x0, 0x0, 0xc000b15e00, 0x31fb340, 0xc0008441b0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:68 +0xc2 fp=0xc00047fe80 sp=0xc00047fde8 pc=0x1b1d282
github.com/elastic/beats/filebeat/channel.(*outlet).OnEvent(0xc00089cba0, 0xf513699, 0xed6346191, 0x0, 0x0, 0xc000b15e00, 0x31fb340, 0xc0008441b0, 0x0, 0x2000001536a01)
        /go/src/github.com/elastic/beats/filebeat/channel/outlet.go:64 +0x7e fp=0xc00047fed8 sp=0xc00047fe80 pc=0x269611e
github.com/elastic/beats/filebeat/channel.SubOutlet.func1(0xc0008bc840, 0x37331e0, 0xc00089cba0)
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:45 +0x10d fp=0xc00047ffc8 sp=0xc00047fed8 pc=0x26967fd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00047ffd0 sp=0xc00047ffc8 pc=0x1416ec1
created by github.com/elastic/beats/filebeat/channel.SubOutlet
        /go/src/github.com/elastic/beats/filebeat/channel/util.go:43 +0xf8

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-04-24T15:24:08Z

Pinging @elastic/integrations-platforms (Team:Platforms)

jsoriano · 2020-04-30T18:06:02Z

Hey @brummelbaer,

Thanks for your report!
I am struggling to reproduce this issue. Is there any additional configuration you are using, as custom fields or some other processor?
Are you providing additional configuration using hints?

brummelbaer · 2020-05-05T14:56:37Z

Hi @jsoriano,

Thank you for looking into this.

I am struggling to reproduce this issue. Is there any additional configuration you are using, as custom fields or some other processor?

I was using extra fields and this might be the reason why you were not able to reproduce the error. I am sorry that I left these out in my previous post. I just ran a quick test and it failed as soon as I added the extra field.

    processors:
      - add_cloud_metadata: ~
      - add_host_metadata:
      - add_fields:
          target: cloud
          fields:
            environment: "env-name"

Are you providing additional configuration using hints?

No I am not providing any additional configuration.

jsoriano · 2020-05-05T15:07:24Z

@brummelbaer ok, it makes more sense now 🙂, the trace you reported contained references to the add_fields processor. Even if it is not used explicitly, it is also used internally by some other features, so I thought that it might be the case here.

I think that the problem is going to be that both add_cloud_metadata and add_fields are writing to the same cloud object, this can lead to unexpected results in some cases, but it shouldn't fail with these concurrent map writes.

Thanks!

jsoriano · 2020-05-06T10:12:31Z

An update, I could reproduce this issue, also after starting an ES cluster with ECK:

...
2020-05-06T09:32:57.802Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/containers/quickstart-es-default-0_default_elasticsearch-b237206fba35fed0c137dbd8ef577620c4c2d5ba534a0d9d52702876f8db7bbb.log
fatal error: concurrent map writes
...

These pods have a hint to use the elasticsearch module to parse the logs, so it must be some interaction between these processors, and something used in the elasticsearch module.

...
kind: Pod
metadata:
  annotations:
    co.elastic.logs/module: elasticsearch
...

Still investigating what can be the issue.

elasticmachine · 2020-05-06T10:17:02Z

Pinging @elastic/integrations-services (Team:Services)

brummelbaer · 2020-05-06T10:32:24Z

@jsoriano I'm glad that your are looking into this issue and also were able to reproduce the error. If you need some more information don't hesitate to contact me

jsoriano · 2020-05-07T13:53:09Z

I think I have found the issue, and I have also found that it is already fixed on master 😄 with this change: #13808. Using an image with this fix I am not able to reproduce the issue.

I am closing this issue and opening backports for 7.6 and 6.8, it is already backported to 7.7, so it will be available in 7.7.0.

brummelbaer · 2020-05-08T06:47:12Z

Thank you for working on this topic. Do you know when 7.7 will be released?

jsoriano · 2020-05-08T11:00:06Z

It will probably be released soon. If you want to try in the meantime, the 7.7 branch has been already created and there are some prebuilt snapshots in https://beats-ci.elastic.co/job/elastic+beats+7.7+multijob-package-linux/lastSuccessfulBuild/gcsObjects/

jsoriano added [zube]: Inbox bug containers Related to containers use case Filebeat Filebeat Team:Platforms Label for the Integrations - Platforms team labels Apr 24, 2020

andresrc added [zube]: Investigate and removed [zube]: Inbox labels Apr 28, 2020

jsoriano self-assigned this Apr 29, 2020

jsoriano added the Team:Services (Deprecated) Label for the former Integrations-Services team label May 6, 2020

jsoriano closed this as completed May 7, 2020

zube bot added [zube]: Done and removed [zube]: Investigate labels May 7, 2020

jsoriano mentioned this issue May 7, 2020

Fix renaming cloud metadata processor fields #13808

Merged

andresrc removed the [zube]: Done label May 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filebeat kubernetes fatal error: concurrent map writes #17968

Filebeat kubernetes fatal error: concurrent map writes #17968

brummelbaer commented Apr 24, 2020

elasticmachine commented Apr 24, 2020

jsoriano commented Apr 30, 2020

brummelbaer commented May 5, 2020

jsoriano commented May 5, 2020

jsoriano commented May 6, 2020

elasticmachine commented May 6, 2020

brummelbaer commented May 6, 2020

jsoriano commented May 7, 2020 •

edited

Loading

brummelbaer commented May 8, 2020

jsoriano commented May 8, 2020

Filebeat kubernetes fatal error: concurrent map writes #17968

Filebeat kubernetes fatal error: concurrent map writes #17968

Comments

brummelbaer commented Apr 24, 2020

Setup information

Filebeat configuration

Debug log

elasticmachine commented Apr 24, 2020

jsoriano commented Apr 30, 2020

brummelbaer commented May 5, 2020

jsoriano commented May 5, 2020

jsoriano commented May 6, 2020

elasticmachine commented May 6, 2020

brummelbaer commented May 6, 2020

jsoriano commented May 7, 2020 • edited Loading

brummelbaer commented May 8, 2020

jsoriano commented May 8, 2020

jsoriano commented May 7, 2020 •

edited

Loading