Memory leak when using the Redis logger

[Naugrimm]

I suspect a memory leak when using the Redis logger.

The following chart shows the amount of memory our redis instance has used over time:

Redis was restarted around 2023-07-23 09:30 to increase maxmemory. The drop at the end occurred the go-dnscollector process was stopped.

Here is the amount of memory the go-dnscollector used in the same timespan:

The following config is in use:

global:
  trace:
    verbose: true
  server-identity: "ns-002"
  text-format: "timestamp-rfc3339ns identity operation rcode queryip queryport family protocol length qname qtype latency"
  text-format-delimiter: " "
  text-format-boundary: "\""
multiplexer:
  collectors:
    - name: sniffer-ens7
      afpacket-sniffer:
        port: 53
        device: ens7
      transforms:
        filtering:
          drop-queryip-file: "/opt/dnscollector/drop-queryip-file.conf"
          drop-domain-file: "/opt/dnscollector/drop-domains-regex.conf"
  loggers:
    - name: prometheus
      prometheus:
        listen-ip: 127.0.0.1
        listen-port: 9154
        basic-auth-enable: false
        basic-auth-login: login
        basic-auth-pwd: password
        tls-support: false
        tls-mutual: false
        tls-min-version: 1.2
        cert-file: ""
        key-file: ""
        prometheus-prefix: "dnscollector"
        top-n: 10
        chan-buffer-size: 65535
        histogram-metrics-enabled: true
    - name: redis
      redispub:
        transport: tcp
        remote-address: redis
        remote-port: 6379
        redis-channel: dnscollector

  routes:
    - from: [ sniffer-ens7 ]
      to: [ prometheus, redis ]

[dmachard]

Hi, can you share the version of the collector used ?

@gallypette what do you think about this report ?

[Naugrimm]

I am using 0.33.0:

 # /opt/dnscollector/go-dnscollector -version
0.33.0

[gallypette]

Hi all,
Is this resident set size?

I used pprof to dig into the logger but I don't find a leak nor managed to reproduce with the following config:

multiplexer:
  collectors:
    - name: wired
      afpacket-sniffer:
        port: 53
        device: enp0s31f6

  loggers:
    - name: redis
      redispub:
        mode: json
        transport: tcp
        remote-address: 127.0.0.1
        remote-port: 6379
        sock-path: null
        connect-timeout: 5
        retry-interval: 10
        flush-interval: 2
        tls-support: false
        tls-insecure: false
        text-format: "timestamp-rfc3339ns identity operation rcode queryip queryport family protocol length qname qtype latency"
        delimiter: "\n"
        buffer-size: 65535

    - name: console
      stdout:
        mode: text

  routes:
    - from: [ wired ]
      to: [ redis ]

Can I ask what version of Golang you use?
I would also be curious to see your client-output-buffer-limit pubsub setting.

[Naugrimm]

Hi @gallypette,

yes, it is almost completely RSS. Here you can see the chart from the first post split by memtype:

The go-Version used is go version go1.10 linux/amd64.

The requested Redis-setting is as follows:

redis-cli> config get client-output-buffer-limit
1) "client-output-buffer-limit"
2) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"