global telemetry: hang during shutdown with high load

[dmachard]

Describe the bug

how to reproduce: send ~70kreq/s and try to stop the DNscollector

INFO: 2024/10/21 20:39:53.081184 worker - [tap] (conn #4) dnstap processor - stopping monitor...
INFO: 2024/10/21 20:39:53.081195 worker - [tap] (conn #4) dnstap processor - monitor terminated
INFO: 2024/10/21 20:39:53.081205 worker - [tap] (conn #4) dnstap processor - stopping collect...
INFO: 2024/10/21 20:39:53.081217 worker - [tap] (conn #4) dnstap processor - collection terminated
INFO: 2024/10/21 20:39:53.081227 worker - [tap] dnstap - conn #4 - cleanup connection handler terminated
INFO: 2024/10/21 20:39:53.081245 worker - [tap] dnstap - conn #2 - force to cleanup the connection handler
INFO: 2024/10/21 20:39:53.081291 worker - [tap] dnstap - conn #2 - connection closed with peer 172.19.0.2:53604
INFO: 2024/10/21 20:39:53.081302 worker - [tap] dnstap - (conn #2 - connection handler terminated
INFO: 2024/10/21 20:39:53.081311 worker - [tap] dnstap - collection terminated
INFO: 2024/10/21 20:39:53.081327 worker - [tofile] file - stopping monitor...
INFO: 2024/10/21 20:39:53.081366 worker - [tap] (conn #1) dnstap processor - monitor terminated
INFO: 2024/10/21 20:39:53.081376 worker - [tap] (conn #1) dnstap processor - stopping collect...
INFO: 2024/10/21 20:39:53.081420 worker - [tap] (conn #2) dnstap processor - stopping monitor...
INFO: 2024/10/21 20:39:53.081433 worker - [tap] (conn #2) dnstap processor - monitor terminated
INFO: 2024/10/21 20:39:53.081441 worker - [tap] (conn #2) dnstap processor - stopping collect...
INFO: 2024/10/21 20:39:53.081451 worker - [tap] (conn #2) dnstap processor - collection terminated
INFO: 2024/10/21 20:39:53.081459 worker - [tap] dnstap - conn #2 - cleanup connection handler terminated
INFO: 2024/10/21 20:39:53.081775 worker - [tofile] file - monitor terminated
INFO: 2024/10/21 20:39:53.081793 worker - [tofile] file - stopping collect...



signal: killed

global:
  trace:
    verbose: true

pipelines:
  - name: tap
    dnstap:
      listen-ip: 0.0.0.0
      listen-port: 6000
    transforms:
      normalize:
        qname-lowercase: true
        qname-replace-nonprintable: true
    routing-policy:
      forward: [ tofile, ]
      dropped: [ ]

  - name: tofile
    logfile:
      file-path: /tmp/dnstap.log
      max-size: 100
      max-files: 10
      max-batch-size: 262144
      flush-interval: 1
      compress: false
      compress-interval: 5
      compress-postcommand: null
      mode: json
      text-format: ""
      postrotate-command: null
      postrotate-delete-success: false
      chan-buffer-size: 0