new global settings for default buffer size (#739)

config.yml

@@ -12,6 +12,7 @@ global:
   pid-file: ""
   worker:
     interval-monitor: 10
+    buffer-size: 4096
   telemetry:
     enabled: true
     web-path: "/metrics"

docs/collectors/collector_afpacket.md

@@ -25,7 +25,8 @@ Options:
   > Enable IP defrag support
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 Defaults:
 
@@ -35,5 +36,5 @@ Defaults:
     port: 53
     device: wlp2s0
     enable-defrag-ip: true
-    chan-buffer-size: 65535
+    chan-buffer-size: 0
 ```

docs/collectors/collector_dnsmessage.md

@@ -5,7 +5,8 @@ Collector to match specific DNS messages.
 Options:
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 * `matching` (map)
     * `include` (map)

docs/collectors/collector_dnstap.md

@@ -36,8 +36,9 @@ Options:
   > Set whether to send a TCP Reset to force the cleanup of the connection on the remote side when the server exits.
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
-  
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
+
 * `disable-dnsparser"` (bool)
   > Disable the minimalist DNS parser. Some JSON keys should not be available, such as `dns.id`, `dns.flags`, ...
 
@@ -63,7 +64,7 @@ Defaults:
     key-file: ""
     sock-rcvbuf: 0
     reset-conn: true
-    chan-buffer-size: 65535
+    chan-buffer-size: 0
     disable-dnsparser: true
     extended-support: false
     compression: none

docs/collectors/collector_fileingestor.md

@@ -27,7 +27,8 @@ Options:
   > Determines whether the pcap file should be deleted after ingestion.
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 Defaults:
 
@@ -38,5 +39,5 @@ Defaults:
     watch-mode: pcap
     pcap-dns-port: 53
     delete-after: false
-    chan-buffer-size: 65535
+    chan-buffer-size: 0
 ```

docs/collectors/collector_powerdns.md

@@ -34,7 +34,8 @@ Settings:
   > Set whether to send a TCP Reset to force the cleanup of the connection on the remote side when the server exits.
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 * `add-dns-payload` (bool)
   > PowerDNS protobuf message does not contain a DNS payload; use this setting to add a raw DNS payload.
@@ -51,7 +52,7 @@ Defaults:
     cert-file: ""
     key-file: ""
     reset-conn: true
-    chan-buffer-size: 65535
+    chan-buffer-size: 0
     add-dns-payload: false
 ```
 

docs/collectors/collector_tail.md

@@ -22,6 +22,10 @@ Options:
 * `pattern-reply` (string)
   > Specifies the regular expression pattern used to match replies.
 
+* `chan-buffer-size` (int)
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
+
 Defaults:
 
 ```yaml
@@ -35,4 +39,5 @@ Defaults:
     pattern-reply: "^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_RESPONSE) (?P<rcode>[^ ]*)
       (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b
       (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$"
+    chan-buffer-size: 0
 ```

docs/collectors/collector_tzsp.md

@@ -12,7 +12,8 @@ Options:
   > Set the local port that the server will bind to.
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 Defaults:
 
@@ -21,7 +22,7 @@ Defaults:
   tzsp:
     listen-ip: 0.0.0.0
     listen-port: 10000
-    chan-buffer-size: 65535
+    chan-buffer-size: 0
 ```
 
 Example rules for Mikrotik brand devices to send the traffic (only works if routed or the device serves as DNS server).

docs/collectors/collector_xdp.md

@@ -20,13 +20,14 @@ Options:
   > Interface name to use for XDP sniffing.
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 Defaults:
 
 ```yaml
 - name: sniffer
   xdp-sniffer:
     device: wlp2s0
-    chan-buffer-size: 65535
+    chan-buffer-size: 0
 ```

docs/configuration.md

@@ -4,18 +4,15 @@ The configuration of DNS-collector is done through one yaml file named [`config.
 
 A typically configuration in [multiplexer](./running_mode.md) mode would have one or more collector to receive DNS traffic, and severals loggers to process the incoming traffics. You can take a look to the list of config [`examples`](examples.md).
 
-## Global
-
 You can find the global settings below
 
-- [Global](#global)
-  - [Trace](#trace)
-  - [Custom text format](#custom-text-format)
-  - [Server identity](#server-identity)
-  - [Pid file](#pid-file)
-  - [Telemetry](#telemetry)
+- [Trace](#trace)
+- [Custom text format](#custom-text-format)
+- [Server identity](#server-identity)
+- [Pid file](#pid-file)
+- [Telemetry](#telemetry)
 
-### Trace
+## Trace
 
 Logs can be enable to have more informations like debug, errors messages generated by the application
 
@@ -48,7 +45,7 @@ INFO: 2022/06/25 20:54:18.174256 [dtap] dnstap collector - running in background
 INFO: 2022/06/25 20:54:18.174286 [dtap] dnstap collector - is listening on [::]:6000
 ```
 
-### Server Identity
+## Server Identity
 
 Set the server identity name. The hostname will be used if empty
 
@@ -57,7 +54,20 @@ global:
   server-identity: "dns-collector"
 ```
 
-### Custom text format
+## Worker
+
+The `interval-monitor` in second(s) is used to count every XX second the number of in/out packets.
+
+The `buffer-size` settings enable to adjust the size of the buffer before discard additional packets. If you encounter the error message buffer is full, xxx packet(s) dropped, consider increasing this parameter to prevent message drops.
+
+```yaml
+global:
+  worker:
+    interval-monitor: 10
+    buffer-size: 4096
+```
+
+## Custom text format
 
 The text format can be customized with the following directives.
 
@@ -126,7 +136,7 @@ Output example:
 2023-04-08T18:27:29.279039Z unbound CLIENT_RESPONSE NOERROR 127.0.0.1 39028 IPv4 UDP 54b google.fr A 0.000000
 ```
 
-### Pid file
+## Pid file
 
 Set path to create DNS-collector PID.
 By default, this settings is empty.
@@ -136,7 +146,7 @@ global:
   pid-file: "/path/to/your/pidfile.pid"
 ```
 
-### Telemetry
+## Telemetry
 
 Enable and configure telemetry
 

docs/loggers/logger_clickhouse.md

@@ -20,6 +20,10 @@ Options:
 * `database` (string)
   > Clickhouse database name
 
+* `chan-buffer-size` (integer)
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
+
 Defaults:
 
 ```yaml
@@ -29,4 +33,5 @@ clickhouse:
   password: "password"
   table: "records"
   database: "dnscollector"
+  chan-buffer-size: 0
 ```

docs/loggers/logger_devnull.md

@@ -3,9 +3,13 @@
 Devnull plugin Logger
 
 Options:
+* `chan-buffer-size` (integer)
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 Default values:
 
 ```yaml
 devnull:
+  chan-buffer-size: 0
 ```

docs/loggers/logger_dnstap.md

@@ -47,7 +47,8 @@ Options:
   > how many DNS messages will be buffered before being sent
 
 * `chan-buffer-size` (integer)
-  > channel buffer size used on incoming dns message, number of messages before to drop it.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 * `extended-support` (boolen)
   > Extend the DNStap message by incorporating additional transformations, such as filtering and ATags, into the extra field.
@@ -75,7 +76,7 @@ Defaults:
     server-id: "dnscollector"
     overwrite-identity: false
     buffer-size: 100
-    chan-buffer-size: 65535
+    chan-buffer-size: 0
     extended-support: false
     compression: none
 ```

docs/loggers/logger_elasticsearch.md

@@ -26,8 +26,8 @@ Options:
   > Specifies the compression algorithm to use.
 
 * `chan-buffer-size` (integer)
-  > Channel buffer size used on incoming dns message, number of messages before to drop it.
-  > Adjust the size of the channel buffer. If you encounter the error message buffer is full, xxx packet(s) dropped, consider increasing this parameter to prevent message drops.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 * `flush-interval` (integer)
   > Interval in seconds before to flush the buffer.
@@ -40,7 +40,7 @@ Defaults:
   elasticsearch:
     server: "http://127.0.0.1:9200/"
     index:  "dnscollector"
-    chan-buffer-size: 4096
+    chan-buffer-size: 0
     bulk-size: 1048576 # 1MB
     flush-interval: 10 # in seconds
     compression: none

docs/loggers/logger_falco.md

@@ -8,12 +8,13 @@ Options:
   > Falco Plugin endpoint url "http://127.0.0.1:9200"
 
 * `chan-buffer-size` (integer)
-  > channel buffer size used on incoming dns message, number of messages before to drop it.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 Default values:
 
 ```yaml
 falco:
   url: "http://127.0.0.1:9200/events"
-  chan-buffer-size: 65535
+  chan-buffer-size: 0
 ```

docs/loggers/logger_file.md

@@ -52,7 +52,8 @@ Options:
   > delete file on script success
 
 * `chan-buffer-size` (integer)
-  > channel buffer size used on incoming dns message, number of messages before to drop it.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 Default values:
 
@@ -69,7 +70,7 @@ logfile:
   text-format: ""
   postrotate-command: null
   postrotate-delete-success: false
-  chan-buffer-size: 65535
+  chan-buffer-size: 0
 ```
 
 The `postrotate-command` can be used to execute a script after each file rotation.

docs/loggers/logger_fluentd.md

@@ -45,7 +45,8 @@ Options:
   > This is a required parameter if TLS support is enabled.
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 Defaults:
 
@@ -64,5 +65,5 @@ fluentd:
   ca-file: ""
   cert-file: ""
   key-file: ""
-  chan-buffer-size: 4096
+  chan-buffer-size: 0
 ```

docs/loggers/logger_influxdb.md

@@ -36,7 +36,8 @@ Options:
   > Specifies the path to the key file corresponding to the certificate file. This is a required parameter if TLS support is enabled.
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 Default values:
 
@@ -52,5 +53,5 @@ influxdb:
   ca-file: ""
   cert-file: ""
   key-file: ""
-  chan-buffer-size: 65535
+  chan-buffer-size: 0
 ```

docs/loggers/logger_kafka.md

@@ -69,7 +69,8 @@ Options:
   > Specifies the Kafka partition to which messages will be sent.
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 * `compression` (string)
   > Specifies the compression algorithm to use for Kafka messages.
@@ -95,6 +96,6 @@ kafkaproducer:
   buffer-size: 100
   topic: "dnscollector"
   partition: 0
-  chan-buffer-size: 4096
+  chan-buffer-size: 0
   compression: none
 ```

docs/loggers/logger_loki.md

@@ -44,7 +44,8 @@ Options:
   > Specifies the path to the key file corresponding to the certificate file. This is a required parameter if TLS support is enabled.
 
 * `chan-buffer-size` (int)
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+  > Specifies the maximum number of packets that can be buffered before discard additional packets.
+  > Set to zero to use the default global value.
 
 * `basic-auth-login` (string)
   > basic auth login
@@ -83,7 +84,7 @@ lokiclient:
   basic-auth-pwd-file: ""
   tenant-id: ""
   relabel-configs: []
-  chan-buffer-size: 65535
+  chan-buffer-size: 0
 ```
 
 ## Grafana dashboard with Loki datasource