fix docs (#650)

* fix docs
* update docs

README.md

@@ -2,7 +2,7 @@
 <img src="https://goreportcard.com/badge/github.com/dmachard/go-dns-collector" alt="Go Report"/>
 <img src="https://img.shields.io/badge/go%20version-min%201.20-green" alt="Go version"/>
 <img src="https://img.shields.io/badge/go%20tests-429-green" alt="Go tests"/>
-<img src="https://img.shields.io/badge/go%20bench-18-green" alt="Go bench"/>
+<img src="https://img.shields.io/badge/go%20bench-19-green" alt="Go bench"/>
 <img src="https://img.shields.io/badge/go%20lines-38661-green" alt="Go lines"/>
 </p>
 
@@ -22,6 +22,7 @@
 > - DNS parser with [Extension Mechanisms for DNS (EDNS)](https://github.com/dmachard/go-dns-collector/blob/main/docs/dnsparser.md) support
 > - IPv4/v6 defragmentation and TCP reassembly
 > - Nanoseconds in timestamps
+> - [Extended DNStap](https://github.com/dmachard/go-dns-collector/blob/main/docs/extended_dnstap.md)
 
 *NOTE: The code before version 1.x is considered beta quality and is subject to breaking changes.*
 

docs/_integration/elasticsearch/README.md

@@ -1,8 +1,6 @@
 
 # DNS-collector with Elastic and Kibana
 
-- Copy this [folder](./docs/_integration/elasticsearch).
-
 - Create the `data` folder.
 
 - Start the docker stack:

docs/_integration/fluentd/README.md

@@ -1,8 +1,6 @@
 
 # DNS-collector with Fluentd
 
-- Copy this [folder](./docs/_integration/fluentd)
-
 - Create the `data` folder.
 
 - Start the docker stack:

docs/_integration/kafka/README.md

@@ -1,8 +1,6 @@
 
 # DNS-collector with Kafka
 
-- Copy this [folder](./docs/_integration/kafka).
-
 - Create the `data` folder.
 
 - Start the docker stack:

docs/collectors/collector_afpacket.md

@@ -15,8 +15,18 @@ sudo setcap cap_net_admin,cap_net_raw=eip go-dnscollector
 
 Options:
 
-* `port` (int) filter on source and destination port. Defaults to `53`.
-* `device` (str) interface name to sniff. Defaults to `wlp2s0`.
+* `port` (int) filter on source and destination port.
+* `device` (str) interface name to sniff.
   > if value is empty, bind on all interfaces.
-* `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+* `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+
+Defaults:
+
+```yaml
+- name: sniffer
+  afpacket-sniffer:
+    port: 53
+    device: wlp2s0
+    chan-buffer-size: 65535
+```

docs/collectors/collector_dnstap.md

@@ -7,33 +7,52 @@ The traffic can be a tcp or unix DNStap stream. TLS is also supported.
 
 Options:
 
-- `listen-ip` (str) local address to bind to. Defaults to `0.0.0.0`.
+- `listen-ip` (str) local address to bind to.
   > Set the local address that the server will bind to. If not provided, the server will bind to all available network interfaces (0.0.0.0).
-- `listen-port` (int) local port to bind to. Defaults to `6000`.
+- `listen-port` (int) local port to bind to.
   > Set the local port that the server will listen on. If not provided, use the default port.
-- `sock-path` (str) Unix socket path. Default to `null`.
+- `sock-path` (str) Unix socket path.
   > Specify the path for the Unix socket to be created.
-- `tls-support` (bool) set to true to enable TLS. Defaults to `false`.
+- `tls-support` (bool) set to true to enable TLS.
   > Enables or disables TLS (Transport Layer Security) support. If set to true, TLS will be used for secure communication.
-- `tls-min-version` (str) Minimun TLS version to use. Default to `1.2`.
+- `tls-min-version` (str) Minimun TLS version to use.
   > Specifies the minimum TLS version that the server will support.
-- `cert-file` (str) path to a certificate server file to use. Default to `(empty)`.
+- `cert-file` (str) path to a certificate server file to use.
   > Specifies the path to the certificate file to be used for TLS. This is a required parameter if TLS support is enabled.
-- `key-file`(str) path to a key server file to use. Default to `(empty)`.
+- `key-file`(str) path to a key server file to use.
   > Specifies the path to the key file corresponding to the certificate file. This is a required parameter if TLS support is enabled.
-- `sock-rcvbuf` (int) sets the socket receive buffer in bytes SO_RCVBUF. Default to `0`.
+- `sock-rcvbuf` (int) sets the socket receive buffer in bytes SO_RCVBUF.
   > This advanced parameter allows fine-tuning of network performance by adjusting the amount of data the socket can receive before signaling to the sender to slow down.
   > Set to zero to use the default system value.
-- `reset-conn` (bool) reset TCP connection on exit. Default to `true`.
+- `reset-conn` (bool) reset TCP connection on exit.
   > Set whether to send a TCP Reset to force the cleanup of the connection on the remote side when the server exits.
-- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
-- `disable-dnsparser"` (bool) disable the minimalist DNS parser. Defaults to `false`.
+- `disable-dnsparser"` (bool) disable the minimalist DNS parser.
   > Some JSON keys should not be available, such as `dns.id`, `dns.flags`, ...
-- `extended-support` (bool) decode the extended extra field sent by DNScollector. Defaults to `false`.
+- `extended-support` (bool) decode the extended extra field sent by DNScollector. 
   > If this setting is enabled, DNScollector will expect receiving the specific [protobuf structure](./../../dnsutils/extended_dnstap.proto) in the extra field, which must be sent by another DNS collector.
   > This field will contain additional metadata generated by various transformations such as filtering, ATags, and others.
 
+Defaults:
+
+```yaml
+- name: dnstap
+  dnstap:
+    listen-ip: 0.0.0.0
+    listen-port: 6000
+    sock-path: null
+    tls-support: false
+    tls-min-version: 1.2
+    cert-file: ""
+    key-file: ""
+    sock-rcvbuf: 0
+    reset-conn: true
+    chan-buffer-size: 65535
+    disable-dnsparser: true
+    extended-support: false
+```
+
 ## DNS tap Proxifier
 
 Collector that receives DNSTAP traffic and relays it without decoding or transformations.
@@ -45,17 +64,31 @@ For config examples, take a look to the following [one](../_examples/use-case-12
 
 Options:
 
-- `listen-ip` (str) local address to bind to. Defaults to `0.0.0.0`.
+- `listen-ip` (str) local address to bind to.
   > Set the local address that the server will bind to. If not provided, the server will bind to all available network interfaces (0.0.0.0).
-- `listen-port` (int) local port to bind to. Defaults to `6000`.
+- `listen-port` (int) local port to bind to.
   > Set the local port that the server will listen on. If not provided, use the default port.
-- `sock-path` (str) Unix socket path. Default to `null`.
+- `sock-path` (str) Unix socket path.
   > Specify the path for the Unix socket to be created.
-- `tls-support` (bool) set to true to enable TLS. Defaults to `false`.
+- `tls-support` (bool) set to true to enable TLS.
   > Enables or disables TLS (Transport Layer Security) support. If set to true, TLS will be used for secure communication.
-- `tls-min-version` (str) Minimun TLS version to use. Default to `1.2`.
+- `tls-min-version` (str) Minimun TLS version to use.
   > Specifies the minimum TLS version that the server will support.
-- `cert-file` (str) path to a certificate server file to use. Default to `(empty)`.
+- `cert-file` (str) path to a certificate server file to use.
   > Specifies the path to the certificate file to be used for TLS. This is a required parameter if TLS support is enabled.
-- `key-file`(str) path to a key server file to use. Default to `(empty)`.
+- `key-file`(str) path to a key server file to use.
   > Specifies the path to the key file corresponding to the certificate file. This is a required parameter if TLS support is enabled.
+
+Defaults
+
+```yaml
+- name: relay
+  dnstap-relay:
+    listen-ip: 0.0.0.0
+    listen-port: 6000
+    sock-path: null
+    tls-support: false
+    tls-min-version: 1.2
+    cert-file: ""
+    key-file: ""
+```

docs/collectors/collector_fileingestor.md

@@ -14,13 +14,25 @@ For config examples, take a look to the following links:
 
 Options:
 
-- `watch-dir` (str) directory to watch for pcap files ingest. Defaults to `/tmp`.
+- `watch-dir` (str) directory to watch for pcap files ingest.
   > Specifies the directory where pcap files are monitored for ingestion.
-- `watch-mode` (str) watch the directory pcap or dnstap file. Defaults to `pcap`.
+- `watch-mode` (str) watch the directory pcap or dnstap file.
   >  `*.pcap` extension or dnstap stream with `*.fstrm` extension are expected.
-- `pcap-dns-port` (int) dns source or destination port. Defaults port to `53`.
+- `pcap-dns-port` (int) dns source or destination port.
   > Expects a port number use for DNS communication.
-- `delete-after:` (boolean) delete pcap file after ingest. Default to `false`.
+- `delete-after:` (boolean) delete pcap file after ingest.
   > Determines whether the pcap file should be deleted after ingestion.
-- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+
+Defaults:
+
+```yaml
+- name: ingest
+  file-ingestor:
+    watch-dir: /tmp
+    watch-mode: pcap
+    pcap-dns-port: 53
+    delete-after: false
+    chan-buffer-size: 65535
+```

docs/collectors/collector_powerdns.md

@@ -4,28 +4,44 @@ Collector to logging protobuf streams from PowerDNS servers. The DNS-collector h
 
 Settings:
 
-- `listen-ip` (str) local address to bind to. Defaults to `0.0.0.0`.
+- `listen-ip` (str) local address to bind to.
   > Set the local address that the server will bind to. If not provided, the server will bind to all available network interfaces (0.0.0.0).
-- `listen-port` (int) local port to bind to. Defaults to `6001`.
+- `listen-port` (int) local port to bind to.
   > Set the local port that the server will listen on. If not provided, use the default port.
-- `tls-support` (bool) set to true to enable TLS. Defaults to `false`.
+- `tls-support` (bool) set to true to enable TLS.
   > Enables or disables TLS (Transport Layer Security) support. If set to true, TLS will be used for secure communication.
-- `tls-min-version` (str) Minimun TLS version to use. Default to `1.2`.
+- `tls-min-version` (str) Minimun TLS version to use.
   > Specifies the minimum TLS version that the server will support.
-- `cert-file` (str) path to a certificate server file to use. Default to `(empty)`.
+- `cert-file` (str) path to a certificate server file to use.
   > Specifies the path to the certificate file to be used for TLS. This is a required parameter if TLS support is enabled.
-- `key-file`(str) path to a key server file to use. Default to `(empty)`.
+- `key-file`(str) path to a key server file to use.
   > Specifies the path to the key file corresponding to the certificate file. This is a required parameter if TLS support is enabled.
-- `sock-rcvbuf` (int) sets the socket receive buffer in bytes SO_RCVBUF. Default to `0`.
+- `sock-rcvbuf` (int) sets the socket receive buffer in bytes SO_RCVBUF.
   > This advanced parameter allows fine-tuning of network performance by adjusting the amount of data the socket can receive before signaling to the sender to slow down.
   > Set to zero to use the default system value.
-- `reset-conn` (bool) reset TCP connection on exit. Default to `true`.
+- `reset-conn` (bool) reset TCP connection on exit.
   > Set whether to send a TCP Reset to force the cleanup of the connection on the remote side when the server exits.
-- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
-- `add-dns-payload` (bool) generate and add fake DNS payload. Default to `false`.
+- `add-dns-payload` (bool) generate and add fake DNS payload.
   > PowerDNS protobuf message does not contain a DNS payload; use this setting to add a raw DNS payload.
 
+Defaults:
+
+```yaml
+- name: powerdns
+  powerdns:
+    listen-ip: 0.0.0.0
+    listen-port: 6001
+    tls-support: false
+    tls-min-version: 1.2
+    cert-file: ""
+    key-file: ""
+    reset-conn: true
+    chan-buffer-size: 65535
+    add-dns-payload: false
+```
+
 ## Custom text format
 
 If you logs your DNS traffic in basic text format, you can use the specific directives:

docs/collectors/collector_tail.md

@@ -10,11 +10,26 @@ Enable the tail by provided the path of the file to follow
 
 Options:
 
-* `file-path`: (string) file to follow. Defaults to `null`.
+* `file-path`: (string) file to follow.
   > Specifies the path to the file that will be monitored.
-* `time-layout`: (string)  Use the exact layout numbers. Defaults to `2006-01-02T15:04:05.999999999Z07:00`.
+* `time-layout`: (string)  Use the exact layout numbers.
   > Specifies the layout format for time representation, following the layout numbers defined in https://golang.org/src/time format.go.
-* `pattern-query`: (string) regexp pattern for queries. Defaults to `^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_QUERY) (?P<rcode>[^ ]*) (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$`.
+* `pattern-query`: (string) regexp pattern for queries.
   > Specifies the regular expression pattern used to match queries.
-* `pattern-reply`: (string) regexp pattern for replies. Defaults to `^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_RESPONSE) (?P<rcode>[^ ]*) (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$`.
+* `pattern-reply`: (string) regexp pattern for replies.
   > Specifies the regular expression pattern used to match replies.
+
+Defaults:
+
+```yaml
+- name: tailf
+  tail:
+    file-path: null
+    time-layout: "2006-01-02T15:04:05.999999999Z07:00"
+    pattern-query: "^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_QUERY) (?P<rcode>[^ ]*)
+      (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*)
+      (?P<length>[^ ]*)b (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$"
+    pattern-reply: "^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_RESPONSE) (?P<rcode>[^ ]*)
+      (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b
+      (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$"
+```

docs/collectors/collector_tzsp.md

@@ -5,12 +5,22 @@ Its primary purpose is to suppport DNS packet capture from Mikrotik brand device
 
 Options:
 
-- `listen-ip` (str) listen on ip. Defaults to `0.0.0.0`.
-- `listen-port` (int) listening on port. Defaults to `10000`.
-- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+- `listen-ip` (str) listen on ip.
+- `listen-port` (int) listening on port.
+- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
 
 
+Defaults:
+
+```yaml
+- name: sniffer
+  tzsp:
+    listen-ip: 0.0.0.0
+    listen-port: 10000
+    chan-buffer-size: 65535
+```
+
 Example rules for Mikrotik brand devices to send the traffic (only works if routed or the device serves as DNS server).
 
 ```routeros

docs/collectors/collector_xdp.md

@@ -16,7 +16,16 @@ sudo setcap cap_sys_resource,cap_net_raw,cap_perfmon+ep go-dnscollector
 
 Options:
 
-- `device` (str) interface name to sniff. Defaults to `wlp2s0`.
+- `device` (str) interface name to sniff.
   > Interface to use for XDP.
-- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+
+Defaults:
+
+```yaml
+- name: sniffer
+  xdp-sniffer:
+    device: wlp2s0
+    chan-buffer-size: 65535
+```

docs/extended_dnstap.md

@@ -0,0 +1,22 @@
+# DNS-collector - Extended DNStap
+
+If this feature is enabled, DNScollector will extend the DNStap protocol by incorporating additional metadata added through transformations, such as filtering, geo, ATags.
+These metadata are encoded in the extra field with the following [protobuf structure](./../../dnsutils/extended_dnstap.proto).
+
+This feature can be only used between two `DNS-collector` instance.
+
+How to enable it on the collector side ?
+
+```yaml
+- name: dnstap_collector
+  dnstap:
+   extended-support: true
+```
+
+How to enable it on the sender side ?
+
+```yaml
+- name: dnstap_sender
+  dnstapclient:
+   extended-support: true
+```

docs/loggers.md

@@ -13,7 +13,7 @@
 | [Fluentd](loggers/logger_fluentd.md)            | Send logs to Fluentd server                           |
 | [InfluxDB](loggers/logger_influxdb.md)          | Send logs to InfluxDB server                          |
 | [Loki](loggers/logger_loki.md)                  | Send logs to Loki server                              |
-| [ElasticSearch](loggers/logger_elasticserch.md) | Send logs to Elastic instance                         |
+| [ElasticSearch](loggers/logger_elasticsearch.md) | Send logs to Elastic instance                         |
 | [Scalyr](loggers/logger_scalyr.md)              | Client for the Scalyr/DataSet addEvents API endpoint. |
 | [Redis](loggers/logger_redis.md)                | Redis pub logger                                      |
 | [Kafka](loggers/logger_kafka.md)                | Kafka DNS producer                                    |