Add new jinja format settings in console and logfile (#856)

* Update docs * add new jinja format settings in console and logfile * update README.md
dmachard · Oct 28, 2024 · 6ca11dc · 6ca11dc
1 parent b1320b1
commit 6ca11dc
Show file tree

Hide file tree

Showing 16 changed files with 458 additions and 295 deletions.
diff --git a/README.md b/README.md
@@ -19,8 +19,9 @@
 > Additionally, DNS-collector also support
 >
 > - [Extended](https://github.com/dmachard/go-dns-collector/blob/main/docs/extended_dnstap.md) DNStap with TLS encryption, compression, and more metadata capabilities
-> - DNS protocol conversions (to [plain text](https://github.com/dmachard/go-dns-collector/blob/main/docs/configuration.md#custom-text-format), [JSON](https://github.com/dmachard/go-dns-collector/blob/main/docs/dnsjson.md), and more... )
+> - DNS protocol conversions to [plain text, JSON, jinja and more](https://github.com/dmachard/go-dns-collector/blob/main/docs/dnsconversions.md)
 > - DNS parser with [Extension Mechanisms for DNS (EDNS)](https://github.com/dmachard/go-dns-collector/blob/main/docs/dnsparser.md) support
+> - Live capture on a network interface
 > - IPv4/v6 defragmentation and TCP reassembly
 > - Nanoseconds in timestamps
 
@@ -98,22 +99,17 @@ If you prefer run it from docker, follow this [guide](./docs/docker.md).
 
 ## Configuration
 
-The configuration of DNS-collector is done through a file named [`config.yml`](config.yml). When the DNS-collector starts, it will look for the config.yml from the current working directory.
+The configuration of DNS-collector is done through a file named [`config.yml`](config.yml). 
+When the DNS-collector starts, it will look for the config.yml from the current working directory.
+A typical [configuration in pipeline](./running_mode.md) mode includes one or more collectors to receive DNS traffic and several loggers to process the incoming data. 
 
-See the full [configuration guide](./docs/configuration.md) for more details.
+To get started quickly, you can use this default [`config.yml`](config.yml).
+For advanced settings, see the [advanced configuration guide](./docs/advanced_config.md).
 
-Run the DNS-collector in dry mode to verify the configuration.
+You can also see  the [`_examples`](./docs/_examples) folder from documentation witch contains a number of [various configurations](./docs/examples.md) to get you started with the DNS-collector in different ways.
 
-```bash
-./go-dnscollector -config config.yml -test-config
-INFO: 2023/12/24 14:43:29.043730 main - config OK!
-```
-
-## Usage examples
-
-The [`_examples`](./docs/_examples) folder from documentation contains a number of [various configurations](./docs/examples.md) to get you started with the DNS-collector in different ways.
-
-The [`_integration`](./docs/_integration) folder contains DNS-collector `configuration` files and `docker compose` examples for popular tools:
+Additionally, the [`_integration`](./docs/_integration) folder contains preconfigured files and `docker compose` examples
+for integrating DNS-collector with popular tools:
 
 - [Fluentd](./docs/_integration/fluentd/README.md)
 - [Elasticsearch](./docs/_integration/elasticsearch/README.md)

diff --git a/docs/_examples/use-case-27.yml b/docs/_examples/use-case-27.yml
@@ -0,0 +1,33 @@
+global:
+  trace:
+    verbose: true
+
+pipelines:
+  - name: tap
+    dnstap:
+      listen-ip: 0.0.0.0
+      listen-port: 6000
+    transforms:
+      normalize:
+        qname-lowercase: true
+        qname-replace-nonprintable: true
+    routing-policy:
+      forward: [ console ]
+      dropped: [ ]
+
+  - name: console
+    stdout:
+      mode: jinja
+      jinja-format: |+
+          ;; Got {% if dm.DNS.Type == "QUERY" %}query{% else %}answer{% endif %} from {{ dm.NetworkInfo.QueryIP }}#{{ dm.NetworkInfo.QueryPort }}:
+          ;; ->>HEADER<<- opcode: {{ dm.DNS.Opcode }}, status: {{ dm.DNS.Rcode }}, id: {{ dm.DNS.ID }}
+          ;; flags: {{ dm.DNS.Flags.QR | yesno:"qr ," }}{{ dm.DNS.Flags.RD | yesno:"rd ," }}{{ dm.DNS.Flags.RA | yesno:"ra ," }}; QUERY: {{ dm.DNS.QdCount }}, ANSWER: {{ dm.DNS.AnCount }}, AUTHORITY: {{ dm.DNS.NsCount }}, ADDITIONAL: {{ dm.DNS.ArCount }}
+          
+          ;; QUESTION SECTION:
+          ;{{ dm.DNS.Qname }}		{{ dm.DNS.Qclass }}	{{ dm.DNS.Qtype }}
+
+          ;; ANSWER SECTION: {% for rr in dm.DNS.DNSRRs.Answers %}
+          {{ rr.Name }}		{{ rr.TTL }} {{ rr.Class }} {{ rr.Rdatatype }} {{ rr.Rdata }}{% endfor %}
+
+          ;; WHEN: {{ dm.DNSTap.Timestamp }}
+          ;; MSG SIZE  rcvd: {{ dm.DNS.Length }}
diff --git a/docs/_examples/use-case-28.yml b/docs/_examples/use-case-28.yml
@@ -0,0 +1,22 @@
+
+global:
+  trace:
+    verbose: true
+
+pipelines:
+  - name: tap
+    dnstap:
+      listen-ip: 0.0.0.0
+      listen-port: 6000
+    transforms:
+      normalize:
+        qname-lowercase: true
+        qname-replace-nonprintable: true
+    routing-policy:
+      forward: [ console ]
+      dropped: [ ]
+
+  - name: console
+    stdout:
+      mode: text
+      text-format: "timestamp-rfc3339ns identity version operation rcode queryip qname qtype ancount"
diff --git a/docs/_examples/use-case-29.yml b/docs/_examples/use-case-29.yml
@@ -0,0 +1,21 @@
+
+global:
+  trace:
+    verbose: true
+
+pipelines:
+  - name: tap
+    dnstap:
+      listen-ip: 0.0.0.0
+      listen-port: 6000
+    transforms:
+      normalize:
+        qname-lowercase: true
+        qname-replace-nonprintable: true
+    routing-policy:
+      forward: [ console ]
+      dropped: [ ]
+
+  - name: console
+    stdout:
+      mode: flat-json
diff --git a/docs/_examples/use-case-30.yml b/docs/_examples/use-case-30.yml
@@ -0,0 +1,22 @@
+global:
+  trace:
+    verbose: true
+
+pipelines:
+  - name: tap
+    dnstap:
+      listen-ip: 0.0.0.0
+      listen-port: 6000
+    transforms:
+      normalize:
+        qname-lowercase: true
+        qname-replace-nonprintable: true
+    routing-policy:
+      forward: [ console ]
+      dropped: [ ]
+
+  - name: console
+    stdout:
+      mode: text
+      text-format: "timestamp-rfc3339ns identity version operation rcode queryip qname qtype ancount"
+      text-format-delimiter: ";"
diff --git a/docs/advanced_config.md b/docs/advanced_config.md
@@ -0,0 +1,136 @@
+# DNS-collector - Advanced configuration Guide
+
+- [Configuration checks](#configuration-checks)
+- [Global settings](#global-settings)
+  - [Trace](#trace)
+  - [Server identity](#server-identity)
+  - [Pid file](#pid-file)
+  - [Telemetry](#telemetry)
+- [Configuration reloading](#configuration-reloading)
+
+## Configuration checks
+
+DNS-collector also supports a dry-run mode for configuration verification. Run the following command to check your setup:
+
+```bash
+./go-dnscollector -config config.yml -test-config
+INFO: 2023/12/24 14:43:29.043730 main - config OK!
+```
+
+## Global settings
+
+### Trace
+
+Enable logging to capture detailed information, such as debug and error messages generated by the application.
+
+**Options**
+
+- `verbose`: (boolean) debug informations, if turned on, log some applications messages
+- `filename`: (string) filename is the file to write logs to.
+- `max-size`: (integer) maximum size in megabytes of the log file it gets rotated
+- `max-backups`: (integer) maximum number of old log files to retain
+- `log-malformed`: (boolean) log malformed packet
+
+**Example Configuration**
+
+```yaml
+global:
+  trace:
+    verbose: true
+    log-malformed: false
+    filename: ""
+    max-size: 10
+    max-backups: 10
+```
+
+**Sample Log Output**
+
+```bash
+INFO: 2022/06/25 20:54:18.173239 main - version 0.0.0
+INFO: 2022/06/25 20:54:18.173271 main - config loaded...
+INFO: 2022/06/25 20:54:18.173277 main - starting dns-collector...
+....
+INFO: 2022/06/25 20:54:18.174256 [dtap] dnstap collector - running in background...
+INFO: 2022/06/25 20:54:18.174286 [dtap] dnstap collector - is listening on [::]:6000
+```
+
+### Server Identity
+
+Set the server identity name. If this field is left empty, the hostname will be used as the identity.
+
+**Example Configuration**
+
+```yaml
+global:
+  server-identity: "dns-collector"
+```
+
+### Worker Settings
+
+The `interval-monitor` in second(s) is used to count every XX second the number of in/out packets.
+
+The `buffer-size` settings enable to adjust the size of the buffer before discard additional packets. If you encounter the warning message buffer is full, xxx packet(s) dropped, consider increasing this parameter to prevent message drops.
+
+**Example Configuration**
+
+```yaml
+global:
+  worker:
+    interval-monitor: 10
+    buffer-size: 8192
+```
+
+### PID file
+
+Specify the path for the DNS-Collector PID file. By default, this setting is empty.
+
+**Example Configuration**
+
+```yaml
+global:
+  pid-file: "/path/to/your/pidfile.pid"
+```
+
+### Telemetry
+
+Enable and configure telemetry for monitoring purposes.
+
+**Example Configuration**
+
+```yaml
+global:
+  telemetry:
+    enabled: false
+    web-path: "/metrics"
+    web-listen: ":9165"
+    prometheus-prefix: "dnscollector_exporter"
+    tls-support: false
+    tls-cert-file: ""
+    tls-key-file: ""
+    client-ca-file: ""
+    basic-auth-enable: false
+    basic-auth-login: admin
+    basic-auth-pwd: changeme
+```
+
+
+## Configuration reloading
+
+DNS-collector automatically reloads its configuration upon receiving a SIGHUP signal, allowing you to update settings without restarting the service.
+
+To trigger a reload, send the SIGHUP signal:
+
+```
+$ sudo pkill -HUP dnscollector
+```
+
+Expected output:
+
+```
+WARNING: 2024/10/28 18:37:05.046321 main - SIGHUP received
+INFO: 2024/10/28 18:37:05.049529 worker - [tap] dnstap - reload configuration...
+INFO: 2024/10/28 18:37:05.050071 worker - [tofile] file - reload configuration...
+INFO: 2024/10/28 18:37:05.050132 worker - [tofile] file - running in mode: json
+INFO: 2024/10/28 18:37:05.050765 worker - [prom] prometheus - reload configuration...
+INFO: 2024/10/28 18:37:05.051304 worker - [console] stdout - reload configuration...
+```