update docs

dmachard · Oct 28, 2024 · 2187ac0 · 2187ac0
1 parent 974857a
commit 2187ac0
Show file tree

Hide file tree

Showing 9 changed files with 133 additions and 134 deletions.
diff --git a/docs/_examples/use-case-28.yml b/docs/_examples/use-case-28.yml
@@ -1,7 +1,4 @@
-# This configuration sets up DNS traffic monitoring through DNStap on port 6000;
-# removes duplicate traffic and log to the console
 
-# If turned on, debug messages are printed in the standard output
 global:
   trace:
     verbose: true

diff --git a/docs/_examples/use-case-29.yml b/docs/_examples/use-case-29.yml
@@ -1,7 +1,4 @@
-# This configuration sets up DNS traffic monitoring through DNStap on port 6000;
-# removes duplicate traffic and log to the console
 
-# If turned on, debug messages are printed in the standard output
 global:
   trace:
     verbose: true

diff --git a/docs/configuration.md b/docs/configuration.md
@@ -1,28 +1,29 @@
 # DNS-collector - Configuration Guide
 
-The configuration of DNS-collector is done through one yaml file named [`config.yml`](https://github.com/dmachard/go-dnscollector/blob/main/config.yml). When the DNS-collector starts, it will look for the config.yml from the current working directory.
+The configuration of DNS-collector is done through one yaml file named [`config.yml`](https://github.com/dmachard/go-dnscollector/blob/main/config.yml). When the DNS-collector starts, it will look for the config.yml from the current working directory. A typical [configuration in pipeline](./running_mode.md) mode includes one or more collectors to receive DNS traffic and several loggers to process the incoming data. 
 
-A typically [configuration in pipeline](./running_mode.md) mode would have one or more collector to receive DNS traffic, and severals loggers to process the incoming traffics.  You can take a look to the list of config [`full examples`](examples.md).
-
-You can find the global settings below
+>  You can explore a list of configuration [`examples`](examples.md).
 
+Below, you will find the global settings:
 - [Trace](#trace)
 - [Server identity](#server-identity)
 - [Pid file](#pid-file)
 - [Telemetry](#telemetry)
 
 ## Trace
 
-Logs can be enable to have more informations like debug, errors messages generated by the application
+Enable logging to capture detailed information, such as debug and error messages generated by the application.
 
-Options:
+### Options:
 
 - `verbose`: (boolean) debug informations, if turned on, log some applications messages
 - `filename`: (string) filename is the file to write logs to.
 - `max-size`: (integer) maximum size in megabytes of the log file it gets rotated
 - `max-backups`: (integer) maximum number of old log files to retain
 - `log-malformed`: (boolean) log malformed packet
 
+### Example Configuration:
+
 ```yaml
 global:
   trace:
@@ -33,7 +34,7 @@ global:
     max-backups: 10
 ```
 
-Example:
+### Sample Log Output:
 
 ```bash
 INFO: 2022/06/25 20:54:18.173239 main - version 0.0.0
@@ -46,30 +47,35 @@ INFO: 2022/06/25 20:54:18.174286 [dtap] dnstap collector - is listening on [::]:
 
 ## Server Identity
 
-Set the server identity name. The hostname will be used if empty
+Set the server identity name. If this field is left empty, the hostname will be used as the identity.
+
+Example Configuration:
 
 ```yaml
 global:
   server-identity: "dns-collector"
 ```
 
-## Worker
+## Worker Settings
 
 The `interval-monitor` in second(s) is used to count every XX second the number of in/out packets.
 
 The `buffer-size` settings enable to adjust the size of the buffer before discard additional packets. If you encounter the warning message buffer is full, xxx packet(s) dropped, consider increasing this parameter to prevent message drops.
 
+Example Configuration:
+
 ```yaml
 global:
   worker:
     interval-monitor: 10
     buffer-size: 8192
 ```
 
-## Pid file
+## PID file
+
+Specify the path for the DNS-Collector PID file. By default, this setting is empty.
 
-Set path to create DNS-collector PID.
-By default, this settings is empty.
+Example Configuration:
 
 ```yaml
 global:
@@ -78,7 +84,9 @@ global:
 
 ## Telemetry
 
-Enable and configure telemetry
+Enable and configure telemetry for monitoring purposes.
+
+Example Configuration:
 
 ```yaml
 global:

diff --git a/docs/dns2jinja.md b/docs/dns2jinja.md
diff --git a/docs/dnsconversions.md b/docs/dnsconversions.md
@@ -2,20 +2,28 @@
 
 ## Overview
 
-The DNScollector can convert the DNS traffic captured from DNStap stream or network streamdn and transform it to 
-several outputs format
-- text format inline
-- JSON advanced structure
-- Flat JSON (key/value)
-- Jinja templating
-- PCAP
-- DNStap
+DNS-Collector can convert captured DNS traffic from DNStap streams or network streams into several output formats:
 
-## Text format inline
+- [DNSMessage](#dns-message)
+- [Generic outputs](#generic-outputs)
+  - [Text Format Inline](#text-format-inline)
+  - [JSON Advanced Structure](#)
+  - [Flat JSON (Key/Value)](#flat-json-format)
+  - [Jinja templating](#jinja-inline)
 
-The text format can be customized with the following directives.
+## DNS Message
 
-Default directives:
+All DNStap messages or captured from network are transformed first to 
+the [DNSMessage](https://pkg.go.dev/github.com/dmachard/go-dnscollector/dnsutils#DNSMessage) structure
+than after this object can be transformed to text, json, etc.. or specific output format (kafka, redis, etc...)
+
+## Generic outputs
+
+### Text format inline
+
+The text format can be customized using the following directives.
+
+**Default directives:**
 
 - `timestamp-rfc3339ns`: timestamp rfc3339 format, with nano support
 - `timestamp-unixms`: unix timestamp with ms support
@@ -64,15 +72,16 @@ Default directives:
 - `tr`: flag when tcp reassembled occured
 - `edns-csubnet`: display client subnet info
 
-The default text format can be send in the global config or invidually on each loggers, it's your choice
-Here the default format.
+The default text format can be set in the global configuration or individually for each logger. Here’s the default format:
+
+**Example Configuration**
 
 ```yaml
 global:
   text-format: "timestamp-rfc3339ns identity qr operation rcode queryip queryport family protocol length-unit qname qtype latency ttl"
 ```
 
-Output example:
+**Output example**
 
 ```bash
 2023-04-08T18:27:29.268465Z unbound CLIENT_QUERY NOERROR 127.0.0.1 39028 IPv4 UDP 50b google.fr A 0.000000
@@ -92,29 +101,29 @@ global:
 ```
 
 
-## Jinja inline
+### Jinja inline
 
+For a more flexible format, you can use the `text-jinja` setting.
+The DNS to Jinja encoding is very powerful but slow, so use it only under specific conditions. Jinja templates are available with:
+  - Console logger
+  - Log file console
 
-If you want a more flexible format, you can use the `text-jinja` setting
-Example to enable output similiar to dig style:
+**Default directives**
 
+All directives are in https://pkg.go.dev/github.com/dmachard/go-dnscollector/dnsutils#DNSMessage
+
+To use the jinja, add the folowwing code in your text format 
+
+```jinja
+{{ dm.DNS.Opcode }}
 ```
-text-jinja: |+
-    ;; Got {% if dm.DNS.Type == "QUERY" %}query{% else %}answer{% endif %} from {{ dm.NetworkInfo.QueryIP }}#{{ dm.NetworkInfo.QueryPort }}:
-    ;; ->>HEADER<<- opcode: {{ dm.DNS.Opcode }}, status: {{ dm.DNS.Rcode }}, id: {{ dm.DNS.ID }}
-    ;; flags: {{ dm.DNS.Flags.QR | yesno:"qr ," }}{{ dm.DNS.Flags.RD | yesno:"rd ," }}{{ dm.DNS.Flags.RA | yesno:"ra ," }}; QUERY: {{ dm.DNS.QdCount }}, ANSWER: {{ dm.DNS.AnCount }}, AUTHORITY: {{ dm.DNS.NsCount }}, ADDITIONAL: {{ dm.DNS.ArCount }}
-    
-    ;; QUESTION SECTION:
-    ;{{ dm.DNS.Qname }}		{{ dm.DNS.Qclass }}	{{ dm.DNS.Qtype }}
-
-    ;; ANSWER SECTION: {% for rr in dm.DNS.DNSRRs.Answers %}
-    {{ rr.Name }}		{{ rr.TTL }} {{ rr.Class }} {{ rr.Rdatatype }} {{ rr.Rdata }}{% endfor %}
-
-    ;; WHEN: {{ dm.DNSTap.Timestamp }}
-    ;; MSG SIZE  rcvd: {{ dm.DNS.Length }}
-```
 
-## JSON encoding
+**Full configuration examples**
+
+* [`Dig style output`](../_examples/use-case-27.yml)
+
+
+### JSON encoding
 
 
 The `DNS-collector` enables the transformation of DNS queries or replies into `JSON` format.
@@ -127,7 +136,7 @@ The default JSON payload parts:
 - `dns`: DNS fields.
 - `edns`: Extended DNS options.
 
-Example:
+**Example JSON Output**
 
 ```json
 {
@@ -210,7 +219,26 @@ Example:
 }
 ```
 
-## Flat JSON format (recommended)
+
+**Extended JSON format**
+
+This JSON message can be extended by collector(s):
+
+- [PowerDNS collector](collectors/collector_powerdns.md)
+
+This JSON message can be also extended by transformer(s):
+
+- [Atags](transformers/transformer_atags.md)
+- [GeoIP](transformers/transformer_geoip.md)
+- [Suspicious traffic detector](transformers/transform_suspiciousdetector.md)
+- [Public suffix](transformers/transform_normalize.md)
+- [Traffic reducer](transformers/transform_trafficreducer.md)
+- [Traffic filtering](transformers/transformer_trafficfiltering.md)
+*
+
+### Flat JSON format
+
+Flat JSON is a simple key-value pair structure that supports customization of the keys in the format.
 
 At times, a single level key-value output in JSON is easier to ingest than multi-level JSON structures.
 Utilizing `flat-json` delivers every output field as its own key/value pair but requires more processing
@@ -278,34 +306,3 @@ Here's a flat JSON output formatted using `jq`:
   "network.tcp-reassembled": false,
 }
 ```
-
-### Extended JSON format
-
-This JSON message can be extended by collector(s):
-
-- [PowerDNS collector](collectors/collector_powerdns.md)
-
-This JSON message can be also extended by transformer(s):
-
-- [Atags](transformers/transformer_atags.md)
-- [GeoIP](transformers/transformer_geoip.md)
-- [Suspicious traffic detector](transformers/transform_suspiciousdetector.md)
-- [Public suffix](transformers/transform_normalize.md)
-- [Traffic reducer](transformers/transform_trafficreducer.md)
-- [Traffic filtering](transformers/transformer_trafficfiltering.md)
-
-## Save to PCAP files
-
-In PCAP mode, DNS traffic is logged in binary form, capturing details over various protocols. The following mappings are used:
-
-| Origin protocol        | Translated to                  |
-| -----------------------|--------------------------------|
-| DNS/53 over UDP        | DNS UDP/53                     |
-| DNS/53 over TCP        | DNS TCP/53                     |
-| DoH/443                | DNS UDP/443 (no cipher)        |
-| DoT/853                | DNS UDP/853 (no cipher)        |
-| DoQ                    | Currently unsupported          |
-
-## Save to DNStap files
-
-todo
diff --git a/docs/examples.md b/docs/examples.md
@@ -10,7 +10,6 @@ You will find below some examples of configurations to manage your DNS logs.
 
 - **Capture DNS traffic from incoming DNSTap streams**
   - [x] [Read from UNIX DNSTap socket and forward it to TLS stream](./_examples/use-case-5.yml)
-  - [x] [Transform DNSTap as input to JSON format as output](./_examples/use-case-3.yml)
   - [x] [Relays DNSTap stream to multiple remote destination without decoding](./_examples/use-case-12.yml)
   - [x] [Aggregate several DNSTap stream and forward it to the same file](./_examples/use-case-7.yml)
   - [x] [Send to syslog TLS](./_examples/use-case-23.yml)
@@ -19,7 +18,8 @@ You will find below some examples of configurations to manage your DNS logs.
   - [x] [Convert to text format output](./_examples/use-case-28.yml)
   - [x] [Convert to CSV output style](./_examples/use-case-30.yml)
   - [x] [Convert to text format with dig style, based on Jinja templating](./_examples/use-case-27.yml)
-  - [x] [Convert to JSON format output](./_examples/use-case-29.yml)
+  - [x] [Transform DNSTap as input to JSON format as output](./_examples/use-case-3.yml)
+  - [x] [Convert to JSON key/value format output](./_examples/use-case-29.yml)
 
 - **Capture DNS traffic from PowerDNS products**
   - [x] [Capture multiple PowerDNS streams](./_examples/use-case-8.yml)

diff --git a/docs/loggers/logger_file.md b/docs/loggers/logger_file.md
@@ -1,10 +1,16 @@
 # Logger: File
 
+- [Configuration Options](#overview)
+- [Log Compression](#log-compression)
+- [Postrotate command](#postrotate-command)
+- [To PCAP](#save-to-pcap-files)
+- [To DNStap](#save-to-dnstap-files)
+
 ## Overview
 
 The File Logger allows you to log DNS traffic to a file in various formats, with support for rotation, compression, custom formatting, and the ability to execute scripts after file rotation.
 
-## Key Features
+**Key Features**
 - **File Rotation**: Automatically rotates log files based on size.
 - **Supported Formats**: Supports multiple output formats - `text`, `jinja`, `json` and `flat json`, `pcap` or `dnstap`
 - **Compression**: Optional gzip compression for rotated log files.
@@ -125,3 +131,21 @@ mv $1 $BACKUP_FOLDER/$3
 ```
 
 > Note: If compression is enabled, the postrotate-command will run only after compression completes.
+
+## Save to PCAP files
+
+You can configure the collector to save traffic in PCAP format. Only available with `logger file`.
+
+In PCAP mode, DNS traffic is logged in binary form, capturing details over various protocols. The following mappings are used:
+
+| Origin protocol        | Translated to                  |
+| -----------------------|--------------------------------|
+| DNS/53 over UDP        | DNS UDP/53                     |
+| DNS/53 over TCP        | DNS TCP/53                     |
+| DoH/443                | DNS UDP/443 (no cipher)        |
+| DoT/853                | DNS UDP/853 (no cipher)        |
+| DoQ                    | Currently unsupported          |
+
+## Save to DNStap files
+
+You can configure the collector to save traffic in DNStap format. Only available with `logger file`.