chore: update makefile and docs (#604)

* chore: update makefile
* update docs

Makefile

@@ -3,10 +3,11 @@ BINARY_NAME := go-dnscollector
 GO_VERSION := $(shell go env GOVERSION | sed -n 's/go\([0-9]\+\.[0-9]\+\).*/\1/p')
 
 GO_LOGGER := 0.4.0
-GO_POWERDNS_PROTOBUF := 1.0.0
+GO_POWERDNS_PROTOBUF := 1.1.0
 GO_DNSTAP_PROTOBUF := 1.0.0
 GO_FRAMESTREAM := 0.7.0
 GO_CLIENTSYSLOG := 0.3.0
+GO_TOPMAP := 1.0.0
 
 BUILD_TIME := $(shell LANG=en_US date +"%F_%T_%z")
 COMMIT := $(shell git rev-parse --short HEAD)
@@ -45,6 +46,7 @@ dep: check-go
 	@go get github.com/dmachard/go-dnstap-protobuf@v$(GO_DNSTAP_PROTOBUF)
 	@go get github.com/dmachard/go-framestream@v$(GO_FRAMESTREAM)
 	@go get github.com/dmachard/go-clientsyslog@v$(GO_CLIENTSYSLOG)
+	@go get github.com/dmachard/go-topmap@v$(GO_TOPMAP)
 	@go mod edit -go=$(GO_VERSION)
 	@go mod tidy
 

README.md

@@ -35,6 +35,7 @@
   - *Listen for logging traffic with streaming network protocols*
     - [`DNStap`](docs/collectors/collector_dnstap.md#dns-tap) with `tls`|`tcp`|`unix` transports support and [`proxifier`](docs/collectors/collector_dnstap.md#dns-tap-proxifier)
     - [`PowerDNS`](docs/collectors/collector_powerdns.md) streams with full  support
+    - [`DNSMessage`](docs/collectors/collector_dnsmessage.md) for internal DNS data structure
     - [`TZSP`](docs/collectors/collector_tzsp.md) protocol support
   - *Live capture on a network interface*
     - [`AF_PACKET`](docs/collectors/collector_afpacket.md) socket with BPF filter
@@ -69,6 +70,7 @@
 
 - **[Transformers](./docs/transformers.md)**
 
+  - Add additionnal [Tags](docs/transformers/transform_atags.md)
   - Traffic [Filtering](docs/transformers/transform_trafficfiltering.md) and [Reducer](docs/transformers/transform_trafficreducer.md)
   - Latency [Computing](docs/transformers/transform_latency.md)
   - Apply [User Privacy](docs/transformers/transform_userprivacy.md)

docs/collectors.md

@@ -7,4 +7,5 @@
 | [Tail](collectors/collector_tail.md)                  | Tail on plain text file |
 | [XDP Sniffer](collectors/collector_xdp.md)            | Live capture on network interface with XDP |
 | [AF_PACKET Sniffer](collectors/collector_afpacket.md) | Live capture on network interface with AF_PACKET socket |
-| [File Ingestor](collectors/collector_fileingestor.md)         | File ingestor like pcap |
+| [File Ingestor](collectors/collector_fileingestor.md) | File ingestor like pcap |
+| [DNS Message](collectors/collector_dnsmessage.md)     | DNS Message like pcap |

docs/collectors/collector_afpacket.md

@@ -15,15 +15,8 @@ sudo setcap cap_net_admin,cap_net_raw=eip go-dnscollector
 
 Options:
 
-* `port`: (integer) filter on source and destination port
-* `device`: (string) if "" bind on all interfaces
-* `chan-buffer-size`: (integer) channel buffer size used on incoming packet, number of packet before to drop it.
-
-Default values:
-
-```yaml
-afpacket-sniffer:
-  port: 53
-  device: wlp2s0
-  chan-buffer-size: 65535
-```
+* `port` (int) filter on source and destination port. Defaults to `53`.
+* `device` (str) interface name to sniff. Defaults to `wlp2s0`.
+  > if value is empty, bind on all interfaces.
+* `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+  > Specifies the maximum number of packets that can be buffered before dropping additional packets.

docs/collectors/collector_dnsmessage.md

@@ -0,0 +1,5 @@
+# Collector: DNSMessage
+
+> Only available with pipelines!
+
+Collector to handle internal DNS data structure.

docs/collectors/collector_fileingestor.md

@@ -14,19 +14,13 @@ For config examples, take a look to the following links:
 
 Options:
 
-- `watch-dir`: (string) directory to watch for pcap files ingest
-- `watch-mode`: (string) watch the directory pcap file with *.pcap extension or dnstap stream with*.fstrm extension, pcap or dnstap expected
-- `pcap-dns-port`: (integer) dns source or destination port
-- `delete-after:`: (boolean) delete pcap file after ingest
-- `chan-buffer-size`: (integer) channel buffer size used on incoming packet, number of packet before to drop it.
-
-Default values:
-
-```yaml
-file-ingestor:
-  watch-dir: /tmp
-  watch-mode: pcap
-  pcap-dns-port: 53
-  delete-after: false
-  chan-buffer-size: 65535
-```
+- `watch-dir` (str) directory to watch for pcap files ingest. Defaults to `/tmp`.
+  > Specifies the directory where pcap files are monitored for ingestion.
+- `watch-mode` (str) watch the directory pcap or dnstap file. Defaults to `pcap`.
+  >  `*.pcap` extension or dnstap stream with `*.fstrm` extension are expected.
+- `pcap-dns-port` (int) dns source or destination port. Defaults port to `53`.
+  > Expects a port number use for DNS communication.
+- `delete-after:` (boolean) delete pcap file after ingest. Default to `false`.
+  > Determines whether the pcap file should be deleted after ingestion.
+- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+  > Specifies the maximum number of packets that can be buffered before dropping additional packets.

docs/collectors/collector_tail.md

@@ -10,18 +10,11 @@ Enable the tail by provided the path of the file to follow
 
 Options:
 
-* `file-path`: (string) file to follow
-* `time-layout`: (string)  Use the exact layout numbers described <https://golang.org/src/time/format.go>
-* `pattern-query`: (string) regexp pattern for queries
-* `pattern-reply`: (string) regexp pattern for replies
-
-Default values:
-
-```yaml
-tail:
-  file-path: null
-  time-layout: "2006-01-02T15:04:05.999999999Z07:00"
-  pattern-query: "^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_QUERY) (?P<rcode>[^ ]*) (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$"
-  pattern-reply: "^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_RESPONSE) (?P<rcode>[^ ]*) (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$"
-```
-
+* `file-path`: (string) file to follow. Defaults to `null`.
+  > Specifies the path to the file that will be monitored.
+* `time-layout`: (string)  Use the exact layout numbers. Defaults to `2006-01-02T15:04:05.999999999Z07:00`.
+  > Specifies the layout format for time representation, following the layout numbers defined in https://golang.org/src/time format.go.
+* `pattern-query`: (string) regexp pattern for queries. Defaults to `^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_QUERY) (?P<rcode>[^ ]*) (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$`.
+  > Specifies the regular expression pattern used to match queries.
+* `pattern-reply`: (string) regexp pattern for replies. Defaults to `^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_RESPONSE) (?P<rcode>[^ ]*) (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$`.
+  > Specifies the regular expression pattern used to match replies.

docs/collectors/collector_tzsp.md

@@ -5,18 +5,11 @@ Its primary purpose is to suppport DNS packet capture from Mikrotik brand device
 
 Options:
 
-- `listen-ip`: (string) listen on ip
-- `listen-port`: (integer) listening on port
-- `chan-buffer-size`: (integer) channel buffer size used on incoming packet, number of packet before to drop it.
+- `listen-ip` (str) listen on ip. Defaults to `0.0.0.0`.
+- `listen-port` (int) listening on port. Defaults to `10000`.
+- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
 
-Default values:
-
-```yaml
-tzsp:
-  listen-ip: "0.0.0.0"
-  listen-port: 10000
-  chan-buffer-size: 65535
-```
 
 Example rules for Mikrotik brand devices to send the traffic (only works if routed or the device serves as DNS server).
 

docs/collectors/collector_xdp.md

@@ -16,13 +16,7 @@ sudo setcap cap_sys_resource,cap_net_raw,cap_perfmon+ep go-dnscollector
 
 Options:
 
-- `device`: (string)
-- `chan-buffer-size`: (integer) channel buffer size used on incoming packet, number of packet before to drop it.
-
-Default values:
-
-```yaml
-xdp-sniffer:
-  device: wlp2s0
-  chan-buffer-size: 65535
-```
+- `device` (str) interface name to sniff. Defaults to `wlp2s0`.
+  > Interface to use for XDP.
+- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`.
+  > Specifies the maximum number of packets that can be buffered before dropping additional packets.

docs/running_mode.md

@@ -1,13 +1,30 @@
 # DNS-collector - Running mode
 
+- [Pipelining](#pipelining)
 - [Multiplexer](#multiplexer)
-  - [Collectors](#collectors)
-  - [Loggers](#loggers)
-  - [Routes](#routes)
 
 ## Pipelining
 
-> EXPERIMENTAL
+> NOTE: EXPERIMENTAL, NOT YET SUITABLE FOR PRODUCTION
+
+The `pipelining` mode offers several enhancements:
+
+- a simplified syntax,
+- conditional statement-based log routing,
+- flexibility to add metadata.
+
+With this mode you can create pipeline with supported [collectors](./collectors.md) and [loggers](./loggers.md).
+
+```yaml
+pipelines:
+  - name: <collector1_name>
+    .....
+    routing-policy:
+      default: [ <collector2_name> ]
+
+  - name: <collector2_name>
+    .....
+```
 
 ## Multiplexer
 
@@ -19,8 +36,6 @@ You must defined the list of
 - `loggers`: list of running outputs
 - `routes`: routing definition
 
-### Collectors
-
 List of supported [collectors](./collectors.md)
 
 ```yaml
@@ -30,8 +45,6 @@ multiplexer:
       .....
 ```
 
-### Loggers
-
 List of supported [loggers](./loggers.md)
 
 ```yaml
@@ -41,9 +54,7 @@ multiplexer:
       ...
 ```
 
-### Routes
-
-Then defines the routing to use between all of them according to the name.
+Defines the routing to use between all of them according to the name.
 You can connect one collector to multiple loggers and you can also
 connect multiple collectors to the same logger.
 
@@ -53,4 +64,3 @@ multiplexer:
     - from: [ list of collectors by name ]
       to: [ list of loggers by name ]
 ```
-

docs/transformers.md

@@ -25,3 +25,4 @@ Transformers processing is currently in this order :
 | [GeoIP metadata](transformers/transform_geoip.md)                 | Country and City                         |
 | [Data Extractor](transformers/transform_dataextractor.md)         | Add base64 encoded dns payload                        |
 | [Traffic Prediction](transformers/transform_trafficprediction.md) | Features to train machine learning models              |
+| [Additionnal Tags](transformers/transform_atags.md) | Add additionnal tags |

docs/transformers/transform_atags.md

@@ -0,0 +1,3 @@
+# Transformer: ATags
+
+Use this transformer to add additional flag in your DNS logs.