Update docs

README.md

@@ -1,17 +1,19 @@
+<p align="center">
+<img src="https://goreportcard.com/badge/github.com/dmachard/go-dns-collector" alt="Go Report"/>
+<img src="https://img.shields.io/badge/go%20version-min%201.20-green" alt="Go version"/>
+<img src="https://img.shields.io/badge/go%20tests-377-green" alt="Go tests"/>
+<img src="https://img.shields.io/badge/go%20lines-36222-green" alt="Go lines"/>
+</p>
 
-![DNS-collector](./docs/dns-collector_logo.png)
-
-[![Go Report Card](https://goreportcard.com/badge/github.com/dmachard/go-dns-collector)](https://goreportcard.com/report/dmachard/go-dns-collector)
-![Go version](https://img.shields.io/badge/go%20version-min%201.20-blue)
-![Go tests](https://img.shields.io/badge/go%20tests-377-green)
-![Go lines](https://img.shields.io/badge/go%20lines-36222-red)
-![Go Tests](https://github.com/dmachard/go-dns-collector/actions/workflows/testing-go.yml/badge.svg)
-![Github Actions](https://github.com/dmachard/go-dns-collector/actions/workflows/testing-dnstap.yml/badge.svg)
-![Github Actions PDNS](https://github.com/dmachard/go-dns-collector/actions/workflows/testing-powerdns.yml/badge.svg)
+<p align="center">
+<img src="docs/dns-collector_logo.png" alt="DNS-collector"/>
+</p>
 
-*NOTE: The code before version 1.x is considered beta quality and is subject to breaking changes.*
+<p align="center">
+<img src="https://img.shields.io/github/v/release/dmachard/go-dnscollector?logo=github&sort=semver" alt="release"/>
+</p>
 
-`DNS-collector` acts as a passive high speed **ingestor, aggregator and distributor** for your DNS logs with usage indicators and security analysis, written in **Golang**. The DNS traffic can be collected and aggregated from simultaneously [sources](./docs/collectors.md) like DNStap streams, network interface or log files and relays it to multiple other [listeners](./docs/loggers.md) with some [transformations](./docs/transformers.md) on it ([traffic filtering](./docs/transformers.md#dns-filtering), [user privacy](./docs/transformers.md#user-privacy), ...).
+`DNS-collector` acts as a passive high speed **ingestor** with **pipelining** support for your DNS logs, written in **Golang**. It allows enhancing your DNS logs by adding metadata, extracting usage patterns, and facilitating security analysis. The DNS traffic can be collected and aggregated from simultaneously [sources](./docs/collectors.md) like DNStap streams, network interface or log files and relays it to multiple other [listeners](./docs/loggers.md) with some [transformations](./docs/transformers.md) on it ([traffic filtering](./docs/transformers.md#dns-filtering), [user privacy](./docs/transformers.md#user-privacy), ...).
 
 > Additionally, DNS-collector also support
 >
@@ -20,17 +22,13 @@
 > - IPv4/v6 defragmentation and TCP reassembly
 > - Nanoseconds in timestamps
 
-Run
-
-<p align="center">
-<img src="docs/_images/terminal.gif" alt="dnscollector"/>
-</p>
+*NOTE: The code before version 1.x is considered beta quality and is subject to breaking changes.*
 
-Multiplexer
+## Features
 
-![overview](./docs/_images/overview.png)
+- **[Pipelining](./docs/running_mode.md)**
 
-## Features
+  [![overview](./docs/_images/overview.png)](./docs/running_mode.md)
 
 - **[Collectors](./docs/collectors.md)**
 
@@ -73,7 +71,7 @@ Multiplexer
 
   - Traffic [Filtering](docs/transformers/transform_trafficfiltering.md) and [Reducer](docs/transformers/transform_trafficreducer.md)
   - Latency [Computing](docs/transformers/transform_latency.md)
-  - Apply user [Privacy](docs/transformers/transform_userprivacy.md)
+  - Apply [User Privacy](docs/transformers/transform_userprivacy.md)
   - [Normalize](docs/transformers/transform_normalize.md) DNS messages
   - Add [Geographical](docs/transformers/transform_geoip.md) metadata
   - Various data [Extractor](docs/transformers/transform_dataextractor.md)
@@ -87,6 +85,8 @@ Download the latest [`release`](https://github.com/dmachard/go-dns-collector/rel
 ./go-dnscollector -config config.yml
 ```
 
+![run](docs/_images/terminal.gif)
+
 If you prefer run it from docker, follow this [guide](./docs/docker.md).
 
 ## Configuration

collectors/file_ingestor.go

@@ -138,7 +138,7 @@ func (c *FileIngestor) ProcessFile(filePath string) {
 			go c.ProcessPcap(filePath)
 		}
 	case pkgconfig.ModeDNSTap:
-		// processs dnstap
+		// process dnstap
 		if filepath.Ext(filePath) == ".fstrm" {
 			c.LogInfo("file ready to process %s", filePath)
 			go c.ProcessDnstap(filePath)
@@ -412,7 +412,7 @@ func (c *FileIngestor) Run() {
 				go c.ProcessPcap(fn)
 			}
 		case pkgconfig.ModeDNSTap:
-			// processs dnstap
+			// process dnstap
 			if filepath.Ext(fn) == ".fstrm" {
 				go c.ProcessDnstap(fn)
 			}

dnsutils/message.go

@@ -178,7 +178,7 @@ type TransformExtracted struct {
 }
 
 type TransformReducer struct {
-	Occurences       int `json:"occurences" msgpack:"occurences"`
+	Occurrences      int `json:"occurrences" msgpack:"occurrences"`
 	CumulativeLength int `json:"cumulative-length" msgpack:"cumulative-length"`
 }
 
@@ -204,7 +204,7 @@ type TransformML struct {
 	ConsecutiveDigits     int     `json:"consecutive-digits" msgpack:"consecutive-digits"`
 	ConsecutiveConsonants int     `json:"consecutive-consonants" msgpack:"consecutive-consonants"`
 	Size                  int     `json:"size" msgpack:"size"`
-	Occurences            int     `json:"occurences" msgpack:"occurences"`
+	Occurrences           int     `json:"occurrences" msgpack:"occurrences"`
 	UncommonQtypes        int     `json:"uncommon-qtypes" msgpack:"uncommon-qtypes"`
 }
 
@@ -421,8 +421,8 @@ func (dm *DNSMessage) handleReducerDirectives(directives []string, s *strings.Bu
 		s.WriteString("-")
 	} else {
 		switch directive := directives[0]; {
-		case directive == "reducer-occurences":
-			s.WriteString(strconv.Itoa(dm.Reducer.Occurences))
+		case directive == "reducer-occurrences":
+			s.WriteString(strconv.Itoa(dm.Reducer.Occurrences))
 		case directive == "reducer-cumulative-length":
 			s.WriteString(strconv.Itoa(dm.Reducer.CumulativeLength))
 		}
@@ -468,8 +468,8 @@ func (dm *DNSMessage) handleMachineLearningDirectives(directives []string, s *st
 			s.WriteString(strconv.Itoa(dm.MachineLearning.ConsecutiveConsonants))
 		case directive == "ml-size":
 			s.WriteString(strconv.Itoa(dm.MachineLearning.Size))
-		case directive == "ml-occurences":
-			s.WriteString(strconv.Itoa(dm.MachineLearning.Occurences))
+		case directive == "ml-occurrences":
+			s.WriteString(strconv.Itoa(dm.MachineLearning.Occurrences))
 		case directive == "ml-uncommon-qtypes":
 			s.WriteString(strconv.Itoa(dm.MachineLearning.UncommonQtypes))
 		}

dnsutils/message_test.go

@@ -156,10 +156,10 @@ func TestDnsMessage_Json_Transforms_Reference(t *testing.T) {
 		},
 		{
 			transform: "reducer",
-			dmRef:     DNSMessage{Reducer: &TransformReducer{Occurences: 10, CumulativeLength: 47}},
+			dmRef:     DNSMessage{Reducer: &TransformReducer{Occurrences: 10, CumulativeLength: 47}},
 			jsonRef: `{
 						"reducer": {
-							"occurences": 10,
+							"occurrences": 10,
 							"cumulative-length": 47
 						}
 					}`,
@@ -589,14 +589,14 @@ func TestDnsMessage_TextFormat_Directives_Reducer(t *testing.T) {
 	}{
 		{
 			name:     "undefined",
-			format:   "reducer-occurences",
+			format:   "reducer-occurrences",
 			dm:       DNSMessage{},
 			expected: "-",
 		},
 		{
 			name:     "default",
-			format:   "reducer-occurences",
-			dm:       DNSMessage{Reducer: &TransformReducer{Occurences: 1}},
+			format:   "reducer-occurrences",
+			dm:       DNSMessage{Reducer: &TransformReducer{Occurrences: 1}},
 			expected: "1",
 		},
 	}

docs/collectors.md

@@ -7,4 +7,4 @@
 | [Tail](collectors/collector_tail.md)                  | Tail on plain text file |
 | [XDP Sniffer](collectors/collector_xdp.md)            | Live capture on network interface with XDP |
 | [AF_PACKET Sniffer](collectors/collector_afpacket.md) | Live capture on network interface with AF_PACKET socket |
-| [File Ingestor](collectors/collector_file.md)         | File ingestor like pcap |
+| [File Ingestor](collectors/collector_fileingestor.md)         | File ingestor like pcap |

docs/configuration.md

@@ -2,17 +2,14 @@
 
 The configuration of DNS-collector is done through one yaml file named [`config.yml`](https://github.com/dmachard/go-dnscollector/blob/main/config.yml). When the DNS-collector starts, it will look for the config.yml from the current working directory.
 
-A typically configuration would have one or more collector to receive DNS traffic, and severals loggers to process the
-incoming traffics. You can take a look to the list of config [`examples`](examples.md).
+A typically configuration in [multiplexer](./docs/running_mode.md) mode would have one or more collector to receive DNS traffic, and severals loggers to process the incoming traffics. You can take a look to the list of config [`examples`](examples.md).
+
+You can find the global settings below
 
 - [Global](#global)
   - [Trace](#trace)
   - [Custom text format](#custom-text-format)
   - [Server identity](#server-identity)
-- [Multiplexer](#multiplexer)
-  - [Collectors](#collectors)
-  - [Loggers](#loggers)
-  - [Routes](#routes)
 
 ## Global
 
@@ -118,48 +115,3 @@ Output example:
 2023-04-08T18:27:29.278929Z unbound FORWARDER_RESPONSE NOERROR 0.0.0.0 20817 IPv4 UDP 54b google.fr A 0.000000
 2023-04-08T18:27:29.279039Z unbound CLIENT_RESPONSE NOERROR 127.0.0.1 39028 IPv4 UDP 54b google.fr A 0.000000
 ```
-
-## Multiplexer
-
-The dns collector can be configured with multiple loggers and collectors at the same time.
-
-You must defined the list of
-
-- `collectors`: list of running inputs
-- `loggers`: list of running outputs
-- `routes`: routing definition
-
-### Collectors
-
-List of supported [collectors](./collectors.md)
-
-```yaml
-multiplexer:
-  collectors: 
-    - name: <collector_name>
-      .....
-```
-
-### Loggers
-
-List of supported [loggers](./loggers.md)
-
-```yaml
-multiplexer:
-  loggers: 
-    - name: <logger_name>
-      ...
-```
-
-### Routes
-
-Then defines the routing to use between all of them according to the name.
-You can connect one collector to multiple loggers and you can also
-connect multiple collectors to the same logger.
-
-```yaml
-multiplexer:
-  routes: ...
-    - from: [ list of collectors by name ]
-      to: [ list of loggers by name ]
-```

docs/loggers/logger_kafka.md

@@ -1,6 +1,6 @@
 # Logger: Kafka Producer
 
-Kafka producer
+Kafka producer, based on [kafka-go](https://github.com/segmentio/kafka-go) library.
 
 Options: