From 6e783977b498337458eaeaac708cd4a92b7711af Mon Sep 17 00:00:00 2001 From: Denis Machard <5562930+dmachard@users.noreply.github.com> Date: Tue, 13 Feb 2024 20:20:55 +0100 Subject: [PATCH] chore: update makefile and docs (#604) * chore: update makefile * update docs --- Makefile | 4 ++- README.md | 2 ++ docs/collectors.md | 3 +- docs/collectors/collector_afpacket.md | 17 ++++-------- docs/collectors/collector_dnsmessage.md | 5 ++++ docs/collectors/collector_fileingestor.md | 26 +++++++---------- docs/collectors/collector_tail.md | 23 ++++++--------- docs/collectors/collector_tzsp.md | 15 +++------- docs/collectors/collector_xdp.md | 14 +++------- docs/running_mode.md | 34 +++++++++++++++-------- docs/transformers.md | 1 + docs/transformers/transform_atags.md | 3 ++ 12 files changed, 69 insertions(+), 78 deletions(-) create mode 100644 docs/collectors/collector_dnsmessage.md create mode 100644 docs/transformers/transform_atags.md diff --git a/Makefile b/Makefile index 93be578b..bbbf9c11 100644 --- a/Makefile +++ b/Makefile @@ -3,10 +3,11 @@ BINARY_NAME := go-dnscollector GO_VERSION := $(shell go env GOVERSION | sed -n 's/go\([0-9]\+\.[0-9]\+\).*/\1/p') GO_LOGGER := 0.4.0 -GO_POWERDNS_PROTOBUF := 1.0.0 +GO_POWERDNS_PROTOBUF := 1.1.0 GO_DNSTAP_PROTOBUF := 1.0.0 GO_FRAMESTREAM := 0.7.0 GO_CLIENTSYSLOG := 0.3.0 +GO_TOPMAP := 1.0.0 BUILD_TIME := $(shell LANG=en_US date +"%F_%T_%z") COMMIT := $(shell git rev-parse --short HEAD) @@ -45,6 +46,7 @@ dep: check-go @go get github.com/dmachard/go-dnstap-protobuf@v$(GO_DNSTAP_PROTOBUF) @go get github.com/dmachard/go-framestream@v$(GO_FRAMESTREAM) @go get github.com/dmachard/go-clientsyslog@v$(GO_CLIENTSYSLOG) + @go get github.com/dmachard/go-topmap@v$(GO_TOPMAP) @go mod edit -go=$(GO_VERSION) @go mod tidy diff --git a/README.md b/README.md index 273ecf4c..90eeab54 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,7 @@ - *Listen for logging traffic with streaming network protocols* - [`DNStap`](docs/collectors/collector_dnstap.md#dns-tap) with `tls`|`tcp`|`unix` transports support and [`proxifier`](docs/collectors/collector_dnstap.md#dns-tap-proxifier) - [`PowerDNS`](docs/collectors/collector_powerdns.md) streams with full support + - [`DNSMessage`](docs/collectors/collector_dnsmessage.md) for internal DNS data structure - [`TZSP`](docs/collectors/collector_tzsp.md) protocol support - *Live capture on a network interface* - [`AF_PACKET`](docs/collectors/collector_afpacket.md) socket with BPF filter @@ -69,6 +70,7 @@ - **[Transformers](./docs/transformers.md)** + - Add additionnal [Tags](docs/transformers/transform_atags.md) - Traffic [Filtering](docs/transformers/transform_trafficfiltering.md) and [Reducer](docs/transformers/transform_trafficreducer.md) - Latency [Computing](docs/transformers/transform_latency.md) - Apply [User Privacy](docs/transformers/transform_userprivacy.md) diff --git a/docs/collectors.md b/docs/collectors.md index 312c89e2..bc73e945 100644 --- a/docs/collectors.md +++ b/docs/collectors.md @@ -7,4 +7,5 @@ | [Tail](collectors/collector_tail.md) | Tail on plain text file | | [XDP Sniffer](collectors/collector_xdp.md) | Live capture on network interface with XDP | | [AF_PACKET Sniffer](collectors/collector_afpacket.md) | Live capture on network interface with AF_PACKET socket | -| [File Ingestor](collectors/collector_fileingestor.md) | File ingestor like pcap | +| [File Ingestor](collectors/collector_fileingestor.md) | File ingestor like pcap | +| [DNS Message](collectors/collector_dnsmessage.md) | DNS Message like pcap | diff --git a/docs/collectors/collector_afpacket.md b/docs/collectors/collector_afpacket.md index d4fe7c89..e9a4cd2a 100644 --- a/docs/collectors/collector_afpacket.md +++ b/docs/collectors/collector_afpacket.md @@ -15,15 +15,8 @@ sudo setcap cap_net_admin,cap_net_raw=eip go-dnscollector Options: -* `port`: (integer) filter on source and destination port -* `device`: (string) if "" bind on all interfaces -* `chan-buffer-size`: (integer) channel buffer size used on incoming packet, number of packet before to drop it. - -Default values: - -```yaml -afpacket-sniffer: - port: 53 - device: wlp2s0 - chan-buffer-size: 65535 -``` +* `port` (int) filter on source and destination port. Defaults to `53`. +* `device` (str) interface name to sniff. Defaults to `wlp2s0`. + > if value is empty, bind on all interfaces. +* `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`. + > Specifies the maximum number of packets that can be buffered before dropping additional packets. diff --git a/docs/collectors/collector_dnsmessage.md b/docs/collectors/collector_dnsmessage.md new file mode 100644 index 00000000..7073d087 --- /dev/null +++ b/docs/collectors/collector_dnsmessage.md @@ -0,0 +1,5 @@ +# Collector: DNSMessage + +> Only available with pipelines! + +Collector to handle internal DNS data structure. diff --git a/docs/collectors/collector_fileingestor.md b/docs/collectors/collector_fileingestor.md index 867ea2ea..c31dd146 100644 --- a/docs/collectors/collector_fileingestor.md +++ b/docs/collectors/collector_fileingestor.md @@ -14,19 +14,13 @@ For config examples, take a look to the following links: Options: -- `watch-dir`: (string) directory to watch for pcap files ingest -- `watch-mode`: (string) watch the directory pcap file with *.pcap extension or dnstap stream with*.fstrm extension, pcap or dnstap expected -- `pcap-dns-port`: (integer) dns source or destination port -- `delete-after:`: (boolean) delete pcap file after ingest -- `chan-buffer-size`: (integer) channel buffer size used on incoming packet, number of packet before to drop it. - -Default values: - -```yaml -file-ingestor: - watch-dir: /tmp - watch-mode: pcap - pcap-dns-port: 53 - delete-after: false - chan-buffer-size: 65535 -``` +- `watch-dir` (str) directory to watch for pcap files ingest. Defaults to `/tmp`. + > Specifies the directory where pcap files are monitored for ingestion. +- `watch-mode` (str) watch the directory pcap or dnstap file. Defaults to `pcap`. + > `*.pcap` extension or dnstap stream with `*.fstrm` extension are expected. +- `pcap-dns-port` (int) dns source or destination port. Defaults port to `53`. + > Expects a port number use for DNS communication. +- `delete-after:` (boolean) delete pcap file after ingest. Default to `false`. + > Determines whether the pcap file should be deleted after ingestion. +- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`. + > Specifies the maximum number of packets that can be buffered before dropping additional packets. diff --git a/docs/collectors/collector_tail.md b/docs/collectors/collector_tail.md index 8d2f1c78..85917dec 100644 --- a/docs/collectors/collector_tail.md +++ b/docs/collectors/collector_tail.md @@ -10,18 +10,11 @@ Enable the tail by provided the path of the file to follow Options: -* `file-path`: (string) file to follow -* `time-layout`: (string) Use the exact layout numbers described -* `pattern-query`: (string) regexp pattern for queries -* `pattern-reply`: (string) regexp pattern for replies - -Default values: - -```yaml -tail: - file-path: null - time-layout: "2006-01-02T15:04:05.999999999Z07:00" - pattern-query: "^(?P[^ ]*) (?P[^ ]*) (?P.*_QUERY) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*)b (?P[^ ]*) (?P[^ ]*) (?P[^ ]*)$" - pattern-reply: "^(?P[^ ]*) (?P[^ ]*) (?P.*_RESPONSE) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*)b (?P[^ ]*) (?P[^ ]*) (?P[^ ]*)$" -``` - +* `file-path`: (string) file to follow. Defaults to `null`. + > Specifies the path to the file that will be monitored. +* `time-layout`: (string) Use the exact layout numbers. Defaults to `2006-01-02T15:04:05.999999999Z07:00`. + > Specifies the layout format for time representation, following the layout numbers defined in https://golang.org/src/time format.go. +* `pattern-query`: (string) regexp pattern for queries. Defaults to `^(?P[^ ]*) (?P[^ ]*) (?P.*_QUERY) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*)b (?P[^ ]*) (?P[^ ]*) (?P[^ ]*)$`. + > Specifies the regular expression pattern used to match queries. +* `pattern-reply`: (string) regexp pattern for replies. Defaults to `^(?P[^ ]*) (?P[^ ]*) (?P.*_RESPONSE) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*) (?P[^ ]*)b (?P[^ ]*) (?P[^ ]*) (?P[^ ]*)$`. + > Specifies the regular expression pattern used to match replies. diff --git a/docs/collectors/collector_tzsp.md b/docs/collectors/collector_tzsp.md index 8edbba04..3d8c3a92 100644 --- a/docs/collectors/collector_tzsp.md +++ b/docs/collectors/collector_tzsp.md @@ -5,18 +5,11 @@ Its primary purpose is to suppport DNS packet capture from Mikrotik brand device Options: -- `listen-ip`: (string) listen on ip -- `listen-port`: (integer) listening on port -- `chan-buffer-size`: (integer) channel buffer size used on incoming packet, number of packet before to drop it. +- `listen-ip` (str) listen on ip. Defaults to `0.0.0.0`. +- `listen-port` (int) listening on port. Defaults to `10000`. +- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`. + > Specifies the maximum number of packets that can be buffered before dropping additional packets. -Default values: - -```yaml -tzsp: - listen-ip: "0.0.0.0" - listen-port: 10000 - chan-buffer-size: 65535 -``` Example rules for Mikrotik brand devices to send the traffic (only works if routed or the device serves as DNS server). diff --git a/docs/collectors/collector_xdp.md b/docs/collectors/collector_xdp.md index 6fff348d..e335285c 100644 --- a/docs/collectors/collector_xdp.md +++ b/docs/collectors/collector_xdp.md @@ -16,13 +16,7 @@ sudo setcap cap_sys_resource,cap_net_raw,cap_perfmon+ep go-dnscollector Options: -- `device`: (string) -- `chan-buffer-size`: (integer) channel buffer size used on incoming packet, number of packet before to drop it. - -Default values: - -```yaml -xdp-sniffer: - device: wlp2s0 - chan-buffer-size: 65535 -``` +- `device` (str) interface name to sniff. Defaults to `wlp2s0`. + > Interface to use for XDP. +- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it. Default to `65535`. + > Specifies the maximum number of packets that can be buffered before dropping additional packets. diff --git a/docs/running_mode.md b/docs/running_mode.md index df991363..00eab2d1 100644 --- a/docs/running_mode.md +++ b/docs/running_mode.md @@ -1,13 +1,30 @@ # DNS-collector - Running mode +- [Pipelining](#pipelining) - [Multiplexer](#multiplexer) - - [Collectors](#collectors) - - [Loggers](#loggers) - - [Routes](#routes) ## Pipelining -> EXPERIMENTAL +> NOTE: EXPERIMENTAL, NOT YET SUITABLE FOR PRODUCTION + +The `pipelining` mode offers several enhancements: + +- a simplified syntax, +- conditional statement-based log routing, +- flexibility to add metadata. + +With this mode you can create pipeline with supported [collectors](./collectors.md) and [loggers](./loggers.md). + +```yaml +pipelines: + - name: + ..... + routing-policy: + default: [ ] + + - name: + ..... +``` ## Multiplexer @@ -19,8 +36,6 @@ You must defined the list of - `loggers`: list of running outputs - `routes`: routing definition -### Collectors - List of supported [collectors](./collectors.md) ```yaml @@ -30,8 +45,6 @@ multiplexer: ..... ``` -### Loggers - List of supported [loggers](./loggers.md) ```yaml @@ -41,9 +54,7 @@ multiplexer: ... ``` -### Routes - -Then defines the routing to use between all of them according to the name. +Defines the routing to use between all of them according to the name. You can connect one collector to multiple loggers and you can also connect multiple collectors to the same logger. @@ -53,4 +64,3 @@ multiplexer: - from: [ list of collectors by name ] to: [ list of loggers by name ] ``` - diff --git a/docs/transformers.md b/docs/transformers.md index 7da38c1b..d2dc43ca 100644 --- a/docs/transformers.md +++ b/docs/transformers.md @@ -25,3 +25,4 @@ Transformers processing is currently in this order : | [GeoIP metadata](transformers/transform_geoip.md) | Country and City | | [Data Extractor](transformers/transform_dataextractor.md) | Add base64 encoded dns payload | | [Traffic Prediction](transformers/transform_trafficprediction.md) | Features to train machine learning models | +| [Additionnal Tags](transformers/transform_atags.md) | Add additionnal tags | diff --git a/docs/transformers/transform_atags.md b/docs/transformers/transform_atags.md new file mode 100644 index 00000000..40256365 --- /dev/null +++ b/docs/transformers/transform_atags.md @@ -0,0 +1,3 @@ +# Transformer: ATags + +Use this transformer to add additional flag in your DNS logs.