Skip to content

Commit

Permalink
doc : add debugging in cheatsheets (#179)
Browse files Browse the repository at this point in the history
  • Loading branch information
@buixor
buixor authored Aug 5, 2020
1 parent 7470652 commit a3b6388
Show file tree
Hide file tree
Showing 3 changed files with 112 additions and 4 deletions.
108 changes: 108 additions & 0 deletions docs/cheat_sheets/debugging_configs.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@



# Debugging Scenarios and Parsers

## General Advice

When trying to debug a parser or a scenario :

- Work on "cold logs" (with the `-file` and `-type` options) rather than live ones
- Use the `/etc/crowdsec/config/user.yaml` configuration files to have logs on stdout

## Example

```bash
crowdsec -c /etc/crowdsec/config/user.yaml -file mylogs.log.gz -type syslog
INFO[05-08-2020 16:15:47] Crowdsec v0.3.0-rc3-7525f11975a0107746213862dc41c69e00122ac7
INFO[05-08-2020 16:15:47] Loading grok library
...
WARN[05-08-2020 16:16:12] 182.x.x.x triggered a 4h0m0s ip ban remediation for [crowdsecurity/http-probing] bucket_id=misty-moon event_time="2019-01-01 22:58:32 +0100 CET" scenario=crowdsecurity/http-probing source_ip=182.x.x.x
...
```

- `/etc/crowdsec/config/user.yaml` disables demonization and push logs to stdout/stderr
- `-type` must respect expected log type (ie. `nginx` `syslog` etc.)
- `-file` must point to a flat file or a gzip file

When processing logs like this, {{crowdsec.name}} runs in "time machine" mode, and relies on the timestamps *in* the logs to evaluate scenarios. You will most likely need the `crowdsecurity/dateparse-enrich` parser for this.


## Using debug

Both scenarios and parsers support a `debug: true|false` option which produce useful debug.

<details>
<summary>Debug parsing output (expand)</summary>
```bash
DEBU[05-08-2020 15:25:36] eval(evt.Parsed.program == 'nginx') = TRUE id=lively-smoke name=crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] eval variables: id=lively-smoke name=crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] evt.Parsed.program = 'nginx' id=lively-smoke name=crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] Event entering node id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] + Grok 'NGINXACCESS' returned 10 entries to merge in Parsed id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['request'] = '/data.php' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['http_user_agent'] = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['http_referer'] = '-' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['remote_addr'] = '123.x.x.x' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['remote_user'] = '-' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['time_local'] = '01/Jan/2019:01:39:06 +0100' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['method'] = 'POST' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['body_bytes_sent'] = '162' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['http_version'] = '1.1' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Parsed['status'] = '404' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] .Meta[log_type] = 'http_access-log' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] evt.StrTime = '01/Jan/2019:01:39:06 +0100' id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] Event leaving node : ok id=icy-dew name=child-crowdsecurity/nginx-logs stage=s01-parse
DEBU[05-08-2020 15:25:36] child is success, OnSuccess=next_stage, skip id=lively-smoke name=crowdsecurity/nginx-logs stage=s01-parse
```
</details>


<details>
<summary>Debug scenario output (expand)</summary>
```bash
DEBU[05-08-2020 16:02:26] eval(evt.Meta.service == 'http' && evt.Meta.http_status in ['404', '403', '400'] && evt.Parsed.static_ressource == 'false') = TRUE cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
DEBU[05-08-2020 16:02:26] eval variables: cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
DEBU[05-08-2020 16:02:26] evt.Meta.service = 'http' cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
DEBU[05-08-2020 16:02:26] evt.Meta.http_status = '404' cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
DEBU[05-08-2020 16:02:26] evt.Parsed.static_ressource = 'false' cfg=black-wave file=config/scenarios/http-probing.yaml name=crowdsecurity/http-probing
```
</details>


# Test environments

From a [{{crowdsec.name}} release archive]({{crowdsec.download_url}}), you can deploy a test (non-root) environment that is very suitable to write/debug/test parsers and scenarios. Environment is deployed using `./test_env.sh` script from tgz directory, and creates a test environment in `./tests` :

```bash
$ cd crowdsec-v0.3.0/
$ ./test_env.sh
...
[08/05/2020:04:19:18 PM][INFO] Setting up configurations
INFO[0000] Wrote new 75065 bytes index to config/crowdsec-cli/.index.json
INFO[0000] crowdsecurity/syslog-logs : OK
INFO[0000] crowdsecurity/geoip-enrich : OK
...
INFO[0007] Enabled collections : crowdsecurity/linux
INFO[0007] Enabled crowdsecurity/linux
[08/05/2020:04:19:26 PM][INFO] Environment is ready in /home/bui/github/crowdsec/crowdsec/crowdsec-v0.3.0/tests
$ cd tests
$ ./cscli -c dev.yaml list
...
INFO[0000] PARSERS:
-------------------------------------------------------------------------------------------------------
NAME 📦 STATUS VERSION LOCAL PATH
-------------------------------------------------------------------------------------------------------
crowdsecurity/geoip-enrich ✔️ enabled 0.2 config/parsers/s02-enrich/geoip-enrich.yaml
crowdsecurity/syslog-logs ✔️ enabled 0.3 config/parsers/s00-raw/syslog-logs.yaml
crowdsecurity/sshd-logs ✔️ enabled 0.2 config/parsers/s01-parse/sshd-logs.yaml
crowdsecurity/dateparse-enrich ✔️ enabled 0.1 config/parsers/s02-enrich/dateparse-enrich.yaml
-------------------------------------------------------------------------------------------------------
...
$ ./crowdsec -c dev.yaml -file sshd.log -type syslog
INFO[05-08-2020 16:23:32] Crowdsec v0.3.0-rc3-7525f11975a0107746213862dc41c69e00122ac7
INFO[05-08-2020 16:23:32] Loading grok library
...
```


7 changes: 3 additions & 4 deletions docs/write_configurations/parsers.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,9 @@
Please ensure that you have working env or setup test environment before writing your parser.

!!! warning "Parser dependency"

The crowdsecurity/syslog-logs parsers is needed by the core parsing
engine. Deletion or modification of this could result of {{crowdsec.name}}
being unable to parse logs, so this should be done very carefully.
The crowdsecurity/syslog-logs parsers is needed by the core parsing
engine. Deletion or modification of this could result of {{crowdsec.name}}
being unable to parse logs, so this should be done very carefully.

> In the current example, we'll write a parser for the logs produced by `iptables` (netfilter) with the `-j LOG` target.
> This document aims at detailing the process of writing and testing new parsers.
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ nav:
- Ban Management: cheat_sheets/ban-mgmt.md
- Configuration Management: cheat_sheets/config-mgmt.md
- Hub's taxonomy: https://hub.crowdsec.net/fields
- Debugging Parsers & Scenarios: cheat_sheets/debugging_configs.md
- Observability:
- Overview: observability/overview.md
- Logs: observability/logs.md
Expand Down

0 comments on commit a3b6388

Please sign in to comment.