Skip to content

Commit

Permalink
Fix Elasticsearch GC logs ingestion from Docker logs (elastic#17164)
Browse files Browse the repository at this point in the history
* [Filebeat][elasticsearch] fix JSON lines picked up in gc.logs

When used with docker autodiscover, the elasticsearch gc module
erroneously picks up JSON formatted line from docker's stdout.

Fix by excluding lines that start with `{`.

* Adding test files

* Adding CHANGELOG entry

* Moving test files to correct fileset

* Introduce GC log entry in test fixture + golden file

* Make sure multiline pattern excludes lines starting with {

* Fixing golden file

Co-authored-by: Magnus Kessler <[email protected]>
  • Loading branch information
ycombinator and kesslerm authored Mar 24, 2020
1 parent df1ebf7 commit 389aba7
Show file tree
Hide file tree
Showing 4 changed files with 163 additions and 2 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix Elasticsearch `_id` field set by S3 and Google Pub/Sub inputs. {pull}17026[17026]
- Fixed various Cisco FTD parsing issues. {issue}16863[16863] {pull}16889[16889]
- Fix default index pattern in IBM MQ filebeat dashboard. {pull}17146[17146]
- Fix `elasticsearch.gc` fileset to not collect _all_ logs when Elasticsearch is running in Docker. {issue}13164[13164] {issue}16583[16583] {pull}17164[17164]

*Heartbeat*

Expand Down
4 changes: 2 additions & 2 deletions filebeat/module/elasticsearch/gc/config/gc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
exclude_lines: ["^(OpenJDK|Java HotSpot).* Server VM ", "^CommandLine flags: ", "^Memory: "] # exclude JVM8 banner
exclude_lines: ["^(OpenJDK|Java HotSpot).* Server VM ", "^CommandLine flags: ", "^Memory: ", "^{"] # exclude JVM8 banner and JSON
multiline:
pattern: '^\[?[0-9]{4}-[0-9]{2}-[0-9]{2}'
pattern: '^(\[?[0-9]{4}-[0-9]{2}-[0-9]{2}|{)'
negate: true
match: after
Loading

0 comments on commit 389aba7

Please sign in to comment.