[7.6] Fix Elasticsearch GC logs ingestion from Docker logs (#17164)

[ycombinator]

Backports the following commits to 7.6:

• Fix Elasticsearch GC logs ingestion from Docker logs (Fix Elasticsearch GC logs ingestion from Docker logs #17164)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[blakerouse]

@ycombinator I was under the impression there would not be another 7.6 release.

[ycombinator]

@blakerouse That's right. There isn't a 7.6.3 release planned as of now. But things could always change, e.g. a critical bug somewhere in the stack comes up and demands another patch release before 7.7.0 goes out. So we backport bugfixes to the 7.6 branch, in case a 7.6.3 is planned in the future.

[blakerouse]

Okay, understandable. Looks good then!

[ycombinator]

Failing CI test is totally related.

08:16:32 ======================================================================
08:16:32 FAIL: test_fileset_file_032_elasticsearch (test_modules.Test)
08:16:32 ----------------------------------------------------------------------
08:16:32 Traceback (most recent call last):
08:16:32   File "/go/src/github.com/elastic/beats/filebeat/build/python-env/local/lib/python2.7/site-packages/parameterized/parameterized.py", line 518, in standalone_func
08:16:32     return func(*(a + p.args), **p.kwargs)
08:16:32   File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_modules.py", line 106, in test_fileset_file
08:16:32     cfgfile=cfgfile)
08:16:32   File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_modules.py", line 179, in run_on_file
08:16:32     self._test_expected_events(test_file, objects)
08:16:32   File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_modules.py", line 214, in _test_expected_events
08:16:32     pretty_json(ev), pretty_json(objects))
08:16:32 AssertionError: The following expected object was not found:
08:16:32  {
08:16:32   "elasticsearch.gc.phase.cpu_time.sys_sec": "0.00",
08:16:32   "elasticsearch.gc.phase.name": "CMS Initial Mark",
08:16:32   "elasticsearch.gc.heap.size_kb": "253440",
08:16:32   "elasticsearch.gc.phase.cpu_time.user_sec": "0.01",
08:16:32   "elasticsearch.gc.old_gen.size_kb": "174784",
08:16:32   "elasticsearch.gc.phase.duration_sec": "0.0021716",
08:16:32   "event.category": "database",
08:16:32   "event.kind": "metric",
08:16:32   "@timestamp": "2020-03-20T18:14:44.032Z",
08:16:32   "input.type": "log",
08:16:32   "event.module": "elasticsearch",
08:16:32   "elasticsearch.gc.heap.used_kb": "142444",
08:16:32   "elasticsearch.gc.old_gen.used_kb": "131804",
08:16:32   "elasticsearch.gc.jvm_runtime_sec": "14597.826",
08:16:32   "elasticsearch.gc.phase.cpu_time.real_sec": "0.00",
08:16:32   "service.type": "elasticsearch",
08:16:32   "message": "2020-03-20T18:14:44,032Z: 14597.826: [GC (CMS Initial Mark) [1 CMS-initial-mark: 131804K(174784K)] 142444K(253440K), 0.0021716 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]",
08:16:32   "event.dataset": "elasticsearch.gc",
08:16:32   "event.type": "info",
08:16:32   "fileset.name": "gc",
08:16:32   "log.offset": 22296
08:16:32 }
08:16:32 Searched in: 
08:16:32 [
08:16:32   {
08:16:32     "log": {
08:16:32       "file": {
08:16:32         "path": "/go/src/github.com/elastic/beats/filebeat/module/elasticsearch/gc/test/test-docker.log"
08:16:32       },
08:16:32       "offset": 22296
08:16:32     },
08:16:32     "service": {
08:16:32       "type": "elasticsearch"
08:16:32     },
08:16:32     "@timestamp": "2020-03-20T18:14:44.032Z",
08:16:32     "fileset": {
08:16:32       "name": "gc"
08:16:32     },
08:16:32     "agent": {
08:16:32       "type": "filebeat",
08:16:32       "hostname": "88019cf0b62e",
08:16:32       "ephemeral_id": "ddf826a7-52fc-40a7-8cef-5697ecdaf7a6",
08:16:32       "id": "d70ccd9f-2fa0-4294-8555-b5a6318e8a93",
08:16:32       "version": "7.6.2"
08:16:32     },
08:16:32     "host": {
08:16:32       "name": "88019cf0b62e"
08:16:32     },
08:16:32     "ecs": {
08:16:32       "version": "1.4.0"
08:16:32     },
08:16:32     "elasticsearch": {
08:16:32       "gc": {
08:16:32         "phase": {
08:16:32           "cpu_time": {
08:16:32             "real_sec": "0.00",
08:16:32             "user_sec": "0.01",
08:16:32             "sys_sec": "0.00"
08:16:32           },
08:16:32           "name": "CMS Initial Mark",
08:16:32           "duration_sec": "0.0021716"
08:16:32         },
08:16:32         "old_gen": {
08:16:32           "used_kb": "131804",
08:16:32           "size_kb": "174784"
08:16:32         },
08:16:32         "jvm_runtime_sec": "14597.826",
08:16:32         "heap": {
08:16:32           "used_kb": "142444",
08:16:32           "size_kb": "253440"
08:16:32         }
08:16:32       }
08:16:32     },
08:16:32     "input": {
08:16:32       "type": "log"
08:16:32     },
08:16:32     "message": "2020-03-20T18:14:44,032Z: 14597.826: [GC (CMS Initial Mark) [1 CMS-initial-mark: 131804K(174784K)] 142444K(253440K), 0.0021716 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]",
08:16:32     "event": {
08:16:32       "dataset": "elasticsearch.gc",
08:16:32       "module": "elasticsearch",
08:16:32       "created": "2020-03-24T15:12:45.560Z"
08:16:32     }
08:16:32   }
08:16:32 ]
08:16:32 -------------------- >> begin captured stdout << ---------------------
08:16:32 Using elasticsearch: http://elasticsearch:9200
08:16:32 Testing elasticsearch/gc on /go/src/github.com/elastic/beats/filebeat/tests/system/../../module/elasticsearch/gc/test/test-docker.log
08:16:32 
08:16:32 --------------------- >> end captured stdout << ----------------------

Will investigate and fix!

[ycombinator]

Heya @leehinman, I requested your review on this backport PR because it's not a clean backport. I had to exclude certain fields from the expected test file, based on the changes made in #16469 (which weren't backported to 7.6). So I'd like you to double check my exclusions here, if you don't mind. Thanks!

[leehinman]

LGTM

[ycombinator]

Jenkins CI failures are unrelated. Merging.