From f06b01e14e9c461904aeec3e7b7fdc03b442f5e8 Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Mon, 18 Feb 2019 14:25:13 -0800 Subject: [PATCH 01/10] Add configuration knob for auto-discover hints to control whether log harvesting is enabled for the pod/container. {issue}10811[10811] --- CHANGELOG.next.asciidoc | 47 +++++ .../autodiscover/builder/hints/logs_test.go | 12 ++ filebeat/docs/modules/kubernetes.asciidoc | 35 ++++ libbeat/autodiscover/builder/helper.go | 7 +- libbeat/autodiscover/builder/helper_test.go | 74 +++++++- .../autodiscover/providers/docker/config.go | 15 +- .../autodiscover/providers/docker/docker.go | 2 +- .../providers/kubernetes/config.go | 11 +- .../providers/kubernetes/kubernetes.go | 33 ++-- .../providers/kubernetes/kubernetes_test.go | 167 +----------------- testing/environments/snapshot.yml | 8 +- 11 files changed, 201 insertions(+), 210 deletions(-) create mode 100644 filebeat/docs/modules/kubernetes.asciidoc diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 59b2424a752d..2b30b7dce63c 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -46,6 +46,51 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d *Filebeat* +- Modify apache/error dataset to follow ECS. {pull}8963[8963] +- Rename many `traefik.access.*` fields to map to ECS. {pull}9005[9005] +- Fix parsing of GC entries in elasticsearch server log. {issue}9513[9513] {pull}9810[9810] +- Rename `read_timestamp` to `event.created` for Redis input. {pull}9924[9924] +- Rename a few `elasticsearch.audit.*` fields to map to ECS. {pull}9293[9293] +- Rename `read_timestamp` to `event.created` for all Filebeat modules using it. {pull}10139[10139] +- Rename many `iis.error.*` fields to map to ECS. {pull}9955[9955] +- Adjust fileset `haproxy.log` to map to ECS. {pull}10143[10143] +- Rename a few `logstash.*` fields to map to ECS, remove logstash.slowlog.message. {pull}9935[9935] +- Rename a few `mongodb.*` fields to map to ECS. {pull}10009[10009] +- Rename a few `mysql.*` fields to map to ECS. {pull}10008[10008] +- Rename a few `nginx.error.*` fields to map to ECS. {pull}10007[10007] +- Rename many `auditd.log.*` fields to map to ECS. {pull}10192[10192] +- Filesets with multiple ingest pipelines added in {pull}8914[8914] only work with Elasticsearch >= 6.5.0 {pull}10001[10001] +- Remove service.name from Elastcsearch module. Replace by service.type. {pull}10042[10042] +- Remove numeric coercions for `user.id` and `group.id`. IDs should be `keyword`. {pull}10233[10233] +- Add grok pattern to support redis 5.0.3 log timestamp. {issue}9819[9819] {pull}10033[10033] +- Now save the 'first seen' timestamp in `event.created` (previously `read_timestamp`), + instead of saving the parsed date. Now aligned with `event.created` semantics elsewhere. {pull}10139[10139] +- Rename `mysql.error.thread_id` and `mysql.slowlog.id` to `mysql.thread_id`. {pull}10161[10161] +- Remove `mysql.error.timestamp` and `mysql.slowlog.timestamp`. {pull}10161[10161] +- Migrate multiple fields to `event.duration`, from modules "apache", "elasticsearch", + "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", + including `http.response.elapsed_time` (ECS). {pull}10188[10188], {pull}10274[10274] +- Rename multiple fields to `http.response.body.bytes`, from modules "apache", "iis", + "kibana", "nginx" and "traefik", including `http.response.content_length` (ECS). {pull}10188[10188] +- Change type from haproxy.log fileset fields from text to keyword: response.captured_headers, request.captured_headers, `raw_request_line`, `mode`. {pull}10397[10397] +- Change type of field backend_url and frontend_name in traefik.access metricset to type keyword. {pull}10401[10401] +- Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above {pull}10352[10352] +- Migrate Elasticsearch audit logs fields to ECS {pull}10352[10352] +- Several text fields in the Logstash module are now indexed as `keyword` fields with `text` multi-fields (ECS). {pull}10417[10417] +- Several text fields in the Elasticsearch module are now indexed as `keyword` fields with `text` multi-fields (ECS). {pull}10414[10414] +- Move dissect pattern for traefik.access fileset from Filbeat to Elasticsearch. {pull}10442[10442] +- The `elasticsearch/deprecation` fileset now indexes the `component` field under `elasticsearch` instead of `elasticsearch.server`. {pull}10445[10445] +- Remove field `kafka.log.trace.full` from kafka.log fielset. {pull}10398[10398] +- Change field `kafka.log.class` for kafka.log fileset from text to keyword. {pull}10398[10398] +- Address add_kubernetes_metadata processor issue where old source field is + still used for matcher. {issue}10505[10505] {pull}10506[10506] +- Change type of haproxy.source from text to keyword. {pull}10506[10506] +- Rename `event.type` to `suricata.eve.event_type` in Suricata module because event.type is reserved for future use by ECS. {pull}10575[10575] +- Populate more ECS fields in the Suricata module. {pull}10006[10006] +- Rename setting `filebeat.registry_flush` to `filebeat.registry.flush`. {pull}10504[10504] +- Rename setting `filebeat.registry_file_permission` to `filebeat.registry.file_permission`. {pull}10504[10504] +- Remove setting `filebeat.registry_file` in favor of `filebeat.registry.path`. The registry file will be stored in a sub-directory by now. {pull}10504[10504] + *Heartbeat* - Remove monitor generator script that was rarely used. {pull}9648[9648] @@ -241,6 +286,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Populate more ECS fields in the Suricata module. {pull}10006[10006] - Add ISO8601 timestamp support in syslog metricset. {issue}8716[8716] {pull}10736[10736] - Add more info to message logged when a duplicated symlink file is found {pull}10845[10845] +- Add Filebeat kubernetes module. Add configuration knob for auto-discover hints to control whether log harvesting is enabled for the pod/container. {issue}10811[10811] {issue}10812[10812] +- Add configuration knob for auto-discover hints to control whether log harvesting is enabled for the pod/container. {issue}10811[10811] *Heartbeat* diff --git a/filebeat/autodiscover/builder/hints/logs_test.go b/filebeat/autodiscover/builder/hints/logs_test.go index de56e777765a..e9e1afa72248 100644 --- a/filebeat/autodiscover/builder/hints/logs_test.go +++ b/filebeat/autodiscover/builder/hints/logs_test.go @@ -47,6 +47,18 @@ func TestGenerateHints(t *testing.T) { len: 0, result: common.MapStr{}, }, + { + msg: "Hints with logs.disable should return nothing", + event: bus.Event{ + "hints": common.MapStr{ + "logs": common.MapStr{ + "disable": "true", + }, + }, + }, + len: 0, + result: common.MapStr{}, + }, { msg: "Empty event hints should return default config", event: bus.Event{ diff --git a/filebeat/docs/modules/kubernetes.asciidoc b/filebeat/docs/modules/kubernetes.asciidoc new file mode 100644 index 000000000000..4fc169c5286b --- /dev/null +++ b/filebeat/docs/modules/kubernetes.asciidoc @@ -0,0 +1,35 @@ +//// +This file is generated! See scripts/docs_collector.py +//// + +[[filebeat-module-kubernetes]] +[role="xpack"] + +:modulename: kubernetes +:has-dashboards: true + +== Kubernetes Module + +This is a filebeat module for Kubernetes. It ingests logs for Kubernetes pods and containers, specifically coredns and envoy. + +[float] +=== Compatibility + +This module has been developed against Kubernetes v1.13.x, but is expected to work +with other versions of Kubernetes. + +[float] +=== Example dashboard + +This module comes with a sample dashboard. + +[role="screenshot"] +image::./images/Filebeat-Kubernetes-Dashboard.png[] + + +[float] +=== Fields + +For a description of each field in the module, see the +<> section. + diff --git a/libbeat/autodiscover/builder/helper.go b/libbeat/autodiscover/builder/helper.go index e8a81cfd869b..3a574675b914 100644 --- a/libbeat/autodiscover/builder/helper.go +++ b/libbeat/autodiscover/builder/helper.go @@ -155,7 +155,7 @@ func IsNoOp(hints common.MapStr, key string) bool { } // GenerateHints parses annotations based on a prefix and sets up hints that can be picked up by individual Beats. -func GenerateHints(annotations common.MapStr, container, prefix string) common.MapStr { +func GenerateHints(annotations common.MapStr, container, prefix string, defaultDisable bool) common.MapStr { hints := common.MapStr{} if rawEntries, err := annotations.GetValue(prefix); err == nil { if entries, ok := rawEntries.(common.MapStr); ok { @@ -195,5 +195,10 @@ func GenerateHints(annotations common.MapStr, container, prefix string) common.M } } + // Update hints: if .disabled annotation does not exist, set according to disabledByDefault flag + if _, err := hints.GetValue("logs.disable"); err != nil && defaultDisable { + hints.Put("logs.disable", "true") + } + return hints } diff --git a/libbeat/autodiscover/builder/helper_test.go b/libbeat/autodiscover/builder/helper_test.go index 477888b93f0e..c282972925ec 100644 --- a/libbeat/autodiscover/builder/helper_test.go +++ b/libbeat/autodiscover/builder/helper_test.go @@ -27,13 +27,15 @@ import ( func TestGenerateHints(t *testing.T) { tests := []struct { - annotations map[string]string - result common.MapStr + annotations map[string]string + defaultDisable bool + result common.MapStr }{ // Empty annotations should return empty hints { - annotations: map[string]string{}, - result: common.MapStr{}, + annotations: map[string]string{}, + defaultDisable: false, + result: common.MapStr{}, }, // Scenarios being tested: @@ -50,6 +52,7 @@ func TestGenerateHints(t *testing.T) { "co.elastic.metrics.foobar1/period": "15s", "not.to.include": "true", }, + defaultDisable: false, result: common.MapStr{ "logs": common.MapStr{ "multiline": common.MapStr{ @@ -62,6 +65,67 @@ func TestGenerateHints(t *testing.T) { }, }, }, + // Scenarios being tested: + // logs.disable must be generated when defaultDisable is set and annotations does not + // have co.elastic.logs/disable set to false. + // logs/multiline.pattern must be a nested common.MapStr under hints.logs + // metrics/module must be found in hints.metrics + // not.to.include must not be part of hints + // period is annotated at both container and pod level. Container level value must be in hints + { + annotations: map[string]string{ + "co.elastic.logs/multiline.pattern": "^test", + "co.elastic.metrics/module": "prometheus", + "co.elastic.metrics/period": "10s", + "co.elastic.metrics.foobar/period": "15s", + "co.elastic.metrics.foobar1/period": "15s", + "not.to.include": "true", + }, + defaultDisable: true, + result: common.MapStr{ + "logs": common.MapStr{ + "multiline": common.MapStr{ + "pattern": "^test", + }, + "disable": "true", + }, + "metrics": common.MapStr{ + "module": "prometheus", + "period": "15s", + }, + }, + }, + // Scenarios being tested: + // logs.disable must not be generated when defaultDisable is set, but annotations + // have co.elastic.logs/disable set to false. + // logs/multiline.pattern must be a nested common.MapStr under hints.logs + // metrics/module must be found in hints.metrics + // not.to.include must not be part of hints + // period is annotated at both container and pod level. Container level value must be in hints + { + annotations: map[string]string{ + "co.elastic.logs/disable": "false", + "co.elastic.logs/multiline.pattern": "^test", + "co.elastic.metrics/module": "prometheus", + "co.elastic.metrics/period": "10s", + "co.elastic.metrics.foobar/period": "15s", + "co.elastic.metrics.foobar1/period": "15s", + "not.to.include": "true", + }, + defaultDisable: true, + result: common.MapStr{ + "logs": common.MapStr{ + "multiline": common.MapStr{ + "pattern": "^test", + }, + "disable": "false", + }, + "metrics": common.MapStr{ + "module": "prometheus", + "period": "15s", + }, + }, + }, } for _, test := range tests { @@ -69,6 +133,6 @@ func TestGenerateHints(t *testing.T) { for k, v := range test.annotations { annMap.Put(k, v) } - assert.Equal(t, GenerateHints(annMap, "foobar", "co.elastic"), test.result) + assert.Equal(t, GenerateHints(annMap, "foobar", "co.elastic", test.defaultDisable), test.result) } } diff --git a/libbeat/autodiscover/providers/docker/config.go b/libbeat/autodiscover/providers/docker/config.go index 0b76887a10f6..a0d714ac69ea 100644 --- a/libbeat/autodiscover/providers/docker/config.go +++ b/libbeat/autodiscover/providers/docker/config.go @@ -25,13 +25,14 @@ import ( // Config for docker autodiscover provider type Config struct { - Host string `config:"host"` - TLS *docker.TLSConfig `config:"ssl"` - Prefix string `config:"prefix"` - HintsEnabled bool `config:"hints.enabled"` - Builders []*common.Config `config:"builders"` - Appenders []*common.Config `config:"appenders"` - Templates template.MapperSettings `config:"templates"` + Host string `config:"host"` + TLS *docker.TLSConfig `config:"ssl"` + Prefix string `config:"prefix"` + HintsEnabled bool `config:"hints.enabled"` + DefaultDisable bool `config:"default.disable"` + Builders []*common.Config `config:"builders"` + Appenders []*common.Config `config:"appenders"` + Templates template.MapperSettings `config:"templates"` } func defaultConfig() *Config { diff --git a/libbeat/autodiscover/providers/docker/docker.go b/libbeat/autodiscover/providers/docker/docker.go index 6275897cb9f7..8e561ca81ede 100644 --- a/libbeat/autodiscover/providers/docker/docker.go +++ b/libbeat/autodiscover/providers/docker/docker.go @@ -212,7 +212,7 @@ func (d *Provider) generateHints(event bus.Event) bus.Event { e["port"] = port } if labels, err := dockerMeta.GetValue("labels"); err == nil { - hints := builder.GenerateHints(labels.(common.MapStr), "", d.config.Prefix) + hints := builder.GenerateHints(labels.(common.MapStr), "", d.config.Prefix, d.config.DefaultDisable) e["hints"] = hints } return e diff --git a/libbeat/autodiscover/providers/kubernetes/config.go b/libbeat/autodiscover/providers/kubernetes/config.go index 5ba5f789f68b..269cb60cff36 100644 --- a/libbeat/autodiscover/providers/kubernetes/config.go +++ b/libbeat/autodiscover/providers/kubernetes/config.go @@ -33,11 +33,12 @@ type Config struct { SyncPeriod time.Duration `config:"sync_period"` CleanupTimeout time.Duration `config:"cleanup_timeout"` - Prefix string `config:"prefix"` - HintsEnabled bool `config:"hints.enabled"` - Builders []*common.Config `config:"builders"` - Appenders []*common.Config `config:"appenders"` - Templates template.MapperSettings `config:"templates"` + Prefix string `config:"prefix"` + HintsEnabled bool `config:"hints.enabled"` + DefaultDisable bool `config:"default.disable"` + Builders []*common.Config `config:"builders"` + Appenders []*common.Config `config:"appenders"` + Templates template.MapperSettings `config:"templates"` } func defaultConfig() *Config { diff --git a/libbeat/autodiscover/providers/kubernetes/kubernetes.go b/libbeat/autodiscover/providers/kubernetes/kubernetes.go index c3d9a03dda48..85857fceb4f4 100644 --- a/libbeat/autodiscover/providers/kubernetes/kubernetes.go +++ b/libbeat/autodiscover/providers/kubernetes/kubernetes.go @@ -18,7 +18,6 @@ package kubernetes import ( - "fmt" "time" "github.com/gofrs/uuid" @@ -145,16 +144,12 @@ func (p *Provider) emitEvents(pod *kubernetes.Pod, flag string, containers []*ku containerstatuses []*kubernetes.PodContainerStatus) { host := pod.Status.GetPodIP() - // If the container doesn't exist in the runtime or its network - // is not configured, it won't have an IP. Skip it as we cannot - // generate configs without host, and an update will arrive when - // the container is ready. - // If stopping, emit the event in any case to ensure cleanup. - if host == "" && flag != "stop" { + // Do not emit events without host (container is still being configured) + if host == "" { return } - // Collect all runtimes from status information. + // Collect all container IDs and runtimes from status information. containerIDs := map[string]string{} runtimes := map[string]string{} for _, c := range containerstatuses { @@ -165,18 +160,13 @@ func (p *Provider) emitEvents(pod *kubernetes.Pod, flag string, containers []*ku // Emit container and port information for _, c := range containers { - // If it doesn't have an ID, container doesn't exist in - // the runtime, emit only an event if we are stopping, so - // we are sure of cleaning up configurations. cid := containerIDs[c.GetName()] - if cid == "" && flag != "stop" { + + // If there is a container ID that is empty then ignore it. It either means that the container is still starting + // up or the container is shutting down. + if cid == "" { continue } - - // This must be an id that doesn't depend on the state of the container - // so it works also on `stop` if containers have been already deleted. - eventID := fmt.Sprintf("%s.%s", pod.Metadata.GetUid(), c.GetName()) - cmeta := common.MapStr{ "id": cid, "name": c.GetName(), @@ -200,7 +190,7 @@ func (p *Provider) emitEvents(pod *kubernetes.Pod, flag string, containers []*ku if len(c.Ports) == 0 { event := bus.Event{ "provider": p.uuid, - "id": eventID, + "id": cid, flag: true, "host": host, "kubernetes": kubemeta, @@ -214,7 +204,7 @@ func (p *Provider) emitEvents(pod *kubernetes.Pod, flag string, containers []*ku for _, port := range c.Ports { event := bus.Event{ "provider": p.uuid, - "id": eventID, + "id": cid, flag: true, "host": host, "port": port.GetContainerPort(), @@ -275,12 +265,13 @@ func (p *Provider) generateHints(event bus.Event) bus.Event { } cname := builder.GetContainerName(container) - hints := builder.GenerateHints(annotations, cname, p.config.Prefix) + hints := builder.GenerateHints(annotations, cname, p.config.Prefix, p.config.DefaultDisable) + logp.Debug("kubernetes", "Generated hints %+v", hints) if len(hints) != 0 { e["hints"] = hints } - logp.Debug("kubernetes", "Generated builder event %v", event) + logp.Debug("kubernetes", "Generated builder event %+v", e) return e } diff --git a/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go b/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go index 38ba1ab1c019..e6f8019dfb1c 100644 --- a/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go +++ b/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go @@ -160,7 +160,6 @@ func TestEmitEvent(t *testing.T) { uid := "005f3b90-4b9d-12f8-acf0-31020a840133" containerImage := "elastic/filebeat:6.3.0" node := "node" - cid := "005f3b90-4b9d-12f8-acf0-31020a840133.filebeat" UUID, err := uuid.NewV4() if err != nil { t.Fatal(err) @@ -205,7 +204,7 @@ func TestEmitEvent(t *testing.T) { Expected: bus.Event{ "start": true, "host": "127.0.0.1", - "id": cid, + "id": "foobar", "provider": UUID, "kubernetes": common.MapStr{ "container": common.MapStr{ @@ -271,170 +270,6 @@ func TestEmitEvent(t *testing.T) { }, Expected: nil, }, - { - Message: "Test pod without container id", - Flag: "start", - Pod: &v1.Pod{ - Metadata: &metav1.ObjectMeta{ - Name: &name, - Uid: &uid, - Namespace: &namespace, - Labels: map[string]string{}, - Annotations: map[string]string{}, - }, - Status: &v1.PodStatus{ - PodIP: &podIP, - ContainerStatuses: []*kubernetes.PodContainerStatus{ - { - Name: &name, - }, - }, - }, - Spec: &v1.PodSpec{ - NodeName: &node, - Containers: []*kubernetes.Container{ - { - Image: &containerImage, - Name: &name, - }, - }, - }, - }, - Expected: nil, - }, - { - Message: "Test stop pod without host", - Flag: "stop", - Pod: &v1.Pod{ - Metadata: &metav1.ObjectMeta{ - Name: &name, - Uid: &uid, - Namespace: &namespace, - Labels: map[string]string{}, - Annotations: map[string]string{}, - }, - Status: &v1.PodStatus{ - ContainerStatuses: []*kubernetes.PodContainerStatus{ - { - Name: &name, - }, - }, - }, - Spec: &v1.PodSpec{ - NodeName: &node, - Containers: []*kubernetes.Container{ - { - Image: &containerImage, - Name: &name, - }, - }, - }, - }, - Expected: bus.Event{ - "stop": true, - "host": "", - "id": cid, - "provider": UUID, - "kubernetes": common.MapStr{ - "container": common.MapStr{ - "id": "", - "name": "filebeat", - "image": "elastic/filebeat:6.3.0", - "runtime": "", - }, - "pod": common.MapStr{ - "name": "filebeat", - "uid": "005f3b90-4b9d-12f8-acf0-31020a840133", - }, - "node": common.MapStr{ - "name": "node", - }, - "namespace": "default", - "annotations": common.MapStr{}, - }, - "meta": common.MapStr{ - "kubernetes": common.MapStr{ - "namespace": "default", - "container": common.MapStr{ - "name": "filebeat", - }, "pod": common.MapStr{ - "name": "filebeat", - "uid": "005f3b90-4b9d-12f8-acf0-31020a840133", - }, "node": common.MapStr{ - "name": "node", - }, - }, - }, - "config": []*common.Config{}, - }, - }, - { - Message: "Test stop pod without container id", - Flag: "stop", - Pod: &v1.Pod{ - Metadata: &metav1.ObjectMeta{ - Name: &name, - Uid: &uid, - Namespace: &namespace, - Labels: map[string]string{}, - Annotations: map[string]string{}, - }, - Status: &v1.PodStatus{ - PodIP: &podIP, - ContainerStatuses: []*kubernetes.PodContainerStatus{ - { - Name: &name, - }, - }, - }, - Spec: &v1.PodSpec{ - NodeName: &node, - Containers: []*kubernetes.Container{ - { - Image: &containerImage, - Name: &name, - }, - }, - }, - }, - Expected: bus.Event{ - "stop": true, - "host": "127.0.0.1", - "id": cid, - "provider": UUID, - "kubernetes": common.MapStr{ - "container": common.MapStr{ - "id": "", - "name": "filebeat", - "image": "elastic/filebeat:6.3.0", - "runtime": "", - }, - "pod": common.MapStr{ - "name": "filebeat", - "uid": "005f3b90-4b9d-12f8-acf0-31020a840133", - }, - "node": common.MapStr{ - "name": "node", - }, - "namespace": "default", - "annotations": common.MapStr{}, - }, - "meta": common.MapStr{ - "kubernetes": common.MapStr{ - "namespace": "default", - "container": common.MapStr{ - "name": "filebeat", - }, "pod": common.MapStr{ - "name": "filebeat", - "uid": "005f3b90-4b9d-12f8-acf0-31020a840133", - }, "node": common.MapStr{ - "name": "node", - }, - }, - }, - "config": []*common.Config{}, - }, - }, } for _, test := range tests { diff --git a/testing/environments/snapshot.yml b/testing/environments/snapshot.yml index 0d0fd4fbc2a2..308ca0d935e0 100644 --- a/testing/environments/snapshot.yml +++ b/testing/environments/snapshot.yml @@ -3,7 +3,7 @@ version: '2.3' services: elasticsearch: - image: docker.elastic.co/elasticsearch/elasticsearch:7.0.0-SNAPSHOT + image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0-SNAPSHOT healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9200"] retries: 300 @@ -11,12 +11,12 @@ services: environment: - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - "network.host=" - - "transport.host=127.0.0.1" + - "transport.host=127.1.0.1" - "http.host=0.0.0.0" - "xpack.security.enabled=false" logstash: - image: docker.elastic.co/logstash/logstash:7.0.0-SNAPSHOT + image: docker.elastic.co/logstash/logstash:7.1.0-SNAPSHOT healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9600/_node/stats"] retries: 600 @@ -26,7 +26,7 @@ services: - ./docker/logstash/pki:/etc/pki:ro kibana: - image: docker.elastic.co/kibana/kibana:7.0.0-SNAPSHOT + image: docker.elastic.co/kibana/kibana:7.1.0-SNAPSHOT healthcheck: test: ["CMD-SHELL", 'python -c ''import urllib, json; response = urllib.urlopen("http://localhost:5601/api/status"); data = json.loads(response.read()); exit(1) if data["status"]["overall"]["state"] != "green" else exit(0);'''] retries: 600 From 1715167814f2f0f744692d1c30fd34367bdf1cfe Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Fri, 22 Feb 2019 10:34:14 -0800 Subject: [PATCH 02/10] Remove filebeat/docs/modules/kubernetes.asciidoc because kubernetes module is moved to a different PR --- filebeat/docs/modules/kubernetes.asciidoc | 35 ----------------------- 1 file changed, 35 deletions(-) delete mode 100644 filebeat/docs/modules/kubernetes.asciidoc diff --git a/filebeat/docs/modules/kubernetes.asciidoc b/filebeat/docs/modules/kubernetes.asciidoc deleted file mode 100644 index 4fc169c5286b..000000000000 --- a/filebeat/docs/modules/kubernetes.asciidoc +++ /dev/null @@ -1,35 +0,0 @@ -//// -This file is generated! See scripts/docs_collector.py -//// - -[[filebeat-module-kubernetes]] -[role="xpack"] - -:modulename: kubernetes -:has-dashboards: true - -== Kubernetes Module - -This is a filebeat module for Kubernetes. It ingests logs for Kubernetes pods and containers, specifically coredns and envoy. - -[float] -=== Compatibility - -This module has been developed against Kubernetes v1.13.x, but is expected to work -with other versions of Kubernetes. - -[float] -=== Example dashboard - -This module comes with a sample dashboard. - -[role="screenshot"] -image::./images/Filebeat-Kubernetes-Dashboard.png[] - - -[float] -=== Fields - -For a description of each field in the module, see the -<> section. - From 0c246d6388def585cbba49c4461bd393a853b3eb Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Fri, 22 Feb 2019 10:42:53 -0800 Subject: [PATCH 03/10] Fix CHANGELOG.next.asciidoc --- CHANGELOG.next.asciidoc | 46 ----------------------------------------- 1 file changed, 46 deletions(-) diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 2b30b7dce63c..bbec9db02def 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -46,51 +46,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d *Filebeat* -- Modify apache/error dataset to follow ECS. {pull}8963[8963] -- Rename many `traefik.access.*` fields to map to ECS. {pull}9005[9005] -- Fix parsing of GC entries in elasticsearch server log. {issue}9513[9513] {pull}9810[9810] -- Rename `read_timestamp` to `event.created` for Redis input. {pull}9924[9924] -- Rename a few `elasticsearch.audit.*` fields to map to ECS. {pull}9293[9293] -- Rename `read_timestamp` to `event.created` for all Filebeat modules using it. {pull}10139[10139] -- Rename many `iis.error.*` fields to map to ECS. {pull}9955[9955] -- Adjust fileset `haproxy.log` to map to ECS. {pull}10143[10143] -- Rename a few `logstash.*` fields to map to ECS, remove logstash.slowlog.message. {pull}9935[9935] -- Rename a few `mongodb.*` fields to map to ECS. {pull}10009[10009] -- Rename a few `mysql.*` fields to map to ECS. {pull}10008[10008] -- Rename a few `nginx.error.*` fields to map to ECS. {pull}10007[10007] -- Rename many `auditd.log.*` fields to map to ECS. {pull}10192[10192] -- Filesets with multiple ingest pipelines added in {pull}8914[8914] only work with Elasticsearch >= 6.5.0 {pull}10001[10001] -- Remove service.name from Elastcsearch module. Replace by service.type. {pull}10042[10042] -- Remove numeric coercions for `user.id` and `group.id`. IDs should be `keyword`. {pull}10233[10233] -- Add grok pattern to support redis 5.0.3 log timestamp. {issue}9819[9819] {pull}10033[10033] -- Now save the 'first seen' timestamp in `event.created` (previously `read_timestamp`), - instead of saving the parsed date. Now aligned with `event.created` semantics elsewhere. {pull}10139[10139] -- Rename `mysql.error.thread_id` and `mysql.slowlog.id` to `mysql.thread_id`. {pull}10161[10161] -- Remove `mysql.error.timestamp` and `mysql.slowlog.timestamp`. {pull}10161[10161] -- Migrate multiple fields to `event.duration`, from modules "apache", "elasticsearch", - "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", - including `http.response.elapsed_time` (ECS). {pull}10188[10188], {pull}10274[10274] -- Rename multiple fields to `http.response.body.bytes`, from modules "apache", "iis", - "kibana", "nginx" and "traefik", including `http.response.content_length` (ECS). {pull}10188[10188] -- Change type from haproxy.log fileset fields from text to keyword: response.captured_headers, request.captured_headers, `raw_request_line`, `mode`. {pull}10397[10397] -- Change type of field backend_url and frontend_name in traefik.access metricset to type keyword. {pull}10401[10401] -- Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above {pull}10352[10352] -- Migrate Elasticsearch audit logs fields to ECS {pull}10352[10352] -- Several text fields in the Logstash module are now indexed as `keyword` fields with `text` multi-fields (ECS). {pull}10417[10417] -- Several text fields in the Elasticsearch module are now indexed as `keyword` fields with `text` multi-fields (ECS). {pull}10414[10414] -- Move dissect pattern for traefik.access fileset from Filbeat to Elasticsearch. {pull}10442[10442] -- The `elasticsearch/deprecation` fileset now indexes the `component` field under `elasticsearch` instead of `elasticsearch.server`. {pull}10445[10445] -- Remove field `kafka.log.trace.full` from kafka.log fielset. {pull}10398[10398] -- Change field `kafka.log.class` for kafka.log fileset from text to keyword. {pull}10398[10398] -- Address add_kubernetes_metadata processor issue where old source field is - still used for matcher. {issue}10505[10505] {pull}10506[10506] -- Change type of haproxy.source from text to keyword. {pull}10506[10506] -- Rename `event.type` to `suricata.eve.event_type` in Suricata module because event.type is reserved for future use by ECS. {pull}10575[10575] -- Populate more ECS fields in the Suricata module. {pull}10006[10006] -- Rename setting `filebeat.registry_flush` to `filebeat.registry.flush`. {pull}10504[10504] -- Rename setting `filebeat.registry_file_permission` to `filebeat.registry.file_permission`. {pull}10504[10504] -- Remove setting `filebeat.registry_file` in favor of `filebeat.registry.path`. The registry file will be stored in a sub-directory by now. {pull}10504[10504] - *Heartbeat* - Remove monitor generator script that was rarely used. {pull}9648[9648] @@ -286,7 +241,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Populate more ECS fields in the Suricata module. {pull}10006[10006] - Add ISO8601 timestamp support in syslog metricset. {issue}8716[8716] {pull}10736[10736] - Add more info to message logged when a duplicated symlink file is found {pull}10845[10845] -- Add Filebeat kubernetes module. Add configuration knob for auto-discover hints to control whether log harvesting is enabled for the pod/container. {issue}10811[10811] {issue}10812[10812] - Add configuration knob for auto-discover hints to control whether log harvesting is enabled for the pod/container. {issue}10811[10811] *Heartbeat* From 5ec18885a11bd66ebc097d82c587475d15e7b5f8 Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Fri, 22 Feb 2019 10:45:57 -0800 Subject: [PATCH 04/10] Fix auto-merge issues on file libbeat/autodiscover/providers/kubernetes/kubernetes_test.go --- .../providers/kubernetes/kubernetes_test.go | 164 ++++++++++++++++++ 1 file changed, 164 insertions(+) diff --git a/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go b/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go index e6f8019dfb1c..2c45f538ec07 100644 --- a/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go +++ b/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go @@ -270,6 +270,170 @@ func TestEmitEvent(t *testing.T) { }, Expected: nil, }, + { + Message: "Test pod without container id", + Flag: "start", + Pod: &v1.Pod{ + Metadata: &metav1.ObjectMeta{ + Name: &name, + Uid: &uid, + Namespace: &namespace, + Labels: map[string]string{}, + Annotations: map[string]string{}, + }, + Status: &v1.PodStatus{ + PodIP: &podIP, + ContainerStatuses: []*kubernetes.PodContainerStatus{ + { + Name: &name, + }, + }, + }, + Spec: &v1.PodSpec{ + NodeName: &node, + Containers: []*kubernetes.Container{ + { + Image: &containerImage, + Name: &name, + }, + }, + }, + }, + Expected: nil, + }, + { + Message: "Test stop pod without host", + Flag: "stop", + Pod: &v1.Pod{ + Metadata: &metav1.ObjectMeta{ + Name: &name, + Uid: &uid, + Namespace: &namespace, + Labels: map[string]string{}, + Annotations: map[string]string{}, + }, + Status: &v1.PodStatus{ + ContainerStatuses: []*kubernetes.PodContainerStatus{ + { + Name: &name, + }, + }, + }, + Spec: &v1.PodSpec{ + NodeName: &node, + Containers: []*kubernetes.Container{ + { + Image: &containerImage, + Name: &name, + }, + }, + }, + }, + Expected: bus.Event{ + "stop": true, + "host": "", + "id": cid, + "provider": UUID, + "kubernetes": common.MapStr{ + "container": common.MapStr{ + "id": "", + "name": "filebeat", + "image": "elastic/filebeat:6.3.0", + "runtime": "", + }, + "pod": common.MapStr{ + "name": "filebeat", + "uid": "005f3b90-4b9d-12f8-acf0-31020a840133", + }, + "node": common.MapStr{ + "name": "node", + }, + "namespace": "default", + "annotations": common.MapStr{}, + }, + "meta": common.MapStr{ + "kubernetes": common.MapStr{ + "namespace": "default", + "container": common.MapStr{ + "name": "filebeat", + }, "pod": common.MapStr{ + "name": "filebeat", + "uid": "005f3b90-4b9d-12f8-acf0-31020a840133", + }, "node": common.MapStr{ + "name": "node", + }, + }, + }, + "config": []*common.Config{}, + }, + }, + { + Message: "Test stop pod without container id", + Flag: "stop", + Pod: &v1.Pod{ + Metadata: &metav1.ObjectMeta{ + Name: &name, + Uid: &uid, + Namespace: &namespace, + Labels: map[string]string{}, + Annotations: map[string]string{}, + }, + Status: &v1.PodStatus{ + PodIP: &podIP, + ContainerStatuses: []*kubernetes.PodContainerStatus{ + { + Name: &name, + }, + }, + }, + Spec: &v1.PodSpec{ + NodeName: &node, + Containers: []*kubernetes.Container{ + { + Image: &containerImage, + Name: &name, + }, + }, + }, + }, + Expected: bus.Event{ + "stop": true, + "host": "127.0.0.1", + "id": cid, + "provider": UUID, + "kubernetes": common.MapStr{ + "container": common.MapStr{ + "id": "", + "name": "filebeat", + "image": "elastic/filebeat:6.3.0", + "runtime": "", + }, + "pod": common.MapStr{ + "name": "filebeat", + "uid": "005f3b90-4b9d-12f8-acf0-31020a840133", + }, + "node": common.MapStr{ + "name": "node", + }, + "namespace": "default", + "annotations": common.MapStr{}, + }, + "meta": common.MapStr{ + "kubernetes": common.MapStr{ + "namespace": "default", + "container": common.MapStr{ + "name": "filebeat", + }, "pod": common.MapStr{ + "name": "filebeat", + "uid": "005f3b90-4b9d-12f8-acf0-31020a840133", + }, "node": common.MapStr{ + "name": "node", + }, + }, + }, + "config": []*common.Config{}, + }, + }, } for _, test := range tests { From 6f3f7808c4c6d712eaf9a30a38ab9a031c8a360e Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Fri, 22 Feb 2019 10:54:57 -0800 Subject: [PATCH 05/10] Fix auto-merge issues on file libbeat/autodiscover/providers/kubernetes/kubernetes.go --- .../providers/kubernetes/kubernetes.go | 28 +++++++++++++------ 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/libbeat/autodiscover/providers/kubernetes/kubernetes.go b/libbeat/autodiscover/providers/kubernetes/kubernetes.go index 85857fceb4f4..41037238edd7 100644 --- a/libbeat/autodiscover/providers/kubernetes/kubernetes.go +++ b/libbeat/autodiscover/providers/kubernetes/kubernetes.go @@ -18,6 +18,7 @@ package kubernetes import ( + "fmt" "time" "github.com/gofrs/uuid" @@ -144,12 +145,16 @@ func (p *Provider) emitEvents(pod *kubernetes.Pod, flag string, containers []*ku containerstatuses []*kubernetes.PodContainerStatus) { host := pod.Status.GetPodIP() - // Do not emit events without host (container is still being configured) - if host == "" { + // If the container doesn't exist in the runtime or its network + // is not configured, it won't have an IP. Skip it as we cannot + // generate configs without host, and an update will arrive when + // the container is ready. + // If stopping, emit the event in any case to ensure cleanup. + if host == "" && flag != "stop" { return } - // Collect all container IDs and runtimes from status information. + // Collect all runtimes from status information. containerIDs := map[string]string{} runtimes := map[string]string{} for _, c := range containerstatuses { @@ -160,13 +165,18 @@ func (p *Provider) emitEvents(pod *kubernetes.Pod, flag string, containers []*ku // Emit container and port information for _, c := range containers { + // If it doesn't have an ID, container doesn't exist in + // the runtime, emit only an event if we are stopping, so + // we are sure of cleaning up configurations. cid := containerIDs[c.GetName()] - - // If there is a container ID that is empty then ignore it. It either means that the container is still starting - // up or the container is shutting down. - if cid == "" { + if cid == "" && flag != "stop" { continue } + + // This must be an id that doesn't depend on the state of the container + // so it works also on `stop` if containers have been already deleted. + eventID := fmt.Sprintf("%s.%s", pod.Metadata.GetUid(), c.GetName()) + cmeta := common.MapStr{ "id": cid, "name": c.GetName(), @@ -190,7 +200,7 @@ func (p *Provider) emitEvents(pod *kubernetes.Pod, flag string, containers []*ku if len(c.Ports) == 0 { event := bus.Event{ "provider": p.uuid, - "id": cid, + "id": eventID, flag: true, "host": host, "kubernetes": kubemeta, @@ -204,7 +214,7 @@ func (p *Provider) emitEvents(pod *kubernetes.Pod, flag string, containers []*ku for _, port := range c.Ports { event := bus.Event{ "provider": p.uuid, - "id": cid, + "id": eventID, flag: true, "host": host, "port": port.GetContainerPort(), From afc4bf82f2f5cb3011781e848ceaf224b6460b06 Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Fri, 22 Feb 2019 10:56:49 -0800 Subject: [PATCH 06/10] Fix auto-merge issues on file libbeat/autodiscover/providers/kubernetes/kubernetes_test.go --- libbeat/autodiscover/providers/kubernetes/kubernetes_test.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go b/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go index 2c45f538ec07..38ba1ab1c019 100644 --- a/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go +++ b/libbeat/autodiscover/providers/kubernetes/kubernetes_test.go @@ -160,6 +160,7 @@ func TestEmitEvent(t *testing.T) { uid := "005f3b90-4b9d-12f8-acf0-31020a840133" containerImage := "elastic/filebeat:6.3.0" node := "node" + cid := "005f3b90-4b9d-12f8-acf0-31020a840133.filebeat" UUID, err := uuid.NewV4() if err != nil { t.Fatal(err) @@ -204,7 +205,7 @@ func TestEmitEvent(t *testing.T) { Expected: bus.Event{ "start": true, "host": "127.0.0.1", - "id": "foobar", + "id": cid, "provider": UUID, "kubernetes": common.MapStr{ "container": common.MapStr{ @@ -270,7 +271,7 @@ func TestEmitEvent(t *testing.T) { }, Expected: nil, }, - { + { Message: "Test pod without container id", Flag: "start", Pod: &v1.Pod{ From 1d495f14771c16b3971a002b080fa5e808e180d7 Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Fri, 22 Feb 2019 11:06:22 -0800 Subject: [PATCH 07/10] Fix a search-and-replace issue --- testing/environments/snapshot.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/testing/environments/snapshot.yml b/testing/environments/snapshot.yml index 308ca0d935e0..a119227ca519 100644 --- a/testing/environments/snapshot.yml +++ b/testing/environments/snapshot.yml @@ -11,7 +11,7 @@ services: environment: - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - "network.host=" - - "transport.host=127.1.0.1" + - "transport.host=127.0.0.1" - "http.host=0.0.0.0" - "xpack.security.enabled=false" From 4fa20af156b6c0ff382bd35745065743fbb70df0 Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Mon, 25 Feb 2019 08:03:33 -0800 Subject: [PATCH 08/10] fmt after merging in web-editor --- libbeat/autodiscover/providers/docker/config.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/libbeat/autodiscover/providers/docker/config.go b/libbeat/autodiscover/providers/docker/config.go index 319ab60c6372..bba4f32cdb0c 100644 --- a/libbeat/autodiscover/providers/docker/config.go +++ b/libbeat/autodiscover/providers/docker/config.go @@ -25,15 +25,15 @@ import ( // Config for docker autodiscover provider type Config struct { - Host string `config:"host"` - TLS *docker.TLSConfig `config:"ssl"` - Prefix string `config:"prefix"` - HintsEnabled bool `config:"hints.enabled"` + Host string `config:"host"` + TLS *docker.TLSConfig `config:"ssl"` + Prefix string `config:"prefix"` + HintsEnabled bool `config:"hints.enabled"` DefaultDisable bool `config:"default.disable"` - Builders []*common.Config `config:"builders"` - Appenders []*common.Config `config:"appenders"` - Templates template.MapperSettings `config:"templates"` - Dedot bool `config:"labels.dedot"` + Builders []*common.Config `config:"builders"` + Appenders []*common.Config `config:"appenders"` + Templates template.MapperSettings `config:"templates"` + Dedot bool `config:"labels.dedot"` } func defaultConfig() *Config { From 1d5fb4ad7b2179611c91f03f3f6c5234d2ec5062 Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Mon, 25 Feb 2019 10:35:31 -0800 Subject: [PATCH 09/10] Add documentation for auto-discover default.disable --- filebeat/docs/autodiscover-hints.asciidoc | 39 +++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/filebeat/docs/autodiscover-hints.asciidoc b/filebeat/docs/autodiscover-hints.asciidoc index a49a8abd055e..b3e236df6aad 100644 --- a/filebeat/docs/autodiscover-hints.asciidoc +++ b/filebeat/docs/autodiscover-hints.asciidoc @@ -94,6 +94,25 @@ filebeat.autodiscover: hints.enabled: true ------------------------------------------------------------------------------------- +Autodiscover provider a way to control whether log harvesting is by default disabled for the pods/containers when auto-discovery is in use. To enable it, just set `default.disable` to true: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------------- +filebeat.autodiscover: + providers: + - type: kubernetes + hints.enabled: true + default.disable: true +------------------------------------------------------------------------------------- + +Then, for the pods/containers that log harvesting should be enabled, you can annotate with hint: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------------- +annotations: + co.elastic.logs/disable: false +------------------------------------------------------------------------------------- + You can annotate Kubernetes Pods with useful info to spin up {beatname_uc} inputs or modules: ["source","yaml",subs="attributes"] @@ -137,6 +156,26 @@ filebeat.autodiscover: hints.enabled: true ------------------------------------------------------------------------------------- +Autodiscover provider a way to control whether log harvesting is by default disabled for the +containers when auto-discovery is in use. To enable it, just set `default.disable` to true: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------------- +filebeat.autodiscover: + providers: + - type: docker + hints.enabled: true + default.disable: true +------------------------------------------------------------------------------------- + +Then, for the containers that log harvesting should be enabled, you can label Docker containers with: + +["source","yaml",subs="attributes"] +------------------------------------------------------------------------------------- +annotations: + co.elastic.logs/disable: false +------------------------------------------------------------------------------------- + You can label Docker containers with useful info to spin up {beatname_uc} inputs, for example: ["source","yaml",subs="attributes"] From 6f139a11d6d221894cdaadc25bd1d8a61b61ce16 Mon Sep 17 00:00:00 2001 From: Ray Qiu Date: Mon, 25 Feb 2019 10:58:20 -0800 Subject: [PATCH 10/10] Add documentation for auto-discover default.disable --- filebeat/docs/autodiscover-hints.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/filebeat/docs/autodiscover-hints.asciidoc b/filebeat/docs/autodiscover-hints.asciidoc index b3e236df6aad..9d80d28c7adb 100644 --- a/filebeat/docs/autodiscover-hints.asciidoc +++ b/filebeat/docs/autodiscover-hints.asciidoc @@ -94,7 +94,7 @@ filebeat.autodiscover: hints.enabled: true ------------------------------------------------------------------------------------- -Autodiscover provider a way to control whether log harvesting is by default disabled for the pods/containers when auto-discovery is in use. To enable it, just set `default.disable` to true: +Autodiscover provides a way to control whether log harvesting is by default disabled for the pods/containers when auto-discovery is in use. To enable it, just set `default.disable` to true: ["source","yaml",subs="attributes"] ------------------------------------------------------------------------------------- @@ -156,7 +156,7 @@ filebeat.autodiscover: hints.enabled: true ------------------------------------------------------------------------------------- -Autodiscover provider a way to control whether log harvesting is by default disabled for the +Autodiscover provides a way to control whether log harvesting is by default disabled for the containers when auto-discovery is in use. To enable it, just set `default.disable` to true: ["source","yaml",subs="attributes"]