Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[elasticsearch] Align mappings with metricbeat #3928

Merged
merged 9 commits into from
Aug 4, 2022
Merged

[elasticsearch] Align mappings with metricbeat #3928

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
fe67816
align ecs fields
klacabane Aug 1, 2022
cc779c9
add index_stats property group
klacabane Aug 1, 2022
e74a67e
remove unused version index_recovery property
klacabane Aug 1, 2022
f7d56b2
align node_stats fields type
klacabane Aug 1, 2022
f4c3179
turn dynamic off for cluster_stats mappings
klacabane Aug 1, 2022
d17cf5c
bump kibana minimum version
klacabane Aug 1, 2022
8ef350e
lint
klacabane Aug 1, 2022
bfec760
disable dynamic mappings for every data streams
klacabane Aug 3, 2022
f856c85
add missing shard property
klacabane Aug 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions packages/elasticsearch/data_stream/ccr/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
26 changes: 24 additions & 2 deletions packages/elasticsearch/data_stream/ccr/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,24 @@
- external: ecs
name: '@timestamp'
- name: '@timestamp'
external: ecs

- name: ecs.version
external: ecs

- name: event.dataset
external: ecs
- name: event.duration
external: ecs
- name: event.module
external: ecs

- name: host.name
external: ecs

- name: service.address
type: keyword
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs

3 changes: 0 additions & 3 deletions packages/elasticsearch/data_stream/cluster_stats/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
15 changes: 15 additions & 0 deletions packages/elasticsearch/data_stream/cluster_stats/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,24 @@
- name: '@timestamp'
external: ecs

- name: ecs.version
external: ecs

- name: event.dataset
external: ecs
- name: event.duration
external: ecs
- name: event.module
external: ecs

- name: host.name
external: ecs

- name: service.address
type: keyword
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs

4 changes: 4 additions & 0 deletions packages/elasticsearch/data_stream/cluster_stats/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
type: metrics
title: Elasticsearch cluster_stats metrics
release: experimental
elasticsearch:
index_template:
mappings:
dynamic: false
Comment on lines +4 to +7
Copy link
Contributor Author

@klacabane klacabane Aug 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cluster_stats is the offending metricset that caused the initial issue in #3918 so I'd be inclined to close it at the same time, unless we want to review all metricsets and apply dynamic: false where necessary

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ended up disabling dynamic mappings on every data streams

streams:
- input: elasticsearch/metrics
title: Cluster stats
Expand Down
3 changes: 0 additions & 3 deletions packages/elasticsearch/data_stream/enrich/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
14 changes: 14 additions & 0 deletions packages/elasticsearch/data_stream/enrich/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
- name: '@timestamp'
external: ecs

- name: ecs.version
external: ecs

- name: event.dataset
external: ecs
- name: event.duration
external: ecs
- name: event.module
external: ecs

- name: host.name
external: ecs

- name: service.address
type: keyword
description: Service address
Expand Down
3 changes: 0 additions & 3 deletions packages/elasticsearch/data_stream/index/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
14 changes: 14 additions & 0 deletions packages/elasticsearch/data_stream/index/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
- name: '@timestamp'
external: ecs

- name: ecs.version
external: ecs

- name: event.dataset
external: ecs
- name: event.duration
external: ecs
- name: event.module
external: ecs

- name: host.name
external: ecs

- name: service.address
type: keyword
description: Service address
Expand Down
147 changes: 147 additions & 0 deletions packages/elasticsearch/data_stream/index/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,153 @@
- name: index_total
type: alias
path: elasticsearch.index.summary.total.indexing.index.count
- name: index_stats
type: group
fields:
- name: primaries
type: group
fields:
- name: docs
type: group
fields:
- name: count
path: elasticsearch.index.primaries.docs.count
type: alias
- name: indexing
type: group
fields:
- name: throttle_time_in_millis
path: elasticsearch.index.primaries.indexing.throttle_time_in_millis
type: alias
- name: index_time_in_millis
path: elasticsearch.index.primaries.indexing.index_time_in_millis
type: alias
- name: index_total
path: elasticsearch.index.primaries.indexing.index_total
type: alias
- name: refresh
type: group
fields:
- name: total_time_in_millis
path: elasticsearch.index.primaries.refresh.total_time_in_millis
type: alias
- name: store
type: group
fields:
- name: size_in_bytes
path: elasticsearch.index.primaries.store.size_in_bytes
type: alias
- name: merges
type: group
fields:
- name: total_size_in_bytes
path: elasticsearch.index.primaries.merges.total_size_in_bytes
type: alias
- name: segments
type: group
fields:
- name: count
path: elasticsearch.index.primaries.segments.count
type: alias
- name: total
type: group
fields:
- name: search
type: group
fields:
- name: query_total
path: elasticsearch.index.total.search.query_total
type: alias
- name: query_time_in_millis
path: elasticsearch.index.total.search.query_time_in_millis
type: alias
- name: query_cache
type: group
fields:
- name: memory_size_in_bytes
path: elasticsearch.index.total.query_cache.memory_size_in_bytes
type: alias
- name: fielddata
type: group
fields:
- name: memory_size_in_bytes
path: elasticsearch.index.total.fielddata.memory_size_in_bytes
type: alias
- name: indexing
type: group
fields:
- name: throttle_time_in_millis
path: elasticsearch.index.total.indexing.throttle_time_in_millis
type: alias
- name: index_time_in_millis
path: elasticsearch.index.total.indexing.index_time_in_millis
type: alias
- name: index_total
path: elasticsearch.index.total.indexing.index_total
type: alias
- name: refresh
type: group
fields:
- name: total_time_in_millis
path: elasticsearch.index.total.refresh.total_time_in_millis
type: alias
- name: request_cache
type: group
fields:
- name: memory_size_in_bytes
path: elasticsearch.index.total.request_cache.memory_size_in_bytes
type: alias
- name: store
type: group
fields:
- name: size_in_bytes
path: elasticsearch.index.total.store.size_in_bytes
type: alias
- name: merges
type: group
fields:
- name: total_size_in_bytes
path: elasticsearch.index.total.merges.total_size_in_bytes
type: alias
- name: segments
type: group
fields:
- name: version_map_memory_in_bytes
path: elasticsearch.index.total.segments.version_map_memory_in_bytes
type: alias
- name: norms_memory_in_bytes
path: elasticsearch.index.total.segments.norms_memory_in_bytes
type: alias
- name: count
path: elasticsearch.index.total.segments.count
type: alias
- name: term_vectors_memory_in_bytes
path: elasticsearch.index.total.segments.term_vectors_memory_in_bytes
type: alias
- name: points_memory_in_bytes
path: elasticsearch.index.total.segments.points_memory_in_bytes
type: alias
- name: index_writer_memory_in_bytes
path: elasticsearch.index.total.segments.index_writer_memory_in_bytes
type: alias
- name: memory_in_bytes
path: elasticsearch.index.total.segments.memory_in_bytes
type: alias
- name: doc_values_memory_in_bytes
path: elasticsearch.index.total.segments.doc_values_memory_in_bytes
type: alias
- name: terms_memory_in_bytes
path: elasticsearch.index.total.segments.terms_memory_in_bytes
type: alias
- name: fixed_bit_set_memory_in_bytes
path: elasticsearch.index.total.segments.fixed_bit_set_memory_in_bytes
type: alias
- name: stored_fields_memory_in_bytes
path: elasticsearch.index.total.segments.stored_fields_memory_in_bytes
type: alias
- name: index
path: elasticsearch.index.name
type: alias
- name: elasticsearch
type: group
fields:
Expand Down
3 changes: 0 additions & 3 deletions packages/elasticsearch/data_stream/index_recovery/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
15 changes: 15 additions & 0 deletions packages/elasticsearch/data_stream/index_recovery/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,24 @@
- name: '@timestamp'
external: ecs

- name: ecs.version
external: ecs

- name: event.dataset
external: ecs
- name: event.duration
external: ecs
- name: event.module
external: ecs

- name: host.name
external: ecs

- name: service.address
type: keyword
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs

2 changes: 0 additions & 2 deletions packages/elasticsearch/data_stream/index_recovery/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@
- name: name
type: alias
path: elasticsearch.node.name
- name: version
type: keyword
- name: index_recovery
type: group
fields:
Expand Down
3 changes: 0 additions & 3 deletions packages/elasticsearch/data_stream/index_summary/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
14 changes: 14 additions & 0 deletions packages/elasticsearch/data_stream/index_summary/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
- name: '@timestamp'
external: ecs

- name: ecs.version
external: ecs

- name: event.dataset
external: ecs
- name: event.duration
external: ecs
- name: event.module
external: ecs

- name: host.name
external: ecs

- name: service.address
type: keyword
description: Service address
Expand Down
3 changes: 0 additions & 3 deletions packages/elasticsearch/data_stream/ml_job/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
30 changes: 22 additions & 8 deletions packages/elasticsearch/data_stream/ml_job/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,23 @@
- name: '@timestamp'
level: core
required: true
type: date
description: |-
Date/time when the event originated.
This is the date/time extracted from the event, typically representing when the event was generated by the source.
If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.
Required field for all events.
external: ecs

- name: ecs.version
external: ecs

- name: event.dataset
external: ecs
- name: event.duration
external: ecs
- name: event.module
external: ecs

- name: host.name
external: ecs

- name: service.address
type: keyword
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
3 changes: 0 additions & 3 deletions packages/elasticsearch/data_stream/node/fields/base-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
Loading