Write deprecation logs to a data stream #61484

pugnascotia · 2020-08-24T15:15:28Z

Closes #46106. Implement a new log4j appender for deprecation logging, in order to write logs to a dedicated data stream. This is controlled by a new setting, cluster.deprecation_indexing.enabled.

Depends on #61474.

Test by running ./gradlew run and then:

AUTH="elastic-admin:elastic-password"

# Enable new setting
curl -u $AUTH -XPUT \
  --data '{ "transient": { "cluster.deprecation_indexing.enabled": true } }' \
  http://localhost:9200/_cluster/settings 

# Trigger deprecation warning
curl -u $AUTH http://localhost:9200/_flush/synced?pretty

# Show indexed deprecation messages
curl -u $AUTH http://localhost:9200/logs-deprecation-elasticsearch/_search?pretty

elasticmachine · 2020-08-24T15:15:30Z

Pinging @elastic/es-core-infra (:Core/Infra/Logging)

.../src/main/java/org/elasticsearch/xpack/deprecation/logging/DeprecationIndexingComponent.java

…cation-logs

pugnascotia · 2020-09-01T13:21:32Z

@elasticmachine run elasticsearch-ci/packaging-sample-windows

.../src/main/java/org/elasticsearch/xpack/deprecation/logging/DeprecationIndexingComponent.java

...n/src/main/java/org/elasticsearch/xpack/deprecation/logging/DeprecationIndexingAppender.java

jakelandis

looking good. a couple of comments and a PR to this PR pugnascotia#3 (to ensure using the latest testing plugins)

update gradle config for new testing plugins

jakelandis

tested locally and worked great! LGTM

pugnascotia · 2020-09-02T16:02:33Z

server/src/main/java/org/elasticsearch/common/logging/DeprecatedMessage.java

+            .field("data_stream.type", "logs")
+            .field("data_stream.datatype", "deprecation")
+            .field("data_stream.namespace", "elasticsearch")
+            .field("ecs.version", ECS_VERSION)


@jakelandis how do these look to you?

pgomulka

LGTM,
we would need to update beats test samples (with new fields). I can help with this.

Backport of elastic#61484. Closes elastic#46106. Implement a new log4j appender for deprecation logging, in order to write logs to a dedicated data stream. This is controlled by a new setting, `cluster.deprecation_indexing.enabled`.

ruflin · 2020-09-24T08:24:47Z

server/src/main/java/org/elasticsearch/common/logging/DeprecatedMessage.java

        return new ESLogMessage(messagePattern, args)
+            .field("data_stream.type", "logs")
+            .field("data_stream.datatype", "deprecation")


@pugnascotia This should be data_stream.dataset to be aligned with the indexing strategy.

I would also propose to keep the namespace as default and use deprecation.elasticsearch as the dataset name. Only important thing is that the dataset does not contain a -.

@ruflin so would we have the following, then?

.field("data_stream.dataset", "default") .field("data_stream.namespace", "deprecation.elasticsearch")

No, the other way around:

.field("data_stream.dataset", "deprecation.elasticsearch") .field("data_stream.namespace", "default")

Thanks, I'll get that changed 👍

Follow-up to #61484.

Backport of #62855. Follow-up to #61484.

pugnascotia added >enhancement :Core/Infra/Logging Log management and logging utilities v8.0.0 v7.10.0 labels Aug 24, 2020

pugnascotia requested review from rjernst and jakelandis August 24, 2020 15:15

elasticmachine added the Team:Core/Infra Meta label for core/infra team label Aug 24, 2020

pugnascotia mentioned this pull request Aug 24, 2020

Write deprecation logs to a data stream #58924

Closed

Allow deprecation logs to be indexed

126f0ca

pugnascotia force-pushed the 46106-index-deprecation-logs branch from 9125b65 to 126f0ca Compare August 25, 2020 10:28

jakelandis reviewed Aug 27, 2020

View reviewed changes

.../src/main/java/org/elasticsearch/xpack/deprecation/logging/DeprecationIndexingComponent.java Outdated Show resolved Hide resolved

pugnascotia added 2 commits September 1, 2020 12:12

Merge remote-tracking branch 'upstream/master' into 46106-index-depre…

c9da4af

…cation-logs

Reconfigure bulk processor after review

e8429ab

jakelandis reviewed Sep 1, 2020

View reviewed changes

.../src/main/java/org/elasticsearch/xpack/deprecation/logging/DeprecationIndexingComponent.java Show resolved Hide resolved

update gradle for new testing plugins

e3c6c82

jakelandis reviewed Sep 1, 2020

View reviewed changes

...n/src/main/java/org/elasticsearch/xpack/deprecation/logging/DeprecationIndexingAppender.java Show resolved Hide resolved

jakelandis reviewed Sep 1, 2020

View reviewed changes

Merge pull request #3 from jakelandis/46106-index-deprecation-logs-jake

035bd6e

update gradle config for new testing plugins

jakelandis approved these changes Sep 2, 2020

View reviewed changes

Include more fields in DeprecatedMessage

e4ad389

pugnascotia commented Sep 2, 2020

View reviewed changes

pgomulka approved these changes Sep 3, 2020

View reviewed changes

pugnascotia merged commit dce2ef9 into elastic:master Sep 3, 2020

pugnascotia deleted the 46106-index-deprecation-logs branch September 3, 2020 13:50

ruflin reviewed Sep 24, 2020

View reviewed changes

pugnascotia mentioned this pull request Sep 24, 2020

Tweak the ECS fields in DeprecatedMessage #62855

Merged

pugnascotia added a commit that referenced this pull request Sep 24, 2020

Tweak the ECS fields in DeprecatedMessage (#62855)

d0c0ca1

Follow-up to #61484.

pugnascotia added a commit that referenced this pull request Sep 24, 2020

Tweak the ECS fields in DeprecatedMessage (#62855)

7771d8b

Backport of #62855. Follow-up to #61484.

pugnascotia mentioned this pull request Oct 19, 2020

Support hidden data streams #63889

Closed

Mpdreamz mentioned this pull request Nov 16, 2020

7.10.1 Meta Ticket elastic/elasticsearch-net#5096

Closed

61 tasks

stevejgordon mentioned this pull request Dec 17, 2020

7.11.0 Meta Ticket elastic/elasticsearch-net#5198

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

alisonelizabeth mentioned this pull request Aug 10, 2021

Enable deprecation log indexing by default #76292

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write deprecation logs to a data stream #61484

Write deprecation logs to a data stream #61484

pugnascotia commented Aug 24, 2020

elasticmachine commented Aug 24, 2020

pugnascotia commented Sep 1, 2020

jakelandis left a comment

jakelandis left a comment

pugnascotia Sep 2, 2020

jakelandis Sep 3, 2020

pgomulka left a comment

ruflin Sep 24, 2020

pugnascotia Sep 24, 2020

ruflin Sep 24, 2020

pugnascotia Sep 24, 2020

Write deprecation logs to a data stream #61484

Write deprecation logs to a data stream #61484

Conversation

pugnascotia commented Aug 24, 2020

elasticmachine commented Aug 24, 2020

pugnascotia commented Sep 1, 2020

jakelandis left a comment

Choose a reason for hiding this comment

jakelandis left a comment

Choose a reason for hiding this comment

pugnascotia Sep 2, 2020

Choose a reason for hiding this comment

jakelandis Sep 3, 2020

Choose a reason for hiding this comment

pgomulka left a comment

Choose a reason for hiding this comment

ruflin Sep 24, 2020

Choose a reason for hiding this comment

pugnascotia Sep 24, 2020

Choose a reason for hiding this comment

ruflin Sep 24, 2020

Choose a reason for hiding this comment

pugnascotia Sep 24, 2020

Choose a reason for hiding this comment