Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Elasticsearch JSON logs ECS compliant #47105

Merged
merged 63 commits into from
Mar 31, 2020

Conversation

pgomulka
Copy link
Contributor

@pgomulka pgomulka commented Sep 25, 2019

Updating log4j2 configuration to use EcsLayout for our JSON logs https://github.com/elastic/ecs-logging-java instead of previously used ESJsonLayout
We still test against 7.x ESJsonLayout to make sure users won't break because of this change. Because of this the previous infrastructure still stays.
This might be worth discussing if we can break them with this. I personally feel that introducing JSON in 7 and then breaking users in 8 might be annoying.

Notable differences compared to 7.x ES JSON:

  1. Stacktrace's message won't be multiline (when for instance a JSON is used as a message)-
  2. Stacktrace itself also is single line
  3. Prefix/markers end up in tags field - previously markers were prefix in a message field
  4. Packages are no longer abbreviated - ECS expects full logger names, previously they were abbreviated as per %-25c{1.} for instance o.e.n.Node
  5. %node_id and %cluster_id are separate converters (previously node_and_cluster_id which not usable with ecs-java-layout lib
  6. FIeld name changes
  • level -> log.level
  • component - log.logger

closes #46119

@pgomulka pgomulka added the :Core/Infra/Logging Log management and logging utilities label Sep 25, 2019
@pgomulka pgomulka self-assigned this Sep 25, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@pgomulka
Copy link
Contributor Author

pgomulka commented Oct 10, 2019

@elasticmachine update branch

@elasticmachine
Copy link
Collaborator

merge conflict between base and head

@pgomulka
Copy link
Contributor Author

ok to test

build.gradle Outdated
@@ -118,6 +118,9 @@ subprojects {
':distribution:tools:plugin-cli',
':qa:os',
':qa:wildfly',
':qa:die-with-dignity',
':qa:logging-config',
':qa:evil-tests',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed this to an opt-out list in gradle/formatting.gradle, so you should remove these project paths from the list in that file.

Copy link
Contributor

@pugnascotia pugnascotia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pgomulka
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/bwc

@pgomulka pgomulka merged commit 979c33b into elastic:master Mar 31, 2020
pgomulka added a commit that referenced this pull request Oct 20, 2021
Adds breaking change docs for #47105. (ECS layout and plaintext log files removal)

Relates to #46119
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Logging] Support for ECS and custom event tags
7 participants