Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor ESLogMessage to not define fields upfront #46702

Merged
merged 17 commits into from
Oct 10, 2019

Conversation

pgomulka
Copy link
Contributor

@pgomulka pgomulka commented Sep 13, 2019

The current behaviour expect all additional fields to be defined in ESLogMessage. This makes adding new custom ESLogMessages harder.
This refactoring is changing this behaviour to only define fields in a properties file when overriding a standard field for instance message. All other custom fields do not need to be defined upfront in property file.
As an example an AllocationService log statement Cluster health status changed from [{}] to [{}] (reason: [{}] will have additional 3 fields in json logs.

{
  "type": "server",
  "timestamp": "2019-09-27T13:22:19,660+02:00",
  "level": "INFO",
  "component": "o.e.c.r.a.AllocationService",
  "cluster.name": "distribution_run",
  "node.name": "node-0",
  "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[twitter45][1]]]).",
  "cluster.uuid": "1WUx41iLT06b2VXRAgm3nw",
  "node.id": "4lYUM1SDSx2HwHFKB_SJWQ",
  "reason": "shards started [[twitter45][1]]",
  "current.health": "YELLOW",
  "previous.health": "RED"
}

relates #46119

@pgomulka pgomulka added the :Core/Infra/Logging Log management and logging utilities label Sep 13, 2019
@pgomulka pgomulka self-assigned this Sep 13, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@pgomulka pgomulka changed the title Refactor ESLogMessage to not defining fields upfront WIP Refactor ESLogMessage to not defining fields upfront Sep 13, 2019
@pgomulka pgomulka added the WIP label Sep 13, 2019
@pgomulka
Copy link
Contributor Author

@pgomulka test

@pgomulka pgomulka changed the title WIP Refactor ESLogMessage to not defining fields upfront Refactor ESLogMessage to not defining fields upfront Sep 13, 2019
@pgomulka pgomulka changed the title Refactor ESLogMessage to not defining fields upfront Refactor ESLogMessage to not define fields upfront Sep 13, 2019
@pgomulka pgomulka removed the WIP label Sep 13, 2019
@pgomulka
Copy link
Contributor Author

@elasticmachine update branch

@@ -72,4 +81,9 @@ public static String asJsonArray(Stream<String> stream) {
.map(ESLogMessage::inQuotes)
.collect(Collectors.joining(", ")) + "]";
}

public static ESLogMessage message(String messagePattern, Object... args){
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we eventually collect the arguments using .with( we could probably structure this in a way that avoids repeating them. It's probably worth it even if we end up using a builder.
I'm thinking something similar to: https://github.com/logstash/logstash-logback-encoder#event-specific-custom-fields

@@ -72,4 +81,9 @@ public static String asJsonArray(Stream<String> stream) {
.map(ESLogMessage::inQuotes)
.collect(Collectors.joining(", ")) + "]";
}

public static ESLogMessage message(String messagePattern, Object... args){
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have tooling to check that loggers are used correctly which I think is now circumvent-able with this method.
Most probably the tooling will have to be updated regardless of how this API is implemented.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you probably mean ESLoggerUsageTests ? I think it should be fine as it was already extended for subclass usages. But I will have a look to make sure it passes fine

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I do mean the ESLoggerUsageChecker I don't know enough about it's implementation to be able to tell if it will work with the builders or not.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pgomulka Could we maybe add an explicit test to ESLoggerUsageChecker that uses ESLogMessage.of just to be clear that this works ?

@pgomulka
Copy link
Contributor Author

pgomulka commented Oct 7, 2019

@elasticmachine update branch

@pgomulka
Copy link
Contributor Author

pgomulka commented Oct 7, 2019

@elasticmachine run elasticsearch-ci/2

@pgomulka pgomulka requested a review from alpar-t October 7, 2019 10:43
sb.append(Chars.DQUOTE).append(':').append(Chars.DQUOTE);
start = sb.length();
sb.append(getIndexedReadOnlyStringMap().getValueAt(i).toString());
// ParameterFormatter.recursiveDeepToString(getIndexedReadOnlyStringMap().getValueAt(i), sb, null);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we remove this commented code ?

@pgomulka
Copy link
Contributor Author

pgomulka commented Oct 8, 2019

@elasticmachine run elasticsearch-ci/2

@pgomulka
Copy link
Contributor Author

pgomulka commented Oct 8, 2019

@elasticmachine run elasticsearch-ci/packaging-sample-matrix

Copy link
Contributor

@alpar-t alpar-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pgomulka
Copy link
Contributor Author

pgomulka commented Oct 9, 2019

@elasticmachine run elasticsearch-ci/2

@pgomulka
Copy link
Contributor Author

pgomulka commented Oct 9, 2019

@elasticmachine run elasticsearch-ci/packaging-sample

@pgomulka pgomulka merged commit a4a7967 into elastic:master Oct 10, 2019
@pgomulka pgomulka mentioned this pull request Nov 14, 2019
14 tasks
pgomulka added a commit that referenced this pull request Mar 19, 2020
Slow log's routing value when null was causing ESLogMessage.asJson
method to throw Null Pointer Exception. This should be fixed in
ESLogMessage as well as prevent passing that key at all.
This only happens in 8 because of previous refactoring #46702
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/Logging Log management and logging utilities >refactoring v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants