Support JSON logging formatting

[markwalkom]

There was discussion on the Logstash list about some grok patterns for Elasticsearch logs to allow easier ingestion and someone suggested that it'd be better if Elasticsearch supported JSON output using https://github.com/logstash/log4j-jsonevent-layout.

Is it possible to do this and have it as an optional setting in the logging config?

[markwalkom]

Had a chat with a customer today and they rolled their own for this using log4j-jsonevent and a few other libraries with some config changes.

It'd be great if we could offer this out of the box.

[samcday]

Here's what we're doing:

• Drop log4j-jsonevent-layout jar-with-deps into ES lib/ dir.
• Setting appender.file.layout.type to net.logstash.log4j.JSONEventLayoutV1 in logging.yml

After that, the ES logfile will be spitting out JSON, and order is restored to the universe :)

[samcday]

If you think a PR that adds log4j-jsonevent-layout as a compile dep in ES, with a json alias for the layout.type (similar to the pattern one) would be accepted, I'm happy to raise one.

[TinLe]

+1

[markwalkom]

Any way we could get this into an early 2.X release?

[faxm0dem]

👍

[miah]

We've hit 2.0 release. ES is still on log4j 1.2, which means we still have to follow this hack workaround.

[miah]

Also should point out that the logstash jsonevent jar doesn't work with ES 2.0 in my experience. Logs are produced (in our non-json aka console logs) until the node has finished starting, then all logging ceases.

[jeremydonahue]

+1 for this feature.

@miah I tested the logstash jsonevent jar with ES 2.0 today and was unable to get it working at all. I tried building from source and a jar downloaded from maven.org. If I understand correctly, log4j version 2 supports json as a layout option. How much work is involved with upgrading log4j?

[kimchy]

I think that "just doing json layout" is not enough. My suggestion is to enhance our custom logging format to allow for parameters, something like this event happened at {timestamp:date} and took {timeInMillis:duration} (just an example). Then, we can produce a proper structured logging that makes any structured format more usable, specifically if we create a common schema around it (for duration, ...). Building a JSON layout on top of it using our logging abstraction will then be simpler.

Note, I am not a fan of the concept of MDC and such, I think this will be a much better solution.

[faxm0dem]

@kimchy can you elaborate on why you don't like MDC? I'm a bit surprised this is coming from someone who developed a tool that is used a lot for indexing structured logs often containing MDC ;-)

[jeremydonahue]

@kimchy As long as building a JSON layout on top of the logging abstraction properly escapes data objects (like hashes, multi-line exceptions, etc) I think that sounds great.

[miah]

I agree that JSON log format may not be enough, but I ask this question. When will this other solution you propose arrive in an Elasticsearch release? How much effort will be required to get to that point?

Until this feature is actually spec'd, developed, and released we are still unable to get logs from Elasticsearch into Elasticsearch without some specialized grok/lpeg.

Can we just get a JSON output?

[samcday]

What @kimchy is describing actually sounds quite awesome, and is exactly what logstash-logback-encoder supports with StructuredArgument.

Of course, adopting something like this would imply Elasticsearch would ditch log4j in favour of Logback. If that were to be done though, then it would be pretty straightforward to support native JSON output, and steadily go through existing logs and swap from the current de-facto standard of wrapping interesting stuff in square brackets, and use StructuredArgs instead.

[faxm0dem]

If I can add my two bitcoins using slf4j would be even more awesome

[ESamir]

+1

[damianpfister]

+1 to JSON logging

[bosinm]

+1

[mcmesser]

+1

[jeremykoerber]

+1
Any updates on this?

[evanvolgas]

Relates to #17697 .... also curious about updates on this. JSON logging for ES would be extremely nice and, imo, consistent with the arguments Elastic made for self monitoring systems

[Xhanti]

+1

[driegel]

Well, I would like to add my thoughts:

1. Drop the YAML file that Elasticsearch uses for logging configuration. For example, using async appenders in Log4j 1.2 requires an XML config file. Asynchronous logging is important for centralized logging in the cluster.
2. Yes, we do need an Logstash Event Layout. It should use the same JARs that Elasticsearch (and Logstash) already use for generating JSON (i.e. Jackson!!!)

[alex-voigt]

+1
any news?

[dakrone]

JSON output for logging will be available in 5.0 with the move to Log4j2 (see #20235), which allows someone to use their own log4j2.properties file to configure logging.

This, however, does not move all logs to be structured, so you will get only the structure that log4j2 applies, rather than any special fields ES would be adding.

[faxm0dem]

@dakrone any technical reason not to use SLF4J instead?

[dakrone]

@faxm0dem check out the discussion in #17697 for more info on the backstory for it.

[hartfordfive]

👍 Any chance this will be added in the 5.0 release or shortly after that?

[MorrieAtElastic]

Users continue to request an out-of-the-box option for JSON-formatted elasticsearch logs. Is this something that will definitely be provided? If so, when? Thanks

[faxm0dem]

https://twitter.com/aphyr/status/454802547291615232

[tomcallahan]

@elastic/es-core-infra

[vladimirdolzhenko]

ping @elastic/es-core-infra

Is there any plan to support it ?

[radzak]

I've noticed ESJsonLayout added recently, so I think it will be out with the next release?

Although I'm really sad I can't get Log4j2 JsonLayout working. I'm getting Unable to invoke factory method in class class org.apache.logging.log4j.core.layout.JsonLayout for element JSONLayout. java.lang.reflect.InvocationTargetException as in #22103 :'(

[faxm0dem]

it's quite ironic that ES can't output structured logs (anymore) ;-)

[pgomulka]

@radzak you are right, the JSON logging will be released with 7.0. We could not use log4j2 JSONLayout for the reasons mentioned in a #22103 (security requirements by jackson dependency )

@faxm0dem How do you mean? You can still provide your own structured pattern or use the previous logging configuration.

Happy to hear more feedback from you

[pgomulka]

closing as this was implemented in #36833