Logging libraries for ECS

[ruflin]

Most ECS data is fetched from log files. To make generation of ECS based logs easier, ECS could provide logging libraries or tools on top of the existing logging libraries for the different programming languages. This should make generation of ECS based logs much easier.

This is partially related to #280 as it could be used as a foundation for the logging libraries.

[tsg]

+1. Somewhat related, I think we should also standardise on some sort of Elastic or ECS protocol, which can be as simple as JSON over TCP/UDP, with the twist that the JSON respects ECS. This means we could have a Filebeat/Logstash input for this protocol, and then any 3rd parties that want to integrate with our stack can implement this.

[ruflin]

@tsg I like the idea, should we also open a separate issue for it to not mix discussions?

[webmat]

I see some merit in looking into this. I'm curious what form this would take concretely, in the usage of such a library. When doing structured logging, there's usually still a free form message (key message), then there's the meta-data, which falls in two categories. First is the metadata collected automatically by the library, which could benefit from being named according to ECS (e.g. PID). But I think in general, most of the metadata is specified upon emission of the log event, and is therefore named by the developer themselves.

How did you see the featureset offered by these logging libraries? Especially wrt the ad hoc key naming.

If we decide to implement this, I call dibs on implementing the Ruby library

The one thing we need to steer away from is actually shipping the logs. An application should never have to manage (or suffer from) problems in the logging pipeline. It shouldn't do the queueing, the retries, etc. Hopefully we're just talking about logging libraries that format a text logfile or a syslog payload.

[ruflin]

Here is an example logging library for go by @urso https://github.com/urso/ecslog

[urso]

This library combines a many concepts and separates between ECS and custom user defined fields.
It does not define the output or shipping part, but provides building blocks for serializing logs to json, ubjson, cbor, plain text, map[string]interface{} (yes, one can capture the log in an object and post-process). Capturing meta-data is optional for all logging backends.

It is based on a many experiments with a many other go libraries and some Serilog. The type safe part is based on generated code from the fields.yml files. Users can define their own type safe namespaces and even generate a template.json file if required. Everything is meant to be compatible to ECS.

I found the key difference between structured loggers is how the context is handled in all go libraries. I was hoping to do a meta-logger that can operate with other structured loggers, but unfortunately this didn't work out well. Yet I didn't find a library doing exactly as I wanted, so I did roll my own.

I will add an Readme with ideas and concepts used in the library the next days, so to have some ground for discussions.

For the curious check out this test application using the logger. What also sets this apart from other go based logging libraries is the custom message formatter can capture user meta-data: log.Info("info with %{custom} message", "user"). This will add {"fields": {"custom": "user"}} to the log event, plus it provides some integrations for dealing with errors (create structured trace of errors). One can also annotate errors with additional context via errx.With(ecs.Http.Request.Path("/file.txt")).Wrap(err, "failed to read %{file}", "file.txt"). This additional context will become part of the structured log.

[codebrain]

++ would be good to write adapters for .NET logging and metrics libraries (Serilog, NLog, AppMetrics...)

[felixbarny]

I've prototyped implementing a log4j2 EcsLayout which can write ECS-compatible NDJSON. The idea is that Filebeat reads a log file where each line contains an ECS-compatible JSON log event, which can more or less be directly sent to ES (after adding metadata).

Is there any corresponding Filebeat module in the works which could correctly set up the index pattern? Or should the default Filebeat index pattern be good enough?

During the implementation, I noticed several log-related fields are either missing or there's no clear mapping to an ECS field:

• The name of the logger, mostly the name of the class which defined the logger. Might map to event.dataset What field to use for Java / Jee logs #501
• The name of the thread in which the log event occurred. We have process.thread.id. I have created an issue to add process.thread.name: Add field: process.thread.name #507
• The message and stacktrace as a string representation. This might map to error.message. See also Using ECS to track the source of errors or exceptions #154 (comment)
• It would also be nice to have ECS fields for the file, function and line number of the log. Maybe log.file.name, log.file.function and log.file.line? There's already file.name, but the purpose might be a different one than logging.

I have mapped the MDC (Thread-local key/value pairs) to labels. In the Elastic APM Java agent, we are leveraging the MDC to associate log messages with the trace and transaction they belong to by setting trace.id and transaction.id MDC values. To facilitate log correlation (linking from APM to the Log UI and vice versa), the UI relies on the top-level trace.id id field, however. So I think it would be useful to have some kind of Filebeat module which makes some well-known or configurable labels top-level. Meaning that labels.trace.id would be converted to trace.id.

[ruflin]

The default index pattern should be "good enough" as it contains all the ECS fields.

For the event.dataset: This normally defines a unique data structure. Do different loggers mean different structures?

+1 on your function and line proposal. I wonder what we use for source code at the moment in APM? I guess this is kind of similar?

Not sure I fully understood the part around labels, probably need to read up on MDC ;-)

[felixbarny]

For the event.dataset: This normally defines a unique data structure. Do different loggers mean different structures?

Not necessarily. It's mostly used to know from which class the log stems from. But it does not have to be the class name. It can also be a user-defined string value.

+1 on your function and line proposal.

One problem is that it sort of clashes with the fields Filebeat adds. For example, Filebeat adds log.file.path which is the path of the log file and not of the source code which was responsible for emitting the log. So a separate prefix like origin or source might make more sense.

I wonder what we use for source code at the moment in APM? I guess this is kind of similar?

In APM, we refer to source code in the stack_trace object. See also #154 (comment).

[webmat]

I used to think that event.dataset may be a good place to put the logger name, but based on the example values you're giving here, I'm starting to doubt it. event.dataset should be more stable than this. It should be useful to get all of the logs of a certain kind for a given application, error logs, vs audit logs, vs access logs, etc.

To take Elasticsearch logs as an example, event.dataset will contain either elasticsearch.gc, elasticsearch.audit, elasticsearch.server, elasticsearch.deprecation or elasticsearch.slowlog. The more fine grained information you have in mind here would fit better in another field, perhaps .component. Some of the Elasticsearch logs have this information as well, and I see it as orthogonal to event.dataset.

As you outline in your comment from today, nesting source file metadata such as file name and line number doesn't make sense under log.file.*, as this isn't information about a log file. So it will not only clash with Filebeat's use of log.file.path for the log file an event was read from, but would also be confusing.

I like the idea of mapping the MDC to labels.*, as by definition, these are user-provided kv pairs. labels is the place for that. Now extracting/renaming some of those automatically will be tricky to message well, I would think. We may have a certain idea of what some of these fields mean, but there's no guarantee that people will use it in this way. I think it would make more sense to define a clear field set to actually track exceptions and stack traces. And perhaps also add a field (or small field set) specifically for event tracing. Then the libraries or loggers should conform to that, rather than sticking those values in user-provided MDC labels.

[felixbarny]

What speaks for event.dataset is that it works nicely with the default Logs UI settings:

But the fact that there is already an event.dataset for ES logs which is orthogonal to the logger name (event.component?) is almost a killer argument. So +1 for event.component from my side.

And perhaps also add a field (or small field set) specifically for event tracing. Then the libraries or loggers should conform to that, rather than sticking those values in user-provided MDC labels.

There's an open issue for that: #99

The problem is that the APM Java agent can only really leverage the MDC in order to annotate log events with the current trace.id. There's no other interface in logging libraries to set arbitrary meta-data for a thread. I'm working on ECS-aware plugins for java logging libraries which also come with a pre-defined set of default top-level labels so that users don't have to configure anything for the common cases.

[ruflin]

As the "log file" and the "origin of the event" can be very different, fully agree it should not be in the same place. The description of the origin of any APM event, be it a log or an error, should be described the same from an ECS perspective I'm thinking. I'm wondering if we need the concept of referring to source code in ECS (source already has an other meaning). Something like code.* that then can be reused. Don't have a great idea yet on how to call it :-(

[webmat]

@ruflin Agreed. Check my answer on #521 (comment)

[ruflin]

I suggest we open a separate issue / thread for this as this issue is initially about logging libraries?

[webmat]

With https://github.com/elastic/ecs-logging live for a little bit, I think we can close this issue, right?

Any unaddressed issue here should be brought up over there, or in the relevant language libs linked in the ecs-logging readme