Extract log event context

[mrdavidlaing]

When troubleshooting the cause of an ERROR log event, its really helpful to be able to see the log event context - i.e, which, say, 10 log events occured before and after the specific ERROR log event.

Loggly has a feature which does this quite nicely, as illustrated below

I have 2 questions concerning this:

1. Is it feasible to add a similar feature to Kibana?
2. Are there any existing plans to add said feature? (I'm happy to contribute development resources to assist or lead this if its possible)

Lastly, thanks!

[ramv]

I would love to have this feature too.

[aeneaswiener]

+1

[fragosti]

+1

[ajczapiewski]

+10

[stuart-warren]

When you expand a log and get the 'table' view by default, perhaps we could get another 'context' view.
This 'context' view might require another request, but essentially you could add which fields to filter on for the same value as the selected log (eg host, tags), a time period before and after (eg -1m, +10s) and which field to sort on.

Might be wise to open a separate window with a smaller (predefined?) dashboard with the given filters?

[kiranos]

+1

[OkkeKlein]

+1

[haagenhasle]

+1

[justyns]

I'd also like this feature. Is there already a similar workflow/workaround that people already use for this sort of thing?

Currently I search for the error in kibana, then change the filter to include only the file/server it came from and then just zoom in on the time around the error.

[tcostermans]

+1

[stbka]

+1

[marcelpanse]

+1

Why is this still not in Kibana?
It makes Kibana kind of useless to me, since I typically need to find something in the logs and then find the error preceding the log row, which is currently impossible.

[steveims]

+1

[markgilmore1322]

+1

[murphe]

+1

[spbever]

I will be working on this task in a forked repository for where I work. When I have it completed I will submit a pull request with how I have it implemented.

[spbever]

So I have not been able to come up with a "creative" enough solution that I think would be generic enough to pulled into the main Kibana source, but I will still post what my solution to this was and anyone will be free to copy that for their own use.

EDIT:
My solution can be found: https://github.com/spstapleton/kibana/tree/Kibana-275-Log-Context
I have added a field to logstash.json (contextMatchFields) that is an array of keys that the context filter will attempt to find related log entries by.

[stbka]

spstapleton this is really great! Thanks!

I`ve done some tests with this feature and found some things not working properly.

• All events in the context view have the same timestamp
• If I choose a field with special characters inside like '-' or '/' or ':' the search seems not to work, because no surrounding events can be found
• Sometimes, it seems like if "Time Variance" and "Max Context Lookup Results" do not fit to each other no results are found. For example: max lookups is 100 and t variance is 300 -> no results. max lookups is 1000 and t variance is 300 -> results are found

[spbever]

I can take a look into these in the next week or so and and let you know what I come up with. I think the special characters is an issue with not having the value uri encoded.

[spbever]

@tma-ka
I made a fix for the timestamp. The current timestamp filter only worked for the original events returned. I had to make a second filter that just filtered on text instead of an event.

Special Characters: Elastic Search by default will split text entries on certain special characters. For instance "te-st" would get two indexes by default, one for "te" and one for "st". Thus, when I have it do a context search for the term "te-st" it will not return any matching results. If you are predicting that the field you will be basing your context off of will have cases like that, I would suggest marking that field as 'not_analyzed' inside of elasticsearch.

For the last point. It could be the case where you are not doing the sorting based of off the default timestamp. If that is the case the time variance is not actually used and it relies only on the max lookups.

[stbka]

I fixed the special char problem by indexing a separate field as an identifier. Inside all special chars are removed so you have got only [A-Za-z0-9].
Unfortunately your timestamp fix is not working for me. I do still have one timestamp in each event.

Anyway, I think the idea and the implementation from user point of view is nice.
Rather the question is the technical implementation good enough for a permanent implementation in kibana.

I would like to use it but cannot while it is in a branch. At the latest when I am updating my kibana from trunk all is lost.

[ghost]

+1 lack of this feature has made Kibana practically useless.

[rifqifatih]

+1

[azitabh]

+1

[weltenwort]

As some might already have noticed by the auto-generated backlink above, I am currently working on a feature related to this enhancement request in PR #9198. Its goals have a large overlap with the requirements described in this issue and the comments.

I would like to invite anyone interested to take a look at the screenshots or even to try out the development version and provide feedback and suggestions. Please keep in mind that this initial PR is intended to only cover basic functionality with more use-case specific features to follow in later PRs.

[allenmchan]

I suspect people are starting to trickle back into work after the holidays.
Please provide feedback / suggestions for @weltenwort.

[larryboymi]

+1

[panol]

+1

[madhavajay]

+1

[acbox]

+1

[danhy87]

+1

[argaldo]

+1

[Jipos]

+1

[ZillaG]

+1

[yami12376]

+1

[roubles]

+1

[mbriggs]

+1

[epixa]

Just in case folks aren't watching it, the event context PR got merged this morning: #9198

This is a first step toward addressing the many different use cases that drive this ticket, but feedback is still welcome!

[9h87f497h4387b9]

It has been very exciting watching the PR progression. Thanks for the update!

[lisakaymera]

Just curious - has anyone used logtrail to address this issue at all? It seems to address the issue of log context. I didn't get to download to play around with it though because my kibana version is slightly different. Would love to hear people's experiences with it.

[wangzhaojiang]

+9

[stolsvik]

+1

I would be happy to have to define the contextual search parameters: For us, that would either "application name" and the timestamp. Or, a more refined, "application name"+"host name" and timestamp. Or "traceId" and timestamp. They are of use in different scenarios. Then it would just be a matter of somehow select a line, and hit one of the context targets in "Get lines around in context 'App', 'App+Host' or 'TraceId'".

[mohaseeb]

+1

[artisanhe]

+1

[mk2543]

+1

[weltenwort]

This is to let you know that the first stage of this feature has just been released as part of Kibana 5.4, together with the whole Elastic Stack.

I want to invite you to give it a try, maybe read the documentation, and let us know what you think:

• If you think you found a bug, please open a GitHub issue with your report.
• For enhancements you can also open a new GitHub issue to describe your use case.
• For questions, clarifications and everything else please feel free to start a thread on the Kibana forum.

And last but not least, we are hard at work adding filtering capabilities to the context view. You can follow the progress in #11466. As always, any and all feedback on that is appreciated and can only help to make Kibana better.