Task timelines

[chadbrewbaker]

I didn't see any code for visualizing task timelines. Does this exist, and if not where in the codebase would you add in such a feature?

Scenario:
The app server creates a UUID for a task, and the UUID gets propagated with work units for that task across the cluster and shows up in logs. The visualization would then give a timeline similar to what you see in https://developer.chrome.com/devtools/docs/timeline .

--Updates--
Jumpshot is the old tool for MPI clusters, you would need a log stash to SLOG conversion script: http://www.mcs.anl.gov/research/projects/perfvis/software/viewers/index.htm

Twitter has the http://twitter.github.io/zipkin/ project.

There is one log stash to Zipkin tool, https://github.com/sidnei/athwart

More than a few lists already talking about adding Zipkin features to Kibana:

• https://github.com/rashidkpc/Kibana/issues/371

*https://www.linkedin.com/groups/Distributed-Tracing-logging-Integrating-Kibana-3393294.S.5862019444968013827

[elvarb]

In a sense you can do this with the histogram tool.

Create one query for each system, that way the histogram will show each one with a different color. Then add a filter to include the UUID. After that the histogram will show all log entries from each system.

For added feature I suggest you go over the logstash measure filter. Allows you to track the time it takes for tasks to finish.

edit:
Looking at the screenshot from the linkedin link I see what you mean. This panel along with a timerange field would be quite useful.

[nearsemiring]

A histogram of run times isn't enough. You need them temporally ordered to see which tasks are holding up which other tasks. http://users.ece.utexas.edu/~adnan/pike.html

Google's timeline visualization chart may be a temporary solution until Kibana implements timelines, https://developers.google.com/chart/interactive/docs/gallery/timeline.

[rkrol]

+1

I implemented this kind of widget for kibana 6 months ago.
It worked pretty well in my context, but this was clearly not perfect and not reusable at all :

• I had to extract a duration/responseTime on a different field depending on log types. These field names were hardcoded. For exemple, the field name is not the same if the log comes from apache, nginx, or application layer. So I think we need to implement a settings form to be able to extract different field names depending on log type
• this was implemented with d3.js, so this was not uniform with other widgets (highcharts if I'm right)

I am also very interested by this feature, and I can help going forward on this

[elvarb]

@rkrol The right way would be to use Logstash mutate filter to rename all of those fields to the same fieldname. In the panel in Kibana you should be able to define what fieldname it should use.

[rkrol]

@elvarb +1 but unfortunately we don't use logstash in my organization :(

[pshemass]

any update on that?

[telune]

+1
This would be really useful. Basically it's a horizontal stacked bar chart, but whose values change based on document timestamps + other relevant fields.

[khoubyari]

+1 for taking advantage of Zipkin generated data in Kibana

[Crazyigor1987]

Do you know "Kibi"? The friendly Kibana Fork?

http://siren.solutions/kibi/

They have a Timeline Visualisation right there as a Plugin:

https://github.com/sirensolutions/kibi_timeline_vis

[stephane-dereppe]

Any update on that topic?

[ksilin]

+1

[ninja-inc]

+1

[welovelain]

+1

[chtlp]

+1

[timductive]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.