Sankey Visualizer - Event Flow

[Aergonus]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Instead of building out a manual flow graph and parsing out buffer/dropped events from internal metrics, I want to be able to generate a graph to view event flows at a glance.

Attempted Solutions

vector graph is great for tracing the topology and internal metrics show how much volume is being processed/dropped
but what I really want is an easy way to inflows and outflows for each component.

Proposal

vector sankey events
vector sankey event_bytes
outputs instantaneous event counts/bytes flows in link format

This would use values like
vector_component_sent_events_total, vector_component_received_events_total, vector_buffer_received_events_total, vector_buffer_sent_events_total

References

Simple link format would be ingestible by https://observablehq.com/@mbostock/flow-o-matic and would be 90% of the way to fit https://visactor.io/vchart/guide/tutorial_docs/Chart_Types/Sankey

Version

n/a

[pront]

Hi @Aergonus, this is a good idea. It is unclear at the moment if we can develop this but if you have an implementation plan, please share it. Maybe someone from the community will be motivated to pick this up.

[Aergonus]

I think the implementation is pretty straightforward. With some vector subcommand, it will output link flows for some external sankey visualizer after pulling metrics like how vector top does.

I was curious if there were plans to do something like this since the graphql release blog post mentions

An exciting new web UI for visualising and interacting with topology.
Component-specific types and metrics that can provide much deeper and more granular insight into what’s happening in your topology

[pront]

I was curious if there were plans to do something like this since the graphql release blog post mentions

Hi @Aergonus, thanks for the reply.

A few things to keep in mind; we want to eventually remove GraphQL completely from Vector. We won't be prioritizing any UI work for a while since we have some other high priority items in the backlog. But I do appreciate their utility and would love to see more visualization tools e.g. https://www.otelbin.io/.

[Aergonus]

we want to eventually remove GraphQL completely from Vector.

I'm assuming vector <tap,top,graph and the new sankey> options would be here to stay despite the GraphQL removal

I'll try to get a PR for this sankey idea after the coming holiday (will not be as nice as the otelbin ui :))

[pront]

I'm assuming vector <tap,top,graph and the new sankey> options would be here to stay despite the GraphQL removal

Absolutely, that should be just a tech debt cleanup task without any user impact.

I'll try to get a PR for this sankey idea after the coming holiday ?

Nice 👍