Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move libbeat input queue metrics to metrics data streams #3334

Open
joshdover opened this issue Aug 30, 2023 · 2 comments
Open

Move libbeat input queue metrics to metrics data streams #3334

joshdover opened this issue Aug 30, 2023 · 2 comments
Labels
Team:Elastic-Agent Label for the Agent team

Comments

@joshdover
Copy link
Contributor

While working on #3315, I noticed that we only include the input / pipeline metrics on log messages, rather than on the metric data streams. We should expose this data on metrics instead in order to optimize storage costs via TSDB and avoid sparse fields on logs.

This will also help users find these metrics more easily. An example metrics are the monitoring.metrics.libbeat.pipeline.* ones, but there may be others that may be useful as well.

As part of this, we may also want to remove the "Non-zero metrics in the last 30s" logs from Beats when running under Agent and instead rely on metrics collection.

@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

@cmacknz
Copy link
Member

cmacknz commented Aug 30, 2023

As part of this, we may also want to remove the "Non-zero metrics in the last 30s" logs from Beats when running under Agent and instead rely on metrics collection.

The nice thing about this is it keeps metrics in the diagnostics as part of the logs (with history). So I'd want to keep this, but we could stop shipping them to Fleet. This is something we can do with a drop processor in the monitoring filestream instance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

No branches or pull requests

3 participants