Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

log: introduce more metrics into log pkg for network logging #72453

Closed
knz opened this issue Nov 4, 2021 · 2 comments · Fixed by #111126
Closed

log: introduce more metrics into log pkg for network logging #72453

knz opened this issue Nov 4, 2021 · 2 comments · Fixed by #111126
Assignees
Labels
A-logging In and around the logging infrastructure. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-sre For issues SRE opened or otherwise cares about tracking.

Comments

@knz
Copy link
Contributor

knz commented Nov 4, 2021

This is follow up work to #70330.

Since #106607, we now have the ability to add metrics to the log package.

While the original PR adds a metric to track fluentbit connection errors, we should add additional metrics such as:

  1. When the output buffer becomes full, the new buffer sink implementation chooses to drop messages instead of blocking the log call (this is a feature). Count these.
  2. Count of log messages flowing through the log package.
  3. Error counts
  4. ???

Use the new infra to add useful new metrics to pkg/util/log, with a focus on improving observability around network logging.

Jira issue: CRDB-11161

@knz knz added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-logging In and around the logging infrastructure. labels Nov 4, 2021
knz added a commit to rauchenstein/cockroach that referenced this issue Nov 4, 2021
knz added a commit to rauchenstein/cockroach that referenced this issue Nov 4, 2021
@joshimhoff joshimhoff added the O-sre For issues SRE opened or otherwise cares about tracking. label Jun 3, 2022
@joshimhoff
Copy link
Collaborator

This is great!

CC is going to release some features soon which increase our need for highly reliable logging. For example, we will release a feature that ships CRDB logs to customers directly.

I bet this is a pretty small work item. If no DB eng time for it, I think we could have an SRE do it too. Do we plan to execute on this item soon?

@abarganier
Copy link
Contributor

This was a bit tricky to get set up originally due to dependency cycles, but as of #106607, this should be much easier to do!

@abarganier abarganier changed the title log: report the count of dropped message when messages get dropped during buffering log: introduce more metrics into log pkg for network logging Sep 11, 2023
@craig craig bot closed this as completed in 422b70f Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-logging In and around the logging infrastructure. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-sre For issues SRE opened or otherwise cares about tracking.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants