-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"event not processed by enough 'sink' nodes" error affecting vault.audit.log_request_failure metric #25549
Comments
FYI this issue can be worked around by setting This will prevent us from upgrading to 1.16. |
Hi @lboynton thanks for the report. We've got this on our radar to look at internally, but cannot commit to saying if/when it will be addressed. It's understandable frustration that if Vault didn't do it before, but does it now (fail this way with server logs and telemetry) - that's not good. But also an awkward one as the audit system isn't incorrect in what it's reporting, it cannot audit due to the context timeout. This issue report (and everything connected to it) has been linked up internally, so when we do have progress to report you'll be aware. |
Hello. We are also facing this same issue without It seems that the new eventlogger does a lot more requests, causes higher latency and fails with the |
I guess #26616 is a potential fix |
Just an update for folks, we're actively looking at this one at the moment, in a few different ways... One or more of those might end up making it out into the wild. Thanks for the patience. |
OK this #26616 should close the issue. We have another ticket to discuss internally which is linked which may show up later. 😄 |
Describe the bug
After upgrading to Vault 1.15.5 we are still observing "event not processed by enough 'sink' nodes" errors in our Vault log. #23871 has partially fixed the problem, however it only addresses audit responses and not audit requests. There is a similar issue - #24376, but it is in a "Closed" state, and nobody answers the questions asked.
Is it possible to ignore the error and avoid incrementing the vault_audit_log_request_failure metric? It is quite an important metric, so we don't want to disable an alert for it.
To Reproduce
STR provided in #23871
Expected behavior
vault.audit.log_request_failure metric is not incremented in this case
Environment:
vault status
): 1.15.5vault version
): 1.15.5The text was updated successfully, but these errors were encountered: