-
Notifications
You must be signed in to change notification settings - Fork 137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Obsolete Kafka Producer metrics should get removed to prevent memory leak #3017
Comments
We should be able to address this analogously to what we do for the AMQP sender links, right? |
@sophokles73 Had to correct the issue title/text here: The memory leak is not about the producers themselves (one producer being used for all tenant-topics of a certain kind), but about the topic-related metrics stored in the producers. |
A solution that removes the metrics when the tenant timeout mechanism (see #1188) is triggered would be feasible for the protocol adapters, but wouldn't work for the command router command response producer (the protocol adapter tenant timeout mechanism can't really be implemented in the command router). Therefore I think the solution should be based on the tenant-deletion notifications introduced in #2837. |
Kafka producer metrics regarding the telemetry, event and command-response topics of a tenant are removed when that tenant got deleted. Signed-off-by: Carsten Lohmann <[email protected]>
Kafka producer metrics regarding the telemetry, event and command-response topics of a tenant are removed when that tenant got deleted. Signed-off-by: Carsten Lohmann <[email protected]>
Kafka producer metrics regarding the telemetry, event and command-response topics of a tenant are removed when that tenant got deleted. Signed-off-by: Carsten Lohmann <[email protected]>
Kafka producer metrics regarding the telemetry, event and command-response topics of a tenant are removed when that tenant got deleted. Signed-off-by: Carsten Lohmann <[email protected]>
Kafka producer metrics regarding the telemetry, event and command-response topics of a tenant are removed when that tenant got deleted. Signed-off-by: Carsten Lohmann <[email protected]>
For Kafka producers concerning tenant-specific topics, topic-specific metrics currently don't get removed when a tenant got deleted.
The metrics are:
byte-rate, byte-total, compression-rate, record-error-rate, record-error-total, record-retry-rate, record-retry-total, record-send-rate, record-send-total
This refers to the telemetry/event/command-response producers in the protocol adapters and the command-response producers in the Command Router.
Without removing the metrics, there is a memory leak if tenants continuously get added and deleted.
Relates to #3016.
The text was updated successfully, but these errors were encountered: