Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Obsolete Kafka Producer metrics should get removed to prevent memory leak #3017

Closed
calohmn opened this issue Jan 10, 2022 · 3 comments · Fixed by #3038
Closed

Obsolete Kafka Producer metrics should get removed to prevent memory leak #3017

calohmn opened this issue Jan 10, 2022 · 3 comments · Fixed by #3038
Assignees
Milestone

Comments

@calohmn
Copy link
Contributor

calohmn commented Jan 10, 2022

For Kafka producers concerning tenant-specific topics, topic-specific metrics currently don't get removed when a tenant got deleted.
The metrics are:
byte-rate, byte-total, compression-rate, record-error-rate, record-error-total, record-retry-rate, record-retry-total, record-send-rate, record-send-total

This refers to the telemetry/event/command-response producers in the protocol adapters and the command-response producers in the Command Router.

Without removing the metrics, there is a memory leak if tenants continuously get added and deleted.
Relates to #3016.

@calohmn calohmn added this to the 1.11.0 milestone Jan 10, 2022
@sophokles73
Copy link
Contributor

We should be able to address this analogously to what we do for the AMQP sender links, right?

@calohmn calohmn changed the title Obsolete Kafka Producers should get closed to prevent memory leak Obsolete Kafka Producer metrics should get removed to prevent memory leak Jan 10, 2022
@calohmn
Copy link
Contributor Author

calohmn commented Jan 10, 2022

@sophokles73 Had to correct the issue title/text here: The memory leak is not about the producers themselves (one producer being used for all tenant-topics of a certain kind), but about the topic-related metrics stored in the producers.

@calohmn
Copy link
Contributor Author

calohmn commented Jan 13, 2022

A solution that removes the metrics when the tenant timeout mechanism (see #1188) is triggered would be feasible for the protocol adapters, but wouldn't work for the command router command response producer (the protocol adapter tenant timeout mechanism can't really be implemented in the command router).

Therefore I think the solution should be based on the tenant-deletion notifications introduced in #2837.

@calohmn calohmn self-assigned this Jan 13, 2022
calohmn added a commit to bosch-io/hono that referenced this issue Jan 19, 2022
Kafka producer metrics regarding the telemetry, event and
command-response topics of a tenant are removed when that
tenant got deleted.

Signed-off-by: Carsten Lohmann <[email protected]>
calohmn added a commit to bosch-io/hono that referenced this issue Jan 20, 2022
Kafka producer metrics regarding the telemetry, event and
command-response topics of a tenant are removed when that
tenant got deleted.

Signed-off-by: Carsten Lohmann <[email protected]>
calohmn added a commit to bosch-io/hono that referenced this issue Jan 20, 2022
Kafka producer metrics regarding the telemetry, event and
command-response topics of a tenant are removed when that
tenant got deleted.

Signed-off-by: Carsten Lohmann <[email protected]>
calohmn added a commit to bosch-io/hono that referenced this issue Jan 20, 2022
Kafka producer metrics regarding the telemetry, event and
command-response topics of a tenant are removed when that
tenant got deleted.

Signed-off-by: Carsten Lohmann <[email protected]>
calohmn added a commit that referenced this issue Jan 20, 2022
Kafka producer metrics regarding the telemetry, event and
command-response topics of a tenant are removed when that
tenant got deleted.

Signed-off-by: Carsten Lohmann <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants