Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Command Router: Memory leak concerning Kafka Metrics (with many added/deleted topics) #3016

Closed
calohmn opened this issue Jan 6, 2022 · 2 comments
Closed

Command Router: Memory leak concerning Kafka Metrics (with many added/deleted topics) #3016

calohmn opened this issue Jan 6, 2022 · 2 comments
Assignees
@calohmn
Labels
C&C Command and Control
Milestone
1.11.0

Comments

@calohmn
Copy link
Contributor

calohmn commented Jan 6, 2022

On a Hono deployment using Kafka messaging, where many (test) tenants get created and removed again, the memory usage of the Command Router component steadily increases, so that after a certain time period (one week in our case) the pod gets OOM-killed.

Comparing heap dumps has revealed that the added org.apache.kafka.common.metrics.KafkaMetric objects related to the tenant topics cause the memory increase. They are currently not getting removed after the topics get deleted.
@calohmn calohmn added the C&C Command and Control label Jan 6, 2022
@calohmn calohmn added this to the 1.11.0 milestone Jan 6, 2022
@calohmn calohmn self-assigned this Jan 6, 2022
@calohmn
Copy link
Contributor Author

calohmn commented Jan 6, 2022

Micrometer 1.7.6 contains a fix - see micrometer-metrics/micrometer#2843.

@calohmn
Copy link
Contributor Author

calohmn commented Jan 10, 2022

Besides the issue in Micrometer mentioned above, there is also an issue in the Kafka client itself.
For topics that a (wildcard-topic) consumer was once subscribed to but that since have been deleted, the topic-related metrics don't get removed.
This refers to the (fetch manager) metrics:

  • fetch-size-avg, fetch-size-max, bytes-consumed-rate, bytes-consumed-total (all handled by the "bytes-fetched" Sensor)
  • records-per-request-avg, records-consumed-rate, records-consumed-total (all handled by the "records-fetched" Sensor)

@calohmn calohmn mentioned this issue Jan 10, 2022
Closed
calohmn added a commit to bosch-io/hono that referenced this issue Jan 13, 2022
@calohmn
[eclipse-hono#3016] Use newer Micrometer version.
efc6141 
Micrometer 1.7.6 contains a fix regarding
a Kafka metrics memory leak.

Signed-off-by: Carsten Lohmann <[email protected]>
@calohmn calohmn mentioned this issue Jan 13, 2022
Merged
calohmn added a commit to bosch-io/hono that referenced this issue Jan 13, 2022
@calohmn
[eclipse-hono#3016] Remove consumer metrics for deleted topics.
d7afad0 
Signed-off-by: Carsten Lohmann <[email protected]>
@calohmn calohmn mentioned this issue Jan 13, 2022
Merged
calohmn added a commit that referenced this issue Jan 13, 2022
@calohmn
[#3016] Use newer Micrometer version.
94480b7 
Micrometer 1.7.6 contains a fix regarding
a Kafka metrics memory leak.

Signed-off-by: Carsten Lohmann <[email protected]>
calohmn added a commit that referenced this issue Jan 13, 2022
@calohmn
[#3016] Remove consumer metrics for deleted topics.
f3ac1a6 
Signed-off-by: Carsten Lohmann <[email protected]>
@calohmn calohmn closed this as completed Jan 13, 2022
@calohmn calohmn mentioned this issue Nov 27, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@calohmn calohmn

Labels
C&C Command and Control
Projects
None yet
Milestone
1.11.0
Development

No branches or pull requests
1 participant
@calohmn