Skip to content
This repository has been archived by the owner on Mar 17, 2024. It is now read-only.

Timeouts and retries #3

Merged
merged 3 commits into from
Apr 17, 2019
Merged

Timeouts and retries #3

merged 3 commits into from
Apr 17, 2019

Conversation

seglo
Copy link
Owner

@seglo seglo commented Mar 11, 2019

Fixes #6

PR features:

  • Add kafka client timeout to helm chart
  • Use kafka client timeouts where ever we make calls to Kafka Consumer or Admin Client
  • Use backoff strategy restarting offset collection logic when transient runtime exceptions are encountered
  • Terminate when prometheus http server can't start (i.e. port can't be bound)

Copy link
Contributor

@deanwampler deanwampler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Added a few minor comments/suggestions.

.gitignore Outdated
*.iml
application.conf
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to add a linefeed at the end...

* Add `kafka-client-timeout` config.
* Tune retry and timeout logic of Kafka admin client and consumer
* Use backoff strategy restarting offset collection logic when transient runtime exceptions are encountered
* Terminate when prometheus http server can't start (i.e. port can't be bound)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider "Prometheus HTTP server..."

@@ -1,9 +1,9 @@
<configuration>
<variable name="KAFKA_LAG_EXPORTER_LOG_LEVEL" value="${KAFKA_LAG_EXPORTER_LOG_LEVEL:-INFO}" />
<variable name="KAFKA_LAG_EXPORTER_LOG_LEVEL" value="${KAFKA_LAG_EXPORTER_LOG_LEVEL:-DEBUG}" />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious if you intended to use DEBUG as the default or was this left over from some testing?

topicPartitions.map(tp => tp -> Measurements.Single(offsets.get(tp.asKafka).toLong,now)).toMap
}

/**
* Get last committed Consumer Group offsets for all group topic partitions. When a topic partition has no matched
* Consumer Group offset then a default offset of 0 is provided.
* @param groups A list of Consumer Groups to request offsets for.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason you removed this Scaladoc @param entry?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just consistency. I didn't have it for the other methods. I've updated the description.

@seglo
Copy link
Owner Author

seglo commented Mar 11, 2019

Thanks for the review @deanwampler ! :)

@seglo seglo merged commit ceff2ff into master Apr 17, 2019
@seglo seglo deleted the seglo/kafka-connect-retry branch April 17, 2019 21:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement backoff strategy for Kafka connections in Kafka Lag Exporter
2 participants