Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set message timestamp to the metric time in kafka output #6746

Merged
merged 1 commit into from
Dec 3, 2019
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions plugins/outputs/kafka/kafka.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ type (
// SASL Password
SASLPassword string `toml:"sasl_password"`

Log telegraf.Logger `toml:"-"`

tlsConfig tls.Config
producer sarama.SyncProducer

Expand Down Expand Up @@ -316,13 +318,14 @@ func (k *Kafka) Write(metrics []telegraf.Metric) error {
for _, metric := range metrics {
buf, err := k.serializer.Serialize(metric)
if err != nil {
log.Printf("D! [outputs.kafka] Could not serialize metric: %v", err)
k.Log.Debugf("Could not serialize metric: %v", err)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's more like error than debug information. Because it's about data loss, and not about "how and where?". It's my opinion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change is made here to the level, so I think we can handle this in a separate issue. I think it may have been added at some point due to metrics without fields, need to do some research on the history.

continue
}

m := &sarama.ProducerMessage{
Topic: k.GetTopicName(metric),
Value: sarama.ByteEncoder(buf),
Topic: k.GetTopicName(metric),
Value: sarama.ByteEncoder(buf),
Timestamp: metric.Time(),
}

key, err := k.routingKey(metric)
Expand All @@ -342,7 +345,11 @@ func (k *Kafka) Write(metrics []telegraf.Metric) error {
if errs, ok := err.(sarama.ProducerErrors); ok {
for _, prodErr := range errs {
if prodErr.Err == sarama.ErrMessageSizeTooLarge {
log.Printf("E! Error writing to output [kafka]: Message too large, consider increasing `max_message_bytes`; dropping batch")
k.Log.Error("Message too large, consider increasing `max_message_bytes`; dropping batch")
return nil
}
if prodErr.Err == sarama.ErrInvalidTimestamp {
k.Log.Error("The timestamp of the message is out of acceptable range, consider increasing broker `message.timestamp.difference.max.ms`; dropping batch")
return nil
}
return prodErr
Expand Down