Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: handle error in kafka consumer loop to prevent premature return #2100

Merged
merged 4 commits into from
Feb 24, 2025
Merged

fix: handle error in kafka consumer loop to prevent premature return #2100

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
348bbe4
fix: handle error in kafka consumer loop to prevent premature return
Gezi-lzq Feb 17, 2025
a2a6553
fix: add comments
Gezi-lzq Feb 19, 2025
a51d5d0
fix: add retry delay in kafka consumer loop to prevent busy looping
Gezi-lzq Feb 21, 2025
b25cd78
chore: trigger ci
Gezi-lzq Feb 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion plugins/input/kafka/input_kafka.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import (
"fmt"
"strings"
"sync"
"time"

"github.com/IBM/sarama"

Expand Down Expand Up @@ -160,8 +161,17 @@ func (k *InputKafka) Init(context pipeline.Context) (int, error) {
defer k.wg.Done()
for {
if err := k.consumerGroupClient.Consume(cancelCtx, k.Topics, k); err != nil {
// Keep retrying Consume when error occurs.
// This loop will only exit when the context is canceled (i.e., when loongcollector process is stopping)
logger.Error(k.context.GetRuntimeContext(), "INPUT_KAFKA_ALARM", "Error from kafka consumer", err)
return

// Add a retry delay to avoid busy loop.
select {
case <-time.After(time.Second * 5):
case <-cancelCtx.Done():
logger.Info(k.context.GetRuntimeContext(), "Consumer was canceled. Leaving consumer group")
return
}
}
// check if context was canceled, signaling that the consumer should stop
if cancelCtx.Err() != nil {
Expand Down
Loading