Consumer stops consuming after log roll

[mildebrandt]

Description

I may be doing something wrong with my configuration. If it seems to be correct, I'll add steps for a recreation.

When the log rolls for a partition, my program no longer consumes messages. Here's the Kafka log:

[2018-05-17 21:16:52,166] INFO Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log)
[2018-05-17 21:16:52,169] INFO Rolled new log segment for 'alerts-93' in 3 ms. (kafka.log.Log)
[2018-05-17 21:16:52,171] INFO Scheduling log segment 0 for log alerts-93 for deletion. (kafka.log.Log)
[2018-05-17 21:16:52,172] INFO Incrementing log start offset of partition alerts-93 to 40593 in dir /opt/kafka/data/logs (kafka.log.Log)
[2018-05-17 21:16:52,175] INFO Cleared earliest 0 entries from epoch cache based on passed offset 40593 leaving 1 in EpochFile for partition alerts-93 (kafka.server.epoch.LeaderEpochFileCache)
[2018-05-17 21:17:52,173] INFO Deleting segment 0 from log alerts-93. (kafka.log.Log)
[2018-05-17 21:17:52,179] INFO Deleting index /opt/kafka/data/logs/alerts-93/00000000000000000000.index.deleted (kafka.log.OffsetIndex)
[2018-05-17 21:17:52,264] INFO Deleting index /opt/kafka/data/logs/alerts-93/00000000000000000000.timeindex.deleted (kafka.log.TimeIndex)

I'm reading from three different topics, and the consumption continues for the other topics within the same Poll() loop. I don't see any kafka.Error event types coming through....my program would exit during errors and Kubernetes will restart it.

Here's my config map:

"bootstrap.servers":        <brokers>,
"group.id":                 <group id>,
"session.timeout.ms":       6000,
"default.topic.config":     kafka.ConfigMap{"auto.offset.reset": "latest"},
"security.protocol":        "ssl",
"ssl.key.location":         <key file>,
"ssl.certificate.location": <cert file>,
"ssl.ca.location":          <ca file>

I don't care about earlier messages during a restart, so the auto.offset.reset is set to latest.

I'm using librdkafka/confluent-kafka-go 0.11.0 and Kafka 1.0.0. If my config looks correct, I'll create a way to recreate this issue. Thanks!

[edenhill]

Please update to librdkafka and go 0.11.4 and try to reproduce with "debug": "fetch" enabled.

[mildebrandt]

Ok...so this isn't expected. :) Sure, I'll provide a recreation scenario with the latest and provide logs.

[mildebrandt]

I'm unable to recreate this with a smaller test case....logs are rolling and the consumer keeps consuming. I'll close this for now and reopen if I can provide a way to consistently reproduce.