Check for insufficient data when we try to get the records. #998
Conversation
In some corner cases (like v2 records produced on the topic but the sarama consumer uses v0 Fetch API with a limit less than one record size) we might receive a truncated record batch. We want to be able to make progress by increasing the fetch limit, so don't return an error, but rater mark the batch as partial trailing record.
|
@wladh what does the java client do here? |
|
This fix makes sense to me, though I'm also curious what the java client does here. |
|
Thanks for digging into this! |
|
From what I gather (not familiar with Kafka codebase) the java client doesn't have this incremental fetch size logic. So it basically fails the way sarama was failing before this commit: https://github.com/apache/kafka/blob/4b8a29f12a142d02be0b64eac71975b6a129d04a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L840-L855 |
|
@wladh that doesn't line up with my testing, which seemed to indicate everything "just worked" with the jvm clients. |
|
@tcrayford like I said I'm not familiar with the code base and have never used the Java client (this was just looking at the code at HEAD in their repo, maybe things were changed) so I might be missing something. One difference that I see is that the default fetch size for Java client is 1MB, whereas sarama starts at 32KB. But I don't know how big your messages were or if you changed any of these params. |
In some corner cases (like v2 records produced on the topic but the
sarama consumer uses v0 Fetch API with a limit less than one record size)
we might receive a truncated record batch. We want to be able to make
progress by increasing the fetch limit, so don't return an error, but
rater mark the batch as partial trailing record.
Fixes issue #962