Consider switching default Kafka version to at least 2.1

[ijuma]

The default Kafka version was set to 1.0 via #1787. This was a huge improvement over the previous default and the hope was that proper api versions support would be supported some time later.

2.5 years have passed and the default remains 1.0 if I understand correctly (please correct me if this is wrong).

I am working on a KIP to remove older protocol versions in Apache Kafka 4.0 and the proposed baseline is Apache Kafka 2.1:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0

The KIP is still under discussion (i.e. it hasn't been approved yet), but I wanted to:

1. Give an early heads up to this project since it would impact your users with the current default
2. Get your input on whether bumping the default Kafka version in the near term is something you would consider

Thanks!

[dnwe]

@ijuma thanks for getting in touch and the great writeup of the KIP and 2.1 sounds like a good baseline. Bumping the default KafkaVersion isn't a problem at all, but as you note our protocol version implementations aren't in sync with the Java client anyway

I'd suggest we sync this up with bumping Sarama to a 2.x and we can work on ensuring that we have implemented all of the protocol baselines to match the Java client at 2.1 (V7 MetadataRequest etc.) and to match what an ApiVersionRequest to a Kafka 2.1 broker returns

[ijuma]

@dnwe that sounds great if the project has the cycles to update all of the protocol baselines as well.

[ijuma]

Hi @dnwe. Is Sarama 2.x on the horizon or would it be better to do something more incremental in the meantime to avoid a situation where Sarama users with reasonably recent clients break by default with Apache Kafka 4.0?

[dnwe]

@ijuma whilst on holiday I spent a bit of time looking into the minimum we'd need to do in order to meet this i.e., if we change the default baked version to sarama.V2.1.0.0 then a) what default request versions do we support, but are not sending correctly atm and b) what protocol versions are not even implemented that need implementing so this is WIP

[ijuma]

Awesome to see the progress @dnwe!

[dnwe]

@ijuma 😀

Is there a branch or patchset anywhere for apache/kafka that has a draft implementation of KAFKA-14560? It would be useful if we could configure a broker and set the minimum APIKey versions somehow so we can test ourselves out against it

[ijuma]

@dnwe not yet, I'll let you know once that exists.

[github-actions]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[ijuma]

Is there a branch or patchset anywhere for apache/kafka that has a draft implementation of KAFKA-14560? It would be useful if we could configure a broker and set the minimum APIKey versions somehow so we can test ourselves out against it

The 4.0 branch of Apache Kafka now contains the KIP-896 change:
apache/kafka#18291