[KIP-848] User documentation #4702

emasab · 2024-04-29T17:28:17Z

Added changelog entry and a section in INTRODUCTION.md

lianetm · 2024-04-30T20:16:25Z

CHANGELOG.md

@@ -32,6 +33,9 @@ librdkafka v2.4.0 is a feature release:
   error. Rest of records in the batch will fail with the new error code
   _INVALID_DIFFERENT_RECORD (Java: KafkaException) and can be retried manually,
   depending on the application logic (#4583).
+ * The new consumer group rebalance protocol, defined in KIP 848, is still in **Early Availability**: _not production-ready_, _not supported_.


I think we usually refer to it as "Early Access", and that't actually the term we see in the logs when running the broker with it...

The new 'consumer' rebalance protocol is enabled along with the new group coordinator. This is part of the early access of KIP-848 and MUST NOT be used in production

I think we can also add a warning log for the same.

pranavrth

First Pass

pranavrth · 2024-05-06T08:52:43Z

CHANGELOG.md

@@ -2,7 +2,11 @@

 librdkafka v2.4.0 is a feature release:

- * [KIP-467](https://cwiki.apache.org/confluence/display/KAFKA/KIP-467%3A+Augment+ProduceResponse+error+messaging+for+specific+culprit+records) Augment ProduceResponse error messaging for specific culprit records (#4583).
+ * [KIP-848 EA](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol): The Next Generation of the Consumer Rebalance Protocol (#4610)


Let's keep this EA outside of the brackets and it should be spelled fully "EARLY ACCESS".

We should explain little more here itself instead of below. Something like "EARLY ACCESS: This should be used only for testing and not to be used in Production. Some of the feature/contract of this KIP might be changed in future."

pranavrth · 2024-05-06T08:54:53Z

CHANGELOG.md

+ * The new consumer group rebalance protocol, defined in KIP 848, is still in **Early Access**: _not production-ready_, _not supported_.
+   It's possible to try it in a non-production enviroment.
+   A [guide](INTRODUCTION.md#next-generation-of-the-consumer-group-protocol-kip-848) is available (#4610).


We should have a separate section for KIP-848 in the CHANGELOG I think.

Added a section for Early Access features separate from Upgrade considerations

pranavrth · 2024-05-06T09:04:39Z

INTRODUCTION.md

+Starting from librdkafka 2.4.0 the next generation consumer group rebalance protocol
+is in **Early Access**. It means it's still  _not production-ready_ and
+_not supported_, given it's still under validation and lacking some needed features.


Suggested change

Starting from librdkafka 2.4.0 the next generation consumer group rebalance protocol

is in **Early Access**. It means it's still _not production-ready_ and

_not supported_, given it's still under validation and lacking some needed features.

Starting from librdkafka 2.4.0 the next generation consumer group rebalance protocol introduced in KIP-848

is introduced.

Warning: It is still in **Early Access** which means it's still _not production-ready_ and

_not supported_, given it's still under validation and lacking some needed features. Some

of the feature and contracts might change in future.

pranavrth · 2024-05-06T09:09:40Z

INTRODUCTION.md

+To test it, a Kafka cluster must be set up, in KRaft mode, and the new group
+protocol enabled with the `group.coordinator.rebalance.protocols` property.
+Broker version must be Apache Kafka 3.7.0 or newer.


Let's add a reference to the broker documentation for it. https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes

pranavrth · 2024-05-06T09:12:10Z

INTRODUCTION.md

+- Client side assignors as described in the KIP
+- Online upgrade of a non-empty consumer group


I don't know if we should call these two as missing features or not. Adding them to missing features shows that we are planning to implement these in future which we are not as of now.

Instead of "Missing Features" we should call them "Future features" and remove these two points from there.

pranavrth · 2024-05-06T09:27:30Z

INTRODUCTION.md

+   For the same reason, when closing or unsubscribing with auto-commit set,
+   the member will try to commit until a specific timeout has passed.
+   Currently the timeout is the same as the `classic` protocol and it corresponds
+   to the deprecated `session.timeout.ms`, but it will change before the feature
+   reaches a stable state.


I don't think we should explain that we are still using deprecated property.

pranavrth · 2024-05-06T09:28:26Z

INTRODUCTION.md

+ - An `UNKNOWN_TOPIC_OR_PART` error won't be received when a consumer is
+   subscribing to a topic that doesn't exist in local cache, as the consumer
+   is still subscribing to the topic and it could be created just after that.


We should explain more on why we are writing this. What used to happen in old protocol.

pranavrth · 2024-05-06T09:28:52Z

INTRODUCTION.md

+ - A consumer won't do a preliminary Metadata call that returns a
+   `TOPIC_AUTHORIZATION_FAILED`, topic partitions will be assigned to the member
+   by the Coordinator only if it's authorized to consume from the topic.


pranavrth · 2024-05-06T09:29:16Z

INTRODUCTION.md

+ - Number of assign/revoke callbacks isn't fixed anymore, as it depends on
+   heartbeat timing.


What does this mean and does it affect the users?

pranavrth · 2024-05-06T09:29:48Z

INTRODUCTION.md

I think we need to adjust few sections and move some part to the changelog instead of here.

pranavrth

Looks good to me. Provided minor comments.

pranavrth · 2024-05-06T17:08:33Z

CHANGELOG.md

@@ -34,6 +36,18 @@ librdkafka v2.4.0 is a feature release:
   depending on the application logic (#4583).


+## Early Access


I think its better to put KIP-848 related section in itself so that we emphasize that this is a major feature we care about.

pranavrth · 2024-05-06T17:09:48Z

CHANGELOG.md

+ * The new consumer group rebalance protocol, defined in KIP 848, is still _not production-ready_ and _not supported_.
+   It's possible to try it in a non-production enviroment.
+
+   With this protocol the role of the Group Leader (a member) is removed and
+   the assignment is calculated by the Group Coordinator (a broker) and sent
+   to each member through heartbeats.


I think these two paragraph can be swapped. We want to explain about the protocol first and then tell about the caveats.

pranavrth · 2024-05-07T07:11:07Z

CHANGELOG.md

+   the assignment is calculated by the Group Coordinator (a broker) and sent
+   to each member through heartbeats.
+
+   The feature is still _not production-ready_ and _not supported_.


What does it mean by KIP-848 is not supported? I think we should explain.

pranavrth

LGTM!.

emasab force-pushed the dev_kip848_documentation branch 3 times, most recently from 532c7fc to b158759 Compare April 29, 2024 17:37

[KIP-848] User documentation

d440241

emasab force-pushed the dev_kip848_documentation branch from b158759 to d440241 Compare April 30, 2024 14:28

lianetm reviewed Apr 30, 2024

View reviewed changes

Address comments

3403d20

pranavrth requested changes May 6, 2024

View reviewed changes

emasab force-pushed the dev_kip848_documentation branch 2 times, most recently from 2f20f13 to c053c06 Compare May 6, 2024 16:38

Address comments

dc886dd

emasab force-pushed the dev_kip848_documentation branch from c053c06 to dc886dd Compare May 6, 2024 16:41

pranavrth reviewed May 6, 2024

View reviewed changes

Address comments

4bc0cdf

emasab force-pushed the dev_kip848_documentation branch from 2217907 to 4bc0cdf Compare May 6, 2024 22:54

pranavrth reviewed May 7, 2024

View reviewed changes

Remove the not supported statement

8ea2cee

pranavrth approved these changes May 7, 2024

View reviewed changes

emasab merged commit 18bc849 into master May 7, 2024
2 checks passed

emasab deleted the dev_kip848_documentation branch May 7, 2024 08:40

anchitj pushed a commit that referenced this pull request Jun 10, 2024

[KIP-848] User documentation (#4702)

9acd2df

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KIP-848] User documentation #4702

[KIP-848] User documentation #4702

emasab commented Apr 29, 2024

lianetm Apr 30, 2024

pranavrth May 6, 2024

pranavrth left a comment

pranavrth May 6, 2024

pranavrth May 6, 2024

emasab May 6, 2024

pranavrth May 6, 2024

pranavrth May 6, 2024

pranavrth May 6, 2024

pranavrth May 6, 2024

pranavrth May 6, 2024

pranavrth May 6, 2024

pranavrth May 6, 2024

emasab May 6, 2024

pranavrth May 6, 2024

pranavrth left a comment

pranavrth May 6, 2024

pranavrth May 6, 2024

pranavrth May 7, 2024

pranavrth left a comment

-Starting from librdkafka 2.4.0 the next generation consumer group rebalance protocol
-is in **Early Access**. It means it's still  _not production-ready_ and
-_not supported_, given it's still under validation and lacking some needed features.
+Starting from librdkafka 2.4.0 the next generation consumer group rebalance protocol introduced in KIP-848
+is introduced.
+Warning: It is still in **Early Access** which means it's still  _not production-ready_ and
+_not supported_, given it's still under validation and lacking some needed features. Some
+of the feature and contracts might change in future.

		- Client side assignors as described in the KIP
		- Online upgrade of a non-empty consumer group

		- Number of assign/revoke callbacks isn't fixed anymore, as it depends on
		heartbeat timing.

		@@ -34,6 +36,18 @@ librdkafka v2.4.0 is a feature release:
		depending on the application logic (#4583).


		## Early Access

[KIP-848] User documentation #4702

[KIP-848] User documentation #4702

Conversation

emasab commented Apr 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment