Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update node-rdkafka to v3.2.0 #869

Closed
godber opened this issue Nov 20, 2024 · 3 comments
Closed

Update node-rdkafka to v3.2.0 #869

godber opened this issue Nov 20, 2024 · 3 comments
Assignees
Labels
dependencies Pull requests that update a dependency file

Comments

@godber
Copy link
Member

godber commented Nov 20, 2024

We should update to this:

https://github.com/Blizzard/node-rdkafka/releases/tag/v3.2.0

It in turn updates to librdkafka 2.6.0: Blizzard/node-rdkafka#1103

We should bump the minor version on our asset and make sure to update the terafoundation_kafka_connector version in the base image.

@godber godber added the dependencies Pull requests that update a dependency file label Nov 20, 2024
@godber
Copy link
Member Author

godber commented Nov 20, 2024

On question we need to answer is how is this cooperative rebalance behavior triggered, then we need to test whether it works in our high-volume scenarios. I this these are clues:

The name of one or more partition assignment strategies. The elected group leader will use a strategy supported by all members of the group to assign partitions to group members. If there is more than one eligible strategy, preference is determined by the order of this list (strategies earlier in the list have higher priority). Cooperative and non-cooperative (eager) strategies must not be mixed. Available strategies: range, roundrobin, cooperative-sticky.
Type: string

@busma13
Copy link
Contributor

busma13 commented Nov 21, 2024

kafka_reader, kafka_sender, and kafka_dead_letter all have configs with a partition_assignment_strategy field. We will need to update the types to allow for cooperative-sticky. We will also need to update the schema.partition_assignment_strategy.format array for each.

Then we just need to update each operation in a job to set the partition_assignment_strategy if applicable.

@busma13
Copy link
Contributor

busma13 commented Nov 26, 2024

partition_assignment_strategy should not be in the sender or dead_letter, so we will be removing them.
ref: #873

godber pushed a commit that referenced this issue Dec 6, 2024
…reader (#882)

This PR makes the following changes:

- Bump the kafka-asset version from 5.3.0 to 5.4.0
- Update the `kafka_reader` and `kafka_reader_api` to allow for
`cooperative-sticky` as a `partition_assignment_strategy` option.
- Set `minimum_teraslice_version` to 2.9.0, as this version uses
node-rdkafka v3.2.0, which is the first version to allow for incremental
rebalancing using `cooperative-sticky`.
- Remove `partition_assignment_strategy` option from `kafka_sender`,
`kafka_sender_api` and `kafka_dead_letter`. It looks like these might
have been copy/paste errors, as this is not a valid property on a kafka
producer config.

Ref: #869, #873
@godber godber closed this as completed Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file
Projects
None yet
Development

No branches or pull requests

3 participants