tutorial: Deploying Kafka on Kubernetes using Strimzi and Pulumi #13818
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Your site preview for commit fb3db12 is ready! 🎉 http://www-testing-pulumi-docs-origin-pr-13818-fb3db124.s3-website.us-west-2.amazonaws.com. |
|
Your site preview for commit 38cfa2d is ready! 🎉 http://www-testing-pulumi-docs-origin-pr-13818-38cfa2d0.s3-website.us-west-2.amazonaws.com. |
interurban
reviewed
Jan 23, 2025
|
|
||
| [Strimzi](https://strimzi.io/) is an open-source project that provides a way to run an Apache Kafka cluster on Kubernetes. It provides operators and custom resources to deploy and manage Kafka clusters on Kubernetes. | ||
|
|
||
| It is a CNCF project and is widely used in the Kubernetes community to deploy Kafka clusters as it makes it very easy handle the lifecycle of Kafka clusters. Before Strimzi, deploying Kafka on Kubernetes was a complex task with many manual steps. From creating topics to managing brokers, everything was a manual task. |
There was a problem hiding this comment.
Suggested change
| It is a CNCF project and is widely used in the Kubernetes community to deploy Kafka clusters as it makes it very easy handle the lifecycle of Kafka clusters. Before Strimzi, deploying Kafka on Kubernetes was a complex task with many manual steps. From creating topics to managing brokers, everything was a manual task. | |
| It is a CNCF project and is widely used in the Kubernetes community to deploy Kafka clusters, as it makes it very easy to handle the lifecycle of Kafka clusters. Before Strimzi, deploying Kafka on Kubernetes was a complex task with many manual steps. From creating topics to managing brokers, everything was a manual task. |
interurban
reviewed
Jan 23, 2025
|
|
||
| ## Deploying the Strimzi Kafka Operator | ||
|
|
||
| In this tutorial, we will two different ways to deploy a Kafka cluster on Kubernetes using Strimzi. The first way is to use Helm to deploy the Strimzi Kafka Operator and then create a Kafka cluster using the Strimzi Kafka Custom Resource Definition (CRD). |
There was a problem hiding this comment.
I'd suggest we simplify this tutorial to make it focused on using Pulumi to deploy Kafka vs a guide to how to use Strimzi. with that I'd drop having two ways, and highlight how to do this w/Pulumi.
interurban
reviewed
Jan 23, 2025
|
|
||
| KRaft is an event-based implementation of the Raft protocol with a quorum controller maintaining an event log and a single-partition topic named __cluster_metadata to store the metadata. Unlike the other topics, this is special because records are written to disk synchronously, which is required by the Raft algorithm for correctness. | ||
|
|
||
| It works in a leader-follower mode, where the leader writes events into the metadata topic which is then replicated to the follower controllers by using the KRaft replication algorithm. The leader of that single-partition topic is actually the controller node of the Kafka cluster. The metadata changes propagation has the benefit of being event-driven via replication instead of using RPCs. The metadata management is part of Kafka with the usage of a new quorum controller service which uses an event-sources storage model. The KRaft protocol is used to ensure that metadata are fully replicated across the quorum. |
There was a problem hiding this comment.
Suggested change
| It works in a leader-follower mode, where the leader writes events into the metadata topic which is then replicated to the follower controllers by using the KRaft replication algorithm. The leader of that single-partition topic is actually the controller node of the Kafka cluster. The metadata changes propagation has the benefit of being event-driven via replication instead of using RPCs. The metadata management is part of Kafka with the usage of a new quorum controller service which uses an event-sources storage model. The KRaft protocol is used to ensure that metadata are fully replicated across the quorum. | |
| It works in a leader-follower mode, where the leader writes events into the metadata topic which is then replicated to the follower controllers by using the KRaft replication algorithm. The leader of that single-partition topic is actually the controller node of the Kafka cluster. The propagation of metadata changes benefits from being event-driven through replication instead of relying on RPCs. Metadata management in Kafka utilizes a new quorum controller service that employs an event-sourced storage model. The KRaft protocol is used to ensure that metadata are fully replicated across the quorum. |
interurban
reviewed
Jan 23, 2025
|
|
||
| #### What is KRaft? | ||
|
|
||
| In order to overcome the limitations related to the ZooKeeper usage, the Kafka community came up with the idea of using Kafka itself to store metadata and use an event-driven pattern to make updates across the nodes. The work started with KIP-500 in late 2019 with the introduction of a built-in consensus protocol based on Raft. That was named Kafka Raft (**KRaft**) |
There was a problem hiding this comment.
Should we explain more about the Zookeeper usage? is there a blog/article to reference?
There was a problem hiding this comment.
Also should we link out to more detail on Kraft ?
interurban
reviewed
Jan 23, 2025
|
|
||
| In this section, we will use Pulumi to deploy the Strimzi Kafka Operator and create a Kafka cluster using Pulumi [CustomResource](/registry/packages/kubernetes/api-docs/apiextensions/customresource/) | ||
|
|
||
| ### Select your Pulumi supported language |
There was a problem hiding this comment.
I don't think this section block is required
interurban
reviewed
Jan 23, 2025
|
|
||
| Similar to the Helm installation, we will install the Strimzi Kafka Operator using the [Pulumi Kubernetes provider](/registry/packages/kubernetes/). | ||
|
|
||
| Replace the content of created Pulumi program with the following code: |
There was a problem hiding this comment.
Suggested change
| Replace the content of created Pulumi program with the following code: | |
| Replace the content of the created Pulumi program with the following code: |
interurban
reviewed
Jan 23, 2025
|
|
||
| Background: | ||
|
|
||
| * We create a dedicated namespace called `kafka` (optional, but often recommended). |
There was a problem hiding this comment.
Suggested change
| * We create a dedicated namespace called `kafka` (optional, but often recommended). | |
| * We create a dedicated namespace called `kafka` (optional, but recommended). |
interurban
reviewed
Jan 23, 2025
|
|
||
| - Learn more about Pulumi and Kubernetes in the [Kubernetes documentation](/docs/iac/clouds/kubernetes/). | ||
| - Learn more about the `Release` resource in the [Pulumi Kubernetes API documentation](/registry/packages/kubernetes/api-docs/helm/v3/release/). | ||
| - Try the out the `Chart` [tutorial](/tutorials/kubernetes-helm-part-two) to learn how to install Helm charts on Kubernetes using the `Chart` resource. |
There was a problem hiding this comment.
Suggested change
| - Try the out the `Chart` [tutorial](/tutorials/kubernetes-helm-part-two) to learn how to install Helm charts on Kubernetes using the `Chart` resource. | |
| - Try out the `Kubernetes Helm Chart` [tutorial](/tutorials/kubernetes-helm-part-two) to learn how to install Helm charts on Kubernetes using the `Chart` resource. |
interurban
reviewed
Jan 23, 2025
| * Sets up both plaintext (plain) and TLS listeners (tls). | ||
| * Includes the Entity Operator, which manages users and topics. | ||
|
|
||
| ### Creating a Kafka Topic |
There was a problem hiding this comment.
Suggested change
| ### Creating a Kafka Topic | |
| ### Create a Kafka Topic |
interurban
reviewed
Jan 23, 2025
|
|
||
| {{% /choosable %}} | ||
|
|
||
| ### Creating a Kafka User (Optional) |
There was a problem hiding this comment.
Suggested change
| ### Creating a Kafka User (Optional) | |
| ### Create a Kafka User (Optional) |
Also; why make this step optional?
interurban
reviewed
Jan 23, 2025
|
|
||
|  | ||
|
|
||
| ## Deploying the Strimzi Kafka Operator |
There was a problem hiding this comment.
Suggested change
| ## Deploying the Strimzi Kafka Operator | |
| ## Deploy the Strimzi Kafka Operator |
interurban
reviewed
Jan 23, 2025
|
|
||
| {{% /choosable %}} | ||
|
|
||
| ### Deploying the whole stack |
There was a problem hiding this comment.
Suggested change
| ### Deploying the whole stack | |
| ### Deploy the Kafka Kubernetes Cluster |
... or something similar
interurban
reviewed
Jan 23, 2025
| - kubernetes | ||
| --- | ||
|
|
||
| ## What is Strimzi? |
There was a problem hiding this comment.
I'd suggest we start up with an overview of Kafka and K8; and what that architecture looks like (with a great visual like you have below), and talk about the common use cases of this scenario ahead of talking about Strimzi. Currently it's a tutorial about Strimzi, but instead it should be about using Pulumi to deploy a Kafka Cluster primarily, then I think it would be still great to say there's a new open source project that makes this much easier than it was in the past.
interurban
reviewed
Jan 23, 2025
| title: Deploying Kafka on Kubernetes using Strimzi and Pulumi | ||
| layout: single | ||
| description: | | ||
| Learn how to deploy a Kafka cluster on Kubernetes using Strimzi and Pulumi. |
There was a problem hiding this comment.
Suggested change
| Learn how to deploy a Kafka cluster on Kubernetes using Strimzi and Pulumi. | |
| Learn how to deploy a Kafka cluster on Kubernetes using Strimzi and Pulumi. |
per my feedback below, it seems like we should make the highlight and topic about creating a Kafka cluster w/Pulumi and not highlight Strimzi in the title, but rather leave that in the tutorial itself.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Unreleased product version (optional)
Related issues (optional)