Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update self-managed Kafka to v0.44 & fix broken URLs #1012

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ spec:
name: strimzi-cluster-operator
containers:
- name: strimzi-cluster-operator
image: quay.io/strimzi/operator:0.38.0
image: quay.io/strimzi/operator:0.44.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
Expand Down Expand Up @@ -57,44 +57,42 @@ spec:
value: "120000"
- name: STRIMZI_OPERATION_TIMEOUT_MS
value: "300000"
- name: STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE
value: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
- name: STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE
value: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
value: quay.io/strimzi/kafka:0.44.0-kafka-3.8.0
- name: STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE
value: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
value: quay.io/strimzi/kafka:0.44.0-kafka-3.8.0
- name: STRIMZI_KAFKA_IMAGES
value: |
3.5.0=quay.io/strimzi/kafka:0.38.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.38.0-kafka-3.5.1
3.6.0=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
3.7.0=quay.io/strimzi/kafka:0.44.0-kafka-3.7.0
3.7.1=quay.io/strimzi/kafka:0.44.0-kafka-3.7.1
3.8.0=quay.io/strimzi/kafka:0.44.0-kafka-3.8.0
- name: STRIMZI_KAFKA_CONNECT_IMAGES
value: |
3.5.0=quay.io/strimzi/kafka:0.38.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.38.0-kafka-3.5.1
3.6.0=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
3.7.0=quay.io/strimzi/kafka:0.44.0-kafka-3.7.0
3.7.1=quay.io/strimzi/kafka:0.44.0-kafka-3.7.1
3.8.0=quay.io/strimzi/kafka:0.44.0-kafka-3.8.0
- name: STRIMZI_KAFKA_MIRROR_MAKER_IMAGES
value: |
3.5.0=quay.io/strimzi/kafka:0.38.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.38.0-kafka-3.5.1
3.6.0=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
3.7.0=quay.io/strimzi/kafka:0.44.0-kafka-3.7.0
3.7.1=quay.io/strimzi/kafka:0.44.0-kafka-3.7.1
3.8.0=quay.io/strimzi/kafka:0.44.0-kafka-3.8.0
- name: STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES
value: |
3.5.0=quay.io/strimzi/kafka:0.38.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.38.0-kafka-3.5.1
3.6.0=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
3.7.0=quay.io/strimzi/kafka:0.44.0-kafka-3.7.0
3.7.1=quay.io/strimzi/kafka:0.44.0-kafka-3.7.1
3.8.0=quay.io/strimzi/kafka:0.44.0-kafka-3.8.0
- name: STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE
value: quay.io/strimzi/operator:0.38.0
value: quay.io/strimzi/operator:0.44.0
- name: STRIMZI_DEFAULT_USER_OPERATOR_IMAGE
value: quay.io/strimzi/operator:0.38.0
value: quay.io/strimzi/operator:0.44.0
- name: STRIMZI_DEFAULT_KAFKA_INIT_IMAGE
value: quay.io/strimzi/operator:0.38.0
value: quay.io/strimzi/operator:0.44.0
- name: STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE
value: quay.io/strimzi/kafka-bridge:0.27.0
value: quay.io/strimzi/kafka-bridge:0.30.0
- name: STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE
value: quay.io/strimzi/kaniko-executor:0.38.0
value: quay.io/strimzi/kaniko-executor:0.44.0
- name: STRIMZI_DEFAULT_MAVEN_BUILDER
value: quay.io/strimzi/maven-builder:0.38.0
value: quay.io/strimzi/maven-builder:0.44.0
- name: STRIMZI_OPERATOR_NAMESPACE
valueFrom:
fieldRef:
Expand All @@ -117,13 +115,13 @@ spec:
httpGet:
path: /healthy
port: http
initialDelaySeconds: 60
initialDelaySeconds: 10
periodSeconds: 30
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 60
initialDelaySeconds: 10
periodSeconds: 30
resources:
limits:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ metadata:
namespace: kafka
spec:
kafka:
version: 3.6.0
version: 3.8.0
replicas: 1
resources:
requests:
Expand Down Expand Up @@ -36,7 +36,7 @@ spec:
transaction.state.log.min.isr: 1
default.replication.factor: 1
min.insync.replicas: 1
inter.broker.protocol.version: "3.6"
inter.broker.protocol.version: "3.8"
storage:
type: jbod
volumes:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ metadata:
namespace: kafka
spec:
containers:
- image: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
- image: quay.io/strimzi/kafka:0.44.0-kafka-3.8.0
name: kafka-consumer
resources:
requests:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ metadata:
namespace: kafka
spec:
containers:
- image: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
- image: quay.io/strimzi/kafka:0.44.0-kafka-3.8.0
name: kafka-producer
resources:
requests:
Expand Down
42 changes: 21 additions & 21 deletions docs/user-guide/self-managed-services/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ This page will show you how to install Strimzi Kafka Operator on Welkin. You can

!!! Note "Supported versions"

This installation guide has been tested with Strimzi Kafka Operator version [0.38.0](https://github.com/strimzi/strimzi-kafka-operator/tree/0.38.0).
This installation guide has been tested with Strimzi Kafka Operator version [0.44.0](https://github.com/strimzi/strimzi-kafka-operator/tree/0.44.0).

## Enable Self-Managed Kafka

Expand All @@ -51,8 +51,8 @@ You need to apply the Custom Resource Definitions (CRDs) required by Strimzi Kaf
```bash
mkdir crds

# Fetches Strimzi Kafka Operator CRDs for v0.38.0 and saves it in the crds directory
curl -L https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.38.0/strimzi-crds-0.38.0.yaml > crds/kafka-crds.yaml
# Fetches Strimzi Kafka Operator CRDs for v0.44.0 and saves it in the crds directory
curl -L https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.44.0/strimzi-crds-0.44.0.yaml > crds/kafka-crds.yaml

kubectl apply -f crds/kafka-crds.yaml
```
Expand All @@ -75,9 +75,9 @@ If you have multiple namespaces that ought to be targets for Strimzi Kafka Opera
mkdir roles

# Fetches the necessary Roles and saves it in the roles directory
curl https://raw.githubusercontent.com/elastisys/welkinmain/docs/user-guide/self-managed-services/kafka-files/roles/kafka-role.yaml > roles/kafka-role.yaml
curl https://raw.githubusercontent.com/elastisys/welkinmain/docs/user-guide/self-managed-services/kafka-files/roles/kafka-rolebinding.yaml > roles/kafka-rolebinding.yaml
curl https://raw.githubusercontent.com/elastisys/welkinmain/docs/user-guide/self-managed-services/kafka-files/roles/kustomization.yaml > roles/kustomization.yaml
curl https://raw.githubusercontent.com/elastisys/welkin/main/docs/user-guide/self-managed-services/kafka-files/roles/kafka-role.yaml > roles/kafka-role.yaml
curl https://raw.githubusercontent.com/elastisys/welkin/main/docs/user-guide/self-managed-services/kafka-files/roles/kafka-rolebinding.yaml > roles/kafka-rolebinding.yaml
curl https://raw.githubusercontent.com/elastisys/welkin/main/docs/user-guide/self-managed-services/kafka-files/roles/kustomization.yaml > roles/kustomization.yaml

# If you created the namespace kafka from another namespace other than production, edit the namespace in roles/kustomization.yaml

Expand All @@ -92,7 +92,7 @@ You need to create the ServiceAccount and ConfigMap that Strimzi Kafka Operator
mkdir sa-cm

# Fetches the ServiceAccount and ConfigMap and saves it in the sa-cm directory
curl https://raw.githubusercontent.com/elastisys/welkinmain/docs/user-guide/self-managed-services/kafka-files/sa-cm/kafka-sa-cm.yaml > sa-cm/kafka-sa-cm.yaml
curl https://raw.githubusercontent.com/elastisys/welkin/main/docs/user-guide/self-managed-services/kafka-files/sa-cm/kafka-sa-cm.yaml > sa-cm/kafka-sa-cm.yaml

kubectl apply -f sa-cm/kafka-sa-cm.yaml
```
Expand All @@ -101,7 +101,7 @@ kubectl apply -f sa-cm/kafka-sa-cm.yaml

With the initial prep done, you are now ready to deploy the operator.

You can find the deployment manifest [here](https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.38.0/install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml). Deploying this on Welkin does require a Security Context to be added.
You can find the deployment manifest [here](https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.44.0/install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml). Deploying this on Welkin does require a Security Context to be added.

Edit the manifest and add this under `spec.template.spec.containers[0]`:

Expand All @@ -125,7 +125,7 @@ Alternatively you can fetch an already edited file:
mkdir deployment

# Fetches the edited operator Deployment and saves it in the deployment directory
curl https://raw.githubusercontent.com/elastisys/welkinmain/docs/user-guide/self-managed-services/kafka-files/deployment/kafka-operator-deployment.yaml > deployment/kafka-operator-deployment.yaml
curl https://raw.githubusercontent.com/elastisys/welkin/main/docs/user-guide/self-managed-services/kafka-files/deployment/kafka-operator-deployment.yaml > deployment/kafka-operator-deployment.yaml

kubectl apply -f deployment/kafka-operator-deployment.yaml
```
Expand All @@ -136,7 +136,7 @@ To configure the Strimzi Kafka Operator to watch multiple namespaces (e.g. runni

You are now ready to deploy your Kafka cluster!

The example files provided by Strimzi [here](https://github.com/strimzi/strimzi-kafka-operator/tree/0.38.0/examples/kafka) serves as a good starting point.
The example files provided by Strimzi [here](https://github.com/strimzi/strimzi-kafka-operator/tree/0.44.0/examples/kafka) serves as a good starting point.

Welkin requires that resource requests are specified for all containers. By default, the Strimzi Cluster Operator does not specify CPU and memory resource requests and limits for its deployed operands.

Expand All @@ -148,7 +148,7 @@ You can fetch a modified persistent-single example that includes resource reques
mkdir kafka-cluster

# Fetches the edited kafka cluster example and saves it in the kafka-cluster directory
curl https://raw.githubusercontent.com/elastisys/welkinmain/docs/user-guide/self-managed-services/kafka-files/kafka-cluster/persistent-single.yaml > kafka-cluster/persistent-single.yaml
curl https://raw.githubusercontent.com/elastisys/welkin/main/docs/user-guide/self-managed-services/kafka-files/kafka-cluster/persistent-single.yaml > kafka-cluster/persistent-single.yaml

kubectl apply -f kafka-cluster/persistent-single.yaml
```
Expand All @@ -165,16 +165,16 @@ After you have deployed your Kafka cluster, you can test sending and receiving m

To do this, you can use a producer and consumer as seen [here](https://strimzi.io/quickstarts/), under the section "Send and receive messages". But since Welkin requires resource requests to be specified, just copy pasting those commands will not work.

You need to create a Pod manifest using the image `quay.io/strimzi/kafka:0.38.0-kafka-3.6.0`, and then you need to add your resource requests to this manifest. You also need to have an initial sleep command in the Pod manifest, to sleep the container for a while, this is to avoid the Pod going into the "Completed" stage instantly.
You need to create a Pod manifest using the image `quay.io/strimzi/kafka:0.44.0-kafka-3.8.0`, and then you need to add your resource requests to this manifest. You also need to have an initial sleep command in the Pod manifest, to sleep the container for a while, this is to avoid the Pod going into the "Completed" stage instantly.

Alternatively you can download a ready to use producer and consumer Pod manifests:

```bash
mkdir kafka-testing

# Fetches Pod manifests for a producer and consumer and saves it in the kafka-testing directory
curl https://raw.githubusercontent.com/elastisys/welkinmain/docs/user-guide/self-managed-services/kafka-files/kafka-testing/kafka-producer.yaml > kafka-testing/kafka-producer.yaml
curl https://raw.githubusercontent.com/elastisys/welkinmain/docs/user-guide/self-managed-services/kafka-files/kafka-testing/kafka-consumer.yaml > kafka-testing/kafka-consumer.yaml
curl https://raw.githubusercontent.com/elastisys/welkin/main/docs/user-guide/self-managed-services/kafka-files/kafka-testing/kafka-producer.yaml > kafka-testing/kafka-producer.yaml
curl https://raw.githubusercontent.com/elastisys/welkin/main/docs/user-guide/self-managed-services/kafka-files/kafka-testing/kafka-consumer.yaml > kafka-testing/kafka-consumer.yaml

kubectl apply -f kafka-testing/kafka-producer.yaml
kubectl apply -f kafka-testing/kafka-consumer.yaml
Expand All @@ -194,16 +194,16 @@ kubectl exec -it kafka-consumer -- bin/kafka-console-consumer.sh --bootstrap-ser

## Further reading

- [Strimzi Overview](https://strimzi.io/docs/operators/0.38.0/overview)
- [Strimzi Overview](https://strimzi.io/docs/operators/0.44.0/overview)

- [Deploying and Upgrading](https://strimzi.io/docs/operators/0.38.0/deploying)
- [Deploying and Upgrading](https://strimzi.io/docs/operators/0.44.0/deploying)

- [API Reference](https://strimzi.io/docs/operators/0.38.0/configuring)
- [API Reference](https://strimzi.io/docs/operators/0.44.0/configuring)

- [Configure Operator to watch multiple namespaces](https://strimzi.io/docs/operators/0.38.0/deploying#deploying-cluster-operator-to-watch-multiple-namespaces-str)
- [Configure Operator to watch multiple namespaces](https://strimzi.io/docs/operators/0.44.0/deploying#deploying-cluster-operator-to-watch-multiple-namespaces-str)

- [About resources for Strimzi containers](https://strimzi.io/docs/operators/0.38.0/configuring#con-common-configuration-resources-reference)
- [About resources for Strimzi containers](https://strimzi.io/docs/operators/0.44.0/configuring#con-common-configuration-resources-reference)

- [Configuring Kafka](https://strimzi.io/docs/operators/0.38.0/deploying#con-config-kafka-str)
- [Configuring Kafka](https://strimzi.io/docs/operators/0.44.0/deploying#con-config-kafka-str)

- [Configuring Kafka and ZooKeeper storage](https://strimzi.io/docs/operators/0.38.0/deploying#assembly-storage-str)
- [Configuring Kafka and ZooKeeper storage](https://strimzi.io/docs/operators/0.44.0/deploying#assembly-storage-str)
Loading