kafka-quarkus-keda-scaler

Sample project that builds a simple QuarkusConsumer, based on Quarkus.

Architecture Overview

                                                         Kubernetes Cluster
┌──────────────────────────┐          ┌───────────────────────────────────────────────────────────┐
│                          │          │                                                           │
│  Apache Kafka cluster    │          │                                                           │
│                          │          │                                                           │
│                          │          │                                   Consumer Group          │
│   ┌──────────────────┐   │          │                                 ┌─────────────────┐       │
│   │ Topic            │   │          │                                 │                 │       │
│   │                  │   │          │  Pull Messages                  │ ┌─────────────┐ │       │
│   │ ┌──────────────┐ ◄───┼──────────┼─────────────────────────────────┤ │Consumer-1   │ │       │
│   │ │Partition-1   │ │   │          │                                 │ │             │ │       │
│   │ └──────────────┘ │   │          │                                 │ └─────────────┘ │       │
│   │                  │   │          │                                 │                 │       │
│   │      ..          │   │          │                                 │     ..          │       │
│   │                  │   │          │                                 │                 │       │
│   │      ..          │   │          │                                 │     ..          │       │
│   │                  │   │          │                                 │                 │       │
│   │      ..          │   │          │                                 │     ...         │       │
│   │ ┌──────────────┐ │   │          │                                 │                 │       │
│   │ │Partition-n   │ │   │          ├─────────────────┐               │  ┌────────────┐ │       │
│   │ └──────────────┘ │   │          │                 │               │  │Consumer-n  │ │       │
│   │                  │   │          │                 │               │  │            │ │       │
│   │                  │   │          │ KEDA*       ┌───┼─────────────► │  └────────────┘ │       │
│   │                  │ ◄─┼──────────┼──┐          │   │               │                 │       │
│   └──────────────────┘   │          │  │          │   │               └─────────────────┘       │
│                          │          │                 │                                         │
└──────────────────────────┘          └─────────────────┴─────────────────────────────────────────┘
   *KEDA scales the consumers of the consumer-gourp, based on records on the topic and its partitions.

Apache Kafka topics

The consumers are getting messages from a topic like:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 10
  replicas: 3
  config:
    retention.bytes: 53687091200
    retention.ms: 36000000

We now use KEDA to dynamically scale our consumers, based on the consumer group for the topic.

The application is defined as a normal Kubernetes deployment.

The load

For generating some load, a batch like below can be used:

apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app: kafka-producer-job-notls-noacks
  name: kafka-producer-job-notls-noacks
  namespace: kafka
spec:
  parallelism: 5
  completions: 5
  backoffLimit: 1
  template:
    metadata:
      name: kafka-perf-producer
      labels:
        app: kafka-perf-producer
    spec:
      restartPolicy: Never
      containers:
      - name: kafka-perf-producer
        image: quay.io/strimzi/kafka:0.24.0-kafka-2.7.1
        command: [ "bin/kafka-producer-perf-test.sh" ]
        args: [ "--topic", "my-topic", "--throughput", "10000000", "--num-records", "1000000", "--producer-props", "bootstrap.servers=my-cluster-kafka-bootstrap:9092", "--record-size", "1000" ]
        volumeMounts:
        - name: strimzi-ca
          readOnly: true
          mountPath: "/etc/strimzi"
        env:
        - name: CA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-cluster-cluster-ca-cert
              key: ca.password
      volumes:
      - name: strimzi-ca
        secret:
          secretName: my-cluster-cluster-ca-cert
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - kafka-perf-producer
              topologyKey: kubernetes.io/hostname

Scaled consumers

After the initial count of one replica from the deployment, once the above load kicks in, KEDA dynamically scales the app to 10 replicas, see:

k get pods           
NAME                                     READY   STATUS    RESTARTS   AGE
kafka-quarkus-consumer-fbbdb4c57-6d77c   1/1     Running   0          50s
kafka-quarkus-consumer-fbbdb4c57-bfjxj   1/1     Running   0          50s
kafka-quarkus-consumer-fbbdb4c57-bp6th   1/1     Running   0          20s
kafka-quarkus-consumer-fbbdb4c57-ccb6q   1/1     Running   0          20s
kafka-quarkus-consumer-fbbdb4c57-g7pr5   1/1     Running   0          35s
kafka-quarkus-consumer-fbbdb4c57-j6289   1/1     Running   0          35s
kafka-quarkus-consumer-fbbdb4c57-ll789   1/1     Running   2          10m
kafka-quarkus-consumer-fbbdb4c57-qrp2l   1/1     Running   0          50s
kafka-quarkus-consumer-fbbdb4c57-s6w7f   1/1     Running   0          35s
kafka-quarkus-consumer-fbbdb4c57-scljw   1/1     Running   0          35s

NOTE: This is a simple POC/demo, and nothing special for rebalancing was done!