Production-ready: Configure PDB

[slopezz]

In 3scale SaaS we have been using successfully limitador for a couple of years together with Redis, to protect all our public endpoints. However:

• We are using an old image community image
• Yamls are managed individually via ArgoCD

We would like to update how we manage limitador application, and use the most recommended limitador setup using limitador-operator, with a production-ready grade.

Current limitador-operator:

• Do not configure PDB by default
• Do not permit to configure PDB via CR

Desired features:

• Permit to configure PDB via CR
• Being thought the operator to possibly having a single limitador pod running at once, maybe PDB should not be enabled by default
• PDB helps when there is more than 1 replica, because it ensures that upon a cluster maintenance where nodes are being updated one after one, there is always a minimum/maximum number of pod replicas providing service so not having a downtime

3scale SaaS specific example

Example of PDB used in 3scale SaaS production to manage between 3,500 and 5,500 requests/second with 3 limitador pods (selector labels need to coincide with the labels managed right now by limitador-operator):

kind: PodDisruptionBudget
apiVersion: policy/v1
metadata:
  name: limitador
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: limitador
  maxUnavailable: 1

Possible CR config

apiVersion: limitador.kuadrant.io/v1alpha1
kind: Limitador
metadata:
  name: limitador-sample
spec:
  pdb:
    maxUnavailable: 1
    minAvailable: 2  # Note this field is mutually exclusive setting with "minAvailable", normally better use maxUnavailable, only one of them can be used at the same time

Example how we externalize PDB config in 3scale SaaS Operator CR.

https://github.com/3scale-ops/saas-operator/blob/main/docs/api-reference/reference.asciidoc#k8s-api-github.aaakk.us.kg-3scale-saas-operator-api-v1alpha1-poddisruptionbudgetspec