salt,docs: Make terminated-pod-gc-threshold configurable

Allow to manage the `terminated-pod-gc-threshold` argument from the
Bootstrap configuration, and default to 500 (note the current version of
controller manager default is 12500)

CHANGELOG.md

@@ -23,6 +23,11 @@
   so that if it's possible each `Dex` pods will sit on a different infra node
   (PR[#3614](https://github.com/scality/metalk8s/pull/3614))
 
+- Allow to manage the number of terminated pods that can exists before the
+  terminated pod garbage collector starts deleting terminated pods from the
+  Bootstrap configuration, and default to `500`
+  (PR[#xx](https://github.com/scality/metalk8s/pull/xx))
+
 ### Removals
 
 - Removed the PDF support for documentation, replaced it with the HTML output

docs/installation/bootstrap.rst

@@ -77,6 +77,9 @@ Configuration
         apiServer:
           featureGates:
             <feature_gate_name>: True
+        controllerManager:
+          config:
+            terminatedPodGCThreshold: 500
         coreDNS:
           replicas: 2
           affinity:
@@ -210,6 +213,11 @@ defaults kubernetes configuration.
               hard:
                 - topologyKey: kubernetes.io/hostname
 
+  From ``controllerManager`` section you can override the number of terminated
+  pods that can exist before the terminated pod garbage collector starts
+  deleting terminated pods. If it's set to 0, the terminated pod garbage
+  collector is disabled (default to ``500``)
+
 .. _Feature Gates: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
 
 .. _Bootstrap SSH Provisioning:

salt/_pillar/metalk8s.py

@@ -183,6 +183,10 @@ def _load_kubernetes(config_data):
     ).setdefault("soft", [{"topologyKey": "kubernetes.io/hostname"}])
     kubernetes_data["coreDNS"].setdefault("replicas", 2)
 
+    kubernetes_data.setdefault("controllerManager", {}).setdefault(
+        "config", {}
+    ).setdefault("terminatedPodGCThreshold", 500)
+
     return kubernetes_data
 
 

salt/metalk8s/kubernetes/controller-manager/installed.sls

@@ -44,6 +44,7 @@ Create kube-controller-manager Pod manifest:
           - --allocate-node-cidrs=true
           - --cluster-cidr={{ networks.pod }}
           - --node-cidr-mask-size=24
+          - --terminated-pod-gc-threshold={{ pillar.kubernetes.controllerManager.config.terminatedPodGCThreshold }}
           - --v={{ 2 if metalk8s.debug else 0 }}
         requested_cpu: 200m
         ports:

salt/tests/unit/formulas/data/base_pillar.yaml

@@ -175,6 +175,9 @@ certificates:
       workload-plane-ingress:
         watched: true
 kubernetes:
+  controllerManager:
+    config:
+      terminatedPodGCThreshold: 500
   coreDNS:
     replicas: 2
     affinity: