Mark thanos storegateway size immutable

The `k8s-service-layer.prometheus.thanos_storegateway_size` yk8s config
 setting feeds into the `storegateway.persistence.size` parameter of the
 bitnami/thanos helm chart which feeds into the `volumeClaimTemplates` field of
 the `thanos-storegateway` statefulset created. That field is currently
 immutable making helm updates with the parameter fail.

Related-to: kubernetes/enhancements#4650

docs/_releasenotes/1498.fix.mark-thanos-storegateway-size-immutable

@@ -0,0 +1,2 @@
+Marked the ``k8s-service-layer.prometheus.thanos_storegateway_size`` config setting immutable
+and outlined a manual workaround to increase the size nonetheless.

docs/user/reference/cluster-configuration.rst

@@ -421,6 +421,19 @@ The used rook setup is explained in more detail
 
    </details>
 
+.. attention::
+
+   The following settings are immutable:
+   - ``mon_volume_storage_class``
+   - ``mon_volume_size``
+   - ``osd_volume_size``
+   - ``osd_storage_class``
+
+   If you still want to change them, follow these manual steps:
+   1. Increase the size of the corresponding PVC
+   2. Delete the stateful set: ``kubectl delete -n monitoring sts --cascade=false <statefulset_name>``
+   3. Re-deploy it with the LCM: ``AFLAGS="--diff --tags rook --tags rook_v2" bash managed-k8s/actions/apply-k8s-supplements.sh``
+
 |
 
 .. _cluster-configuration.prometheus-configuration:
@@ -518,6 +531,17 @@ size for each component like in the following.
    thanos_query_size = "ZGi"
    # [...]
 
+.. attention::
+
+   The following options are immutable:
+   - ``thanos_storegateway_size``
+   - ``thanos_compactor_size``
+
+   If you still want to increase it, follow these manual steps:
+   1. Increase the size of the corresponding PVC
+   2. Delete the stateful set: ``kubectl delete -n monitoring sts --cascade=false thanos-<storegateway|compactor>``
+   3. Re-deploy it with the LCM: ``AFLAGS="--diff --tags thanos" bash managed-k8s/actions/apply-k8s-supplements.sh``
+
 |
 
 .. _cluster-configuration.nginx-ingress-configuration:

templates/config.template.toml

@@ -446,7 +446,7 @@ cluster_name = "rook-ceph" # •ᴗ•
 # storage class you have configured in the kubernetes.local_storage section (or
 # you should know what your are doing). Note that this is not the storage class
 # name that rook will provide.
-#mon_volume_storage_class = "local-storage"
+#mon_volume_storage_class = "local-storage"  # immutable when deployed
 
 # Enables rook to use the host network.
 #use_host_networking = false
@@ -496,7 +496,7 @@ cluster_name = "rook-ceph" # •ᴗ•
 #nosds = 3
 
 # The size of the storage backing each OSD.
-#osd_volume_size = "90Gi"
+#osd_volume_size = "90Gi"  # immutable when deployed
 
 # Enable the rook toolbox, which is a pod with ceph tools installed to
 # introspect the cluster state.
@@ -636,8 +636,8 @@ cluster_name = "rook-ceph" # •ᴗ•
 #thanos_storage_class = null
 # You can explicitly set the PV size for each component.
 # If left undefined, the helm chart defaults will be used
-#thanos_storegateway_size = null
-#thanos_compactor_size = null
+#thanos_storegateway_size = null  # immutable when deployed
+#thanos_compactor_size = null  # immutable when deployed
 
 # By default, the monitoring will capture all namespaces. If this is not
 # desired, the following switch can be turned off. In that case, only the