Mark immutable config settings as such

A few of the yk8s config settings in `k8s-service-layer.prometheus` and
 `k8s-service-layer.rook` are actually immutable because they feed into
 the currently immutable `volumeClaimTemplate` field of Kubernetes objects.

Example: The `k8s-service-layer.prometheus.thanos_storegateway_size` yk8s
 config setting feeds into the `storegateway.persistence.size` parameter of the
 bitnami/thanos helm chart which feeds into the `volumeClaimTemplate` field of
 the `thanos-storegateway` statefulset created. That field is currently
 immutable making helm updates with the parameter fail.

Related-to: kubernetes/enhancements#4650

docs/_releasenotes/1498.fix.mark-immutable-config-settings-as-such

@@ -0,0 +1,11 @@
+Marked the following immutable config setting as such:
+- ``k8s-service-layer.prometheus.thanos_storegateway_size``
+- ``k8s-service-layer.prometheus.thanos_storegateway_size``
+- ``k8s-service-layer.prometheus.thanos_compactor_size``
+- ``k8s-service-layer.rook.mon_volume_storage_class``
+- ``k8s-service-layer.rook.mon_volume_size``
+- ``k8s-service-layer.rook.osd_volume_size``
+- ``k8s-service-layer.rook.osd_storage_class``
+
+A manual workaround to change these settings nonetheless
+is outlined in the docs.

nix/yk8s/k8s-supplements/monitoring.nix

@@ -163,6 +163,17 @@ in {
         k8s-service-layer.prometheus.thanos_storage_class = "SOME_STORAGE_CLASS";
         k8s-service-layer.prometheus.thanos_storegateway_size = "XGi";
         k8s-service-layer.prometheus.thanos_compactor_size = "YGi";
+
+      .. _cluster-configuration.prometheus-configuration.updating-immutable-options:
+
+      Updating immutable options
+      """"""""""""""""""""""""""
+
+      Some options are immutable when deployed.
+      If you want to change them nonetheless, follow these manual steps:
+      1. Increase the size of the corresponding PVC
+      2. Delete the stateful set: ``kubectl delete -n monitoring sts --cascade=false thanos-<storegateway|compactor>``
+      3. Re-deploy it with the LCM: ``AFLAGS="--diff --tags thanos" bash managed-k8s/actions/apply-k8s-supplements.sh``
     '';
 
     install = mkOption {
@@ -287,6 +298,8 @@ in {
       description = ''
         You can explicitly set the PV size for each component.
         If left undefined, the helm chart defaults will be used
+
+        Immutable when deployed. (See also `cluster-configuration.prometheus-configuration.updating-immutable-options`_)
       '';
       type = with types; nullOr k8sSize;
       default = null;
@@ -296,6 +309,8 @@ in {
       description = ''
         You can explicitly set the PV size for each component.
         If left undefined, the helm chart defaults will be used
+
+        Immutable when deployed. (See also `cluster-configuration.prometheus-configuration.updating-immutable-options`_)
       '';
       type = with types; nullOr k8sSize;
       default = null;

nix/yk8s/k8s-supplements/rook.nix

@@ -62,6 +62,17 @@ in {
         ``kubernetes.local_storage.dynamic.enabled`` or
         ``kubernetes.local_storage.static.enabled`` local
         storage (or both) (see :ref:`storage configuration <cluster-configuration.storage-configuration>`).
+
+      .. _cluster-configuration.rook-configuration.updating-immutable-options:
+
+      Updating immutable options
+      """"""""""""""""""""""""""
+
+      Some options are immutable when deployed.
+      If you want to change them nonetheless, follow these manual steps:
+      1. Increase the size of the corresponding PVC
+      2. Delete the stateful set: ``kubectl delete -n monitoring sts --cascade=false <statefulset_name>``
+      3. Re-deploy it with the LCM: ``AFLAGS="--diff --tags rook --tags rook_v2" bash managed-k8s/actions/apply-k8s-supplements.sh``
     '';
 
     # If kubernetes.storage.rook_enabled is enabled, rook will be installed.
@@ -114,6 +125,9 @@ in {
     };
 
     mon_volume_size = mkOption {
+      description = ''
+        Immutable when deployed. (See also `_cluster-configuration.rook-configuration.updating-immutable-options`_)
+      '';
       type = k8sSize;
       default = "10Gi";
     };
@@ -124,6 +138,8 @@ in {
         storage class you have configured in the kubernetes.local_storage section (or
         you should know what your are doing). Note that this is not the storage class
         name that rook will provide.
+
+        Immutable when deployed. (See also `_cluster-configuration.rook-configuration.updating-immutable-options`_)
       '';
       type = types.str;
       default = config.yk8s.kubernetes.local_storage.dynamic.storageclass_name;
@@ -246,13 +262,18 @@ in {
     };
 
     osd_storage_class = mkOption {
+      description = ''
+        Immutable when deployed. (See also `_cluster-configuration.rook-configuration.updating-immutable-options`_)
+      '';
       type = types.str;
       default = "csi-sc-cinderplugin";
     };
 
     osd_volume_size = mkOption {
       description = ''
         The size of the storage backing each OSD.
+
+        Immutable when deployed. (See also `_cluster-configuration.rook-configuration.updating-immutable-options`_)
       '';
       type = k8sSize;
       default = "90Gi";