Merge pull request csi-addons#175 from red-hat-storage/sync_us--main

Syncing latest changes from upstream main for kubernetes-csi-addons

docs/encryptionkeyrotation.md

@@ -0,0 +1,150 @@
+# EncryptionKeyRotationJob
+
+EncryptionKeyRotationJob is a namespaced custom resource designed to rotate encryption keys on target volume.
+
+```yaml
+apiVersion: csiaddons.openshift.io/v1alpha1
+kind: EncryptionKeyRotationJob
+metadata:
+  name: sample-1
+spec:
+  target:
+    persistentVolumeClaim: pvc-1
+  backOffLimit: 10
+  retryDeadlineSeconds: 900
+  timeout: 600
+```
+
++ `target` represents volume target on which the operation will be performed.
+  + `persistentVolumeClaim` contains a string indicating the name of `PersistentVolumeClaim`.
++ `backOfflimit` specifies the number of retries before marking key rotation operation as failed. If not specified, defaults to 6. Maximum allowed value is 60 and minimum allowed value is 0.
++ `retryDeadlineSeconds` specifies the duration in seconds relative to the start time that the operation may be retried; value must be positive integer. If not specified, defaults to 600 seconds. Maximum allowed value is 1800.
++ `timeout` specifies the timeout in seconds for the grpc request sent to the CSI driver. If not specified, defaults to global timeout of 3 minutes. Minimum allowed value is 60.
+
+## EncryptionKeyRotationCronJob
+
+The `EncryptionKeyRotationCronJob` offers an interface very similar to the [Kubernetes
+`batch/CronJob`][batch_cronjob]. With the `schedule` attribute, the CSI-Addons
+Controller will create a `EncryptionKeyRotationJob` at the requested time and interval.
+The Kubernetes documentation for the `CronJob` and the [Golang cron
+package][go_cron] explain the format of the `schedule` attribute in more
+detail.
+
+```yaml
+apiVersion: csiaddons.openshift.io/v1alpha1
+kind: EncryptionKeyRotationCronJob
+metadata:
+  name: encryptionkeyrotationcronjob-sample
+spec:
+  concurrencyPolicy: Forbid
+  failedJobsHistoryLimit: 1
+  jobTemplate:
+    spec:
+      backOffLimit: 6
+      retryDeadlineSeconds: 600
+      target:
+        persistentVolumeClaim: data-pvc
+  schedule: '@weekly'
+  successfulJobsHistoryLimit: 3
+```
+
++ `concurrencyPolicy` describes what happens when a new `EncryptionKeyRotationJob` is
+  scheduled by the `EncryptionKeyRotationCronJob`, while a previous `EncryptionKeyRotationJob` is
+  still running. The default `Forbid` prevents starting new job, whereas
+  `Replace` can be used to delete the running job (potentially in a failure
+  state) and create a new one.
++ `failedJobsHistoryLimit` keeps at most the number of failed
+  `EncryptionKeyRotationJobs` around for troubleshooting
++ `jobTemplate` contains the `EncryptionKeyRotationJob.spec` structure, which describes
+  the details of the requested `EncryptionKeyRotationJob` operation.
++ `schedule` is in the same [format as Kubernetes CronJobs][batch_cronjob] that
+  sets the and/or interval of the recurring operation request.
++ `successfulJobsHistoryLimit` can be used to keep at most number of successful
+  `EncryptionKeyRotationJob` operations.
+
+## Annotating PersistentVolumeClaims
+
+`EncryptionKeyRotationCronJob` CR can also be automatically created by adding
+`keyrotation.csiaddons.openshift.io/schedule: "@weekly"` annotation
+to PersistentVolumeClaim object.
+
+```console
+$ kubectl get pvc data-pvc
+NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
+data-pvc  Bound    pvc-f37b8582-4b04-4676-88dd-e1b95c6abf74   1Gi        RWO            default           20h
+
+$ kubectl annotate pvc data-pvc "keyrotation.csiaddons.openshift.io/schedule=@weekly"
+persistentvolumeclaim/data-pvc annotated
+
+$ kubectl get encryptionkeyrotationcronjobs.csiaddons.openshift.io
+NAME                    SCHEDULE    SUSPEND   ACTIVE   LASTSCHEDULE   AGE
+data-pvc-1642663516   @weekly                                     3s
+
+$ kubectl annotate pvc data-pvc "keyrotation.csiaddons.openshift.io/schedule=*/1 * * * *" --overwrite=true
+persistentvolumeclaim/data-pvc annotated
+
+$ kubectl get encryptionkeyrotationcronjobs.csiaddons.openshift.io
+NAME                  SCHEDULE    SUSPEND   ACTIVE   LASTSCHEDULE   AGE
+data-pvc-1642664617   */1 * * * *                                   3s
+```
+
++ Upon adding annotation as shown above, a `EncryptionKeyRotationCronJob` with name
+  `"<pvc-name>-xxxxxxx"` is created (pvc name suffixed with current time hash when
+  the job was created).
++ `schedule` value is in the same [format as Kubernetes CronJobs][batch_cronjob]
+  that sets the and/or interval of the recurring operation request.
++ Default schedule value `"@weekly"` is used if `schedule` value is empty or in invalid format.
++ `EncryptionKeyRotationCronJob` is recreated when `schedule` is modified and deleted when
+  the annotation is removed.
+
+[batch_cronjob]: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
+[go_cron]:
+    https://pkg.go.dev/github.com/robfig/cron/v3#hdr-CRON_Expression_Format
+
+## Annotating Namespace
+
+You can create `EncryptionKeyRotationCronJob` CR automatically by adding the
+`keyrotation.csiaddons.openshift.io/schedule: "@weekly"` and (optional)
+`keyrotation.csiaddons.openshift.io/drivers: drivernames` annotations to the
+Namespace object. This will only inspect new PersistentVolumeClaims in the
+Namespace for EncryptionKeyRotation operations. You must restart the controller if you
+want kubernetes-csi-addons to consider existing PersistentVolumeClaims.
+
+`drivernames` can be `,` separated driver names that supports EncryptionKeyRotation operations.
+
+```console
+$ kubectl get namespace default
+NAME      STATUS   AGE
+default   Active   5d2h
+
+$ kubectl annotate namespace default "keyrotation.csiaddons.openshift.io/schedule=@weekly"
+namespace/default annotated
+```
+
+**Note** Please note that the PersistentVolumeClaim annotation takes priority over Namespace
+annotation. The kubernetes-csi-addons only generates a `EncryptionKeyRotationCronJob` if the annotation
+exists on the Namespace. If an admin needs to modify or delete the annotation on the Namespace,
+they must perform the same action on all the PersistentVolumeClaims within that Namespace.
+
+## Annotating StorageClass
+
+You can create `EncryptionKeyRotationCronJob` CR automatically by adding the
+`keyrotation.csiaddons.openshift.io/schedule: "@weekly"` annotations to the StorageClass object.
+This will only affect new PersistentVolumeClaims created from this StorageClass for EncryptionKeyRotation
+operations. To include existing PersistentVolumeClaims for EncryptionKeyRotation operations, you must restart the controller. This will ensure that EncryptionKeyRotation annotations are added to the existing
+PersistentVolumeClaims and `EncryptionKeyRotationCronJob` resources are created for them.
+
+```console
+$ kubectl get storageclass rbd-sc
+NAME       PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
+rbd-sc     rbd.csi.ceph.com   Delete          Immediate           true                   5d2h
+
+$ kubectl annotate storageclass rbd-sc "keyrotation.csiaddons.openshift.io/schedule=@weekly"
+storageclass.storage.k8s.io/rbd-sc annotated
+```
+
+**Note** Please note that the PersistentVolumeClaim and Namespace annotation takes priority
+over StorageClass annotation. The kubernetes-csi-addons only generate a `EncryptionKeyRotationCronJob`
+if the annotation exists on the StorageClass. If an admin needs to modify or delete the
+annotation on the StorageClass, they must perform the same action on all the PersistentVolumeClaims
+created from this StorageClass.