Doc 2599 - Configure Redpanda with a Customer-Managed Key

modules/manage/pages/kubernetes/tiered-storage/k-tiered-storage.adoc

@@ -1,10 +1,9 @@
 = Use Tiered Storage in Kubernetes
 :description: Configure your Redpanda cluster to offload log segments to object storage and save storage costs.
-:page-context-links: [{"name": "Linux", "to": "manage:tiered-storage.adoc" },{"name": "Kubernetes", "to": "manage:kubernetes/storage/tiered-storage/k-tiered-storage.adoc" } ]
 :page-categories: Management, High Availability, Data Replication
 :env-kubernetes: true
-:tags: ["Kubernetes", "Helm configuration"]
+:page-toclevels: 4
 :page-aliases: manage:kubernetes/tiered-storage.adoc, manage:kubernetes/storage/tiered-storage.adoc, manage:kubernetes/data-archiving.adoc, manage:kubernetes/storage/data-archiving.adoc, manage:kubernetes/storage/tiered-storage/k-data-archiving.adoc, manage:kubernetes/storage/tiered-storage/k-tiered-storage.adoc
 
 
-include::manage:partial$tiered-storage.adoc[]
+include::manage:partial$tiered-storage.adoc[]

modules/manage/partials/tiered-storage.adoc

@@ -58,9 +58,12 @@ TIP: If deploying Redpanda on an AWS Auto-Scaling group (ASG), keep in mind that
 
 You can configure access to Amazon S3 with either an IAM role attached to the instance or with access keys.
 
-===== Use IAM roles
+[TIP]
+====
+If you need to manage and store encryption keys separately from your cloud provider, you can <<configure-access-to-aws-s3-using-an-aws-kms-key,configure access to an AWS S3 bucket that Redpanda Tiered Storage uses to leverage your AWS KMS key (SSE-KMS)>> instead of the default AWS S3-managed key (SSE-S3). This option enables you to segregate data from different teams or departments and remove that data at will by removing the encryption keys.
+====
 
-To configure access to an S3 bucket with an IAM role:
+===== **Configure access with an IAM role**
 
 . Configure an xref:manage:security/iam-roles.adoc#configuring-iam-roles[IAM role].
 
@@ -138,9 +141,7 @@ Replace the following placeholders:
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
 
-===== Use access keys
-
-To configure access to an S3 bucket with access keys instead of an IAM role:
+===== **Configure access with access keys**
 
 . Grant an IAM user the following permissions to read and create objects in your buckets:
 - `GetObject`
@@ -258,13 +259,13 @@ Replace the following placeholders:
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
 
-==== Google Cloud Storage
+include::manage:partial$tiered-storage/aws-kms-key.adoc[leveloffset=+4]
 
-You can configure access to Google Cloud Storage with either an IAM role attached to the instance or with access keys.
+==== Google Cloud Storage
 
-===== Use IAM roles
+Configure access to Google Cloud Storage with either an IAM role attached to the instance, with access keys, or with customer-managed keys.
 
-To configure access to Google Cloud Storage with an IAM role:
+===== **Configure access with an IAM role**
 
 . Configure an xref:manage:security/iam-roles.adoc#configuring-iam-roles[IAM role].
 . Override the following required cluster properties in the Helm chart:
@@ -343,7 +344,7 @@ Replace the following placeholders:
 
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
 
-===== Use access keys
+===== **Configure access with access keys**
 
 To configure access to Google Cloud Storage with access keys instead of an IAM role:
 
@@ -462,15 +463,15 @@ Replace the following placeholders:
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
 
+include::manage:partial$tiered-storage/gcp-kms-key.adoc[leveloffset=+4]
+
 ==== Microsoft ABS/ADLS
 
 You can configure access to Azure Blob Storage with either account access keys or Azure's managed identities system to securely interact with Azure Blob Storage. Account access keys, as static credentials, require manual management and vigilant security practices to prevent breaches due to their unchanging nature. In contrast, managed identities provide a more secure and maintenance-free solution by automating credential management and rotation, though they are exclusive to the Azure ecosystem.
 
 include::manage:partial$azure-blob-limitations.adoc[]
 
-===== Use managed identities
-
-To configure access to an Azure container with a managed identity in AKS:
+===== **Configure access with a managed identity**
 
 . Configure an xref:manage:security/iam-roles.adoc#configuring-iam-roles[Azure managed identity].
 . Override the following required cluster properties in the Helm chart:
@@ -570,9 +571,7 @@ NOTE: The `serviceAccount` annotations and the `statefulset` Pod labels are esse
 
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.
 
-===== Use access keys
-
-To configure access to ABS/ADLS with account access keys:
+===== **Configure access with account access keys**
 
 . Get an account access key for the Azure container that Redpanda will run on. For information on how to view your account access keys, see the https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#view-account-access-keys[Azure documentation^].
 . Create a Secret in which to store the access key.
@@ -688,9 +687,14 @@ Amazon S3::
 
 TIP: If deploying Redpanda on an AWS Auto-Scaling group (ASG), keep in mind that the ASG controller terminates nodes and spins up replacements if the nodes saturate and are unable to heartbeat the controller (based on the EC2 health check). For more information, see the https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html#types-of-instance-status-checks[AWS documentation^]. Redpanda recommends deploying on Linux or Kubernetes. For more information, see xref:deploy:deployment-option/self-hosted/index.adoc[Deploy Redpanda].
 
-Configure access to Amazon S3 with either an IAM role attached to the instance or with access keys.
+Configure access to Amazon S3 with either an IAM role attached to the instance, with access keys, or with customer-managed keys.
+
+[TIP]
+====
+If you need to manage and store encryption keys separately from your cloud provider, you can <<configure-access-to-aws-s3-using-an-aws-kms-key,configure access to an AWS S3 bucket that Redpanda Tiered Storage uses to leverage your AWS KMS key (SSE-KMS)>> instead of the default AWS S3-managed key (SSE-S3). This option enables you to segregate data from different teams or departments and remove that data at will by removing the encryption keys.
+====
 
-To configure access to an S3 bucket with an IAM role:
+**Configure access with an IAM role**
 
 . Configure an xref:manage:security/iam-roles.adoc#configuring-iam-roles[IAM role].
 . Run the `rpk cluster config edit` command, then edit the following required properties:
@@ -707,7 +711,7 @@ Replace `<placeholders>` with your own values.
 
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value. To reset a property to its default value, run `rpk cluster config force-reset <config-name>` or remove that line from the cluster configuration with `rpk cluster config edit`.
 
-To configure access to an S3 bucket with access keys instead of an IAM role:
+**Configure access with access keys**
 
 . Grant a user the following permissions to read and create objects on the bucket to be used with the cluster (or on all buckets): `GetObject`, `DeleteObject`, `PutObject`, `PutObjectTagging`, `ListBucket`.
 . Copy the access key and secret key for the `cloud_storage_access_key` and `cloud_storage_secret_key` cluster properties.
@@ -726,13 +730,15 @@ Replace `<placeholders>` with your own values.
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value. To reset a property to its default value, run `rpk cluster config force-reset <config-name>` or remove that line from the cluster configuration with `rpk cluster config edit`.
 
+include::manage:partial$tiered-storage/aws-kms-key.adoc[leveloffset=+3]
+
 --
 Google Cloud Storage::
 +
 --
-Configure access to Google Cloud Storage with either an IAM role attached to the instance or with access keys.
+Configure access to Google Cloud Storage with either an IAM role attached to the instance, with access keys, or with customer-managed keys.
 
-To configure access to Google Cloud Storage with an IAM role:
+**Configure access with an IAM role**
 
 . Configure an xref:manage:security/iam-roles.adoc#configuring-iam-roles[IAM role].
 . Run the `rpk cluster config edit` command, then edit the following required properties:
@@ -750,7 +756,7 @@ Replace `<placeholders>` with your own values.
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value. To reset a property to its default value, run `rpk cluster config force-reset <config-name>` or remove that line from the cluster configuration with `rpk cluster config edit`.
 
-To configure access to Google Cloud Storage with access keys instead of an IAM role:
+**Configure access with access keys**
 
 . Choose a uniform access control when you create the bucket.
 . Use a Google-managed encryption key.
@@ -772,14 +778,16 @@ Replace `<placeholders>` with your own values.
 +
 CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value. To reset a property to its default value, run `rpk cluster config force-reset <config-name>` or remove that line from the cluster configuration with `rpk cluster config edit`.
 
+include::manage:partial$tiered-storage/gcp-kms-key.adoc[leveloffset=+3]
+
 --
 Microsoft ABS/ADLS::
 +
 --
 
 include::manage:partial$azure-blob-limitations.adoc[]
 
-To configure access to an Azure container with a managed identity:
+**Configure access with managed identities**
 
 . Configure an xref:manage:security/iam-roles.adoc#azure-prerequisites[Azure managed identity].
 +
@@ -805,7 +813,7 @@ cloud_storage_azure_container: <container-name>
 +
 Replace `<placeholders>` with your own values.
 
-To configure access to Azure Blob Storage with account access keys:
+**Configure access with account access keys**
 
 . Copy an account access key for the Azure container you want Redpanda to use and enter it in the `cloud_storage_azure_shared_key` property. For information on how to view your account access keys, see the https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#view-account-access-keys[Azure documentation^].
 . Run the `rpk cluster config edit` command, then edit the following required properties:

modules/manage/partials/tiered-storage/aws-kms-key.adoc

@@ -0,0 +1,71 @@
+ifndef::env-kubernetes[]
+[discrete]
+endif::[]
+= **Configure access with an AWS KMS key**
+
+When there are strict data compliance requirements and you must manage and store encryption keys separate from your cloud provider, you can configure an Amazon S3 bucket that Tiered Storage can use to leverage your customer-provided key (SSE-KMS) instead of the default AWS-managed key (SSE-S3).
+
+To convert an existing S3 bucket and its contents, you must:
+
+. Create a new KMS key.
+. Configure the S3 bucket to use the new KMS key.
+. (Optional) Re-encrypt existing objects to use the new KMS key.
+
+[NOTE]
+====
+ifdef::env-cloud[]
+You cannot configure a cloud-provider managed encryption key at the topic level or in Redpanda Cloud Dedicated clusters.
+endif::[]
+ifndef::env-cloud[]
+You cannot configure a cloud provider-managed encryption key at the topic level.
+endif::[]
+
+For topic-level control, each CLI Get or Put for a partition must use the correct key as configured on the topic.
+====
+
+ifndef::env-kubernetes[]
+[discrete]
+endif::[]
+== **Prerequisites**
+
+- The user configuring S3 bucket encryption must be assigned the Key admin permission. Without this permission, the user is unable to re-encrypt existing bucket objects to use the KMS key.
+- The S3 bucket must be assigned the Key user permission. Without this permission, Redpanda is unable to write new objects to Tiered Storage.
+- If you intend to retroactively re-encrypt existing data with the new KMS key, store the ARN identifier of the key upon creation. It is required for AWS CLI commands.
+
+To create a new KMS key in the AWS Console:
+
+. In AWS Console, search for “Key Management Service”.
+. Click **Create a key**.
+. On the Configure key page, select the **Symmetric** key type, then select **Encrypt and decrypt**.
+. Click the **Advanced options** tab and configure Key material origin and Regionality as needed. For example, if you are using xref:manage:remote-read-replicas.adoc[Remote Read Replicas] or have Redpanda spanning across regions, select **Multi-Region key**.
+. Click **Next**.
+. On the Add labels page, specify an alias name and description for the key. Do not include sensitive information in these fields.
+. Click **Next**.
+. On the Define key administrative permissions page, specify a user who can administer this key through the KMS API.
+. Click **Next**.
+. On the Define key usage permissions page, assign the S3 bucket as a Key user. This is required for the S3 bucket to encrypt and decrypt.
+. Click **Next**.
+. Review your KMS key configuration and click **Finish**.
+
+For more information, see the https://docs.aws.amazon.com/kms/latest/developerguide/create-symmetric-cmk.html[AWS documentation^].
+
+To configure the S3 bucket to use the new KMS key (and reduce KMS costs through caching):
+
+. In AWS Console, search for "S3".
+. Select the bucket used by Redpanda.
+. Click the **Properties** tab.
+. In Default encryption, click **Edit**.
+. For Encryption type, select “Server-side encryption with AWS Key Management Service keys (SSE-KMS)”.
+. Locate and select your AWS KMS key ARN identifier.
+. Click **Save changes**.
+
+(Optional) To re-encrypt existing data using the new KMS key:
+
+Existing data in your S3 bucket continues to be read using the AWS-managed key, while new objects are encrypted using the new KMS key. If you wish to re-encrypt all S3 bucket data to use the KMS key, run:
+
+[,bash]
+----
+aws s3 cp s3://{BUCKET_NAME}/ s3://{BUCKET_NAME}/ --recursive --sse-kms-key-id {KMS_KEY_ARN} --sse aws:kms
+----
+
+For more information, see the https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-bucket-key.html[AWS documentation^].

modules/manage/partials/tiered-storage/gcp-kms-key.adoc

@@ -0,0 +1,78 @@
+ifndef::env-kubernetes[]
+[discrete]
+endif::[]
+= **Configure access with a KMS key**
+
+To configure the Google Cloud bucket used by Redpanda Tiered Storage to leverage a customer-managed key using the Cloud Key Management Service API instead of the default Google-managed key, you must:
+
+. Create a KMS key.
+. Configure the bucket to use the KMS key.
+. Optionally, re-encrypt existing data with the new KMS key.
+
+To manage Google Cloud KMS using the command line, first install or upgrade to the latest version of https://cloud.google.com/sdk/docs/install[Google Cloud CLI^].
+
+To create a KMS key:
+
+. In Google Cloud Console, search for "Cloud Key Managment Service API". Click **Enable** if the service is not already enabled.
+. Using the Google Cloud CLI, create a new keyring in the https://cloud.google.com/kms/docs/locations^[region] where the bucket used by Redpanda is located. Note that region is case sensitive.
++
+[,bash, indent]
+----
+gcloud kms keyrings create "redpanda-keyring" --location="{REGION}"
+----
++
+. Create a new key for the keyring in the same region as the bucket:
++
+[,bash, indent]
+----
+gcloud kms keys create "redpanda-key" \
+  --location="{REGION}" \
+  --keyring="redpanda-keyring" \
+  --purpose="encryption"
+----
++
+. Get the key identifier:
++
+[,bash]
+----
+gcloud kms keys list \
+  --location="REGION" \
+  --keyring="redpanda-keyring"
+----
++
+The result should look like the following. Be sure to store the name, as this is used to assign and manage the key. Use this as the \{KEY_RESOURCE} placeholder in subsequent commands.
++
+[,bash]
+----
+NAME      PURPOSE      ALGORITHM      PROTECTION_LEVEL    LABELS    PRIMARY_ID    PRIMARY_STATE
+projects/{PROJECT_NAME}/locations/us/keyRings/redpanda-keyring/cryptoKeys/redpanda-key
+ENCRYPT_DECRYPT  GOOGLE_SYMMETRIC_ENCRYPTION  SOFTWARE          1           ENABLED
+----
+
+To configure the GCP bucket to use the KMS key:
+
+. Assign the key to a service agent:
++
+[,bash]
+----
+gcloud storage service-agent \
+  --project={PROJECT_ID_STORING_OBJECTS} \
+  --authorize-cmek={KEY_RESOURCE}
+----
++
+. Set the bucket default encryption key to the KMS key:
++
+[,bash]
+----
+gcloud storage buckets update gs://{BUCKET_NAME} \
+  --default-encryption-key={KEY_RESOURCE}
+----
+
+(Optional) To re-encrypt existing data using the new KMS key:
+
+Existing data in the bucket continues to be read using the Google-managed key, while new objects are encrypted using the new KMS key. If you wish to re-encrypt all data in the bucket to use the KMS key, run:
+[,bash]
+----
+gcloud storage objects update gs://{BUCKET_NAME}/ --recursive \
+  --encryption-key={KEY_RESOURCE}
+----