Skip to content

Commit

Permalink
WIP
Browse files Browse the repository at this point in the history
  • Loading branch information
Feediver1 committed Oct 30, 2024
1 parent 8645794 commit e63834b
Showing 1 changed file with 94 additions and 9 deletions.
103 changes: 94 additions & 9 deletions modules/manage/partials/tiered-storage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -749,10 +749,10 @@ To convert an existing S3 bucket and its contents, you must:
To create a new KMS key in the AWS Console:
. In the AWS Console, search for “Key Management Service”.
. In AWS Console, search for “Key Management Service”.
. Click **Create a key**.
. On the Configure key page, select the Symmetric key type, then select Encrypt and decrypt.
. Click the **Advanced options** tab and configure Key material origin and Regionality as needed. For example, if you are using xref:manage/remote-read-replicas.adoc[Remote Read Replicas] or have Redpanda spanning across regions, select Multi-Region key.
. Click the **Advanced options** tab and configure Key material origin and Regionality as needed. For example, if you are using xref:manage:remote-read-replicas.adoc[Remote Read Replicas] or have Redpanda spanning across regions, select Multi-Region key.
. Click **Next**.
. On the Add labels page, specify an alias name and description for the key. Do not include sensitive information in these fields.
. Click **Next**.
Expand All @@ -764,21 +764,32 @@ To create a new KMS key in the AWS Console:
For more information, see the AWS documentation https://docs.aws.amazon.com/kms/latest/developerguide/create-symmetric-cmk.html[Creating a symmetric encryption KMS key^].
To configure the S3 bucket to use the new KMS key:
To configure the S3 bucket to use the new KMS key (and reduce KMS costs through caching):
. In the AWS Console, search for "S3".
. In AWS Console, search for "S3".
. Select the bucket used by Redpanda.
. Click the **Properties** tab.
. In Default encryption, click **Edit**.
. For Encryption type, select “Server-side encryption with AWS Key Management Service keys (SSE-KMS)”.
. Locate and select your AWS KMS key ARN identifier.
. Click **Save changes**.
For more information, see the AWS documentation https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-bucket-key.html[Enable an S3 Bucket Key for an existing bucket^].
(Optional) To re-encrypt existing data using the new KMS key:
Existing data in your S3 bucket continues to be read using the AWS-managed key, while moving forward, new objects are encrypted using the new KMS key. If you wish to re-encrypt all S3 bucket data to use the KMS key, run the following AWS CLI command:
[,bash]
----
aws s3 cp s3://{BUCKET_NAME}/ s3://{BUCKET_NAME}/ --recursive --sse-kms-key-id {KMS_KEY_ARN} --sse aws:kms
----
. Be sure to select "Server-side encryption with AWS Key Management Service keys (SSE-KMS)" as the encryption type when you edit the Properties for default encryption.
. Select your AWS KMS Key either from the dropdown or using the key ARN identifier.
. Be sure to enable the S3 bucket key, as it reduces KMS costs through caching.
For more information, see the AWS documentation https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-bucket-key.html[Enable an S3 Bucket Key for an existing bucket^].
--
Google Cloud Storage::
+
--
Configure access to Google Cloud Storage with either an IAM role attached to the instance or with access keys.
Configure access to Google Cloud Storage with either an IAM role attached to the instance, with access keys, or customer-managed keys.
To configure access to Google Cloud Storage with an IAM role:
Expand Down Expand Up @@ -820,6 +831,80 @@ Replace `<placeholders>` with your own values.
+
CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value. To reset a property to its default value, run `rpk cluster config force-reset <config-name>` or remove that line from the cluster configuration with `rpk cluster config edit`.
To configure the Google Cloud bucket used by Redpanda Tiered Storage to leverage a customer-managed key using the Cloud Key Management Service API instead of the default Google-managed key, you must:
. Create a KMS key.
. Configure the bucket to use the KMS key.
. Optionally, re-encrypt existing data with the new KMS key.
To manage Google Cloud KMS using the command line, first install or upgrade to the latest version of https://cloud.google.com/sdk/docs/install[Google Cloud CLI^].
To create a KMS key:
. In Google Cloud Console, search for "Cloud Key Managment Service API". Click **Enable** if the service is not already enabled.
. Using the Google Cloud CLI, create a new keyring in the https://cloud.google.com/kms/docs/locations^[region] where the bucket used by Redpanda is located. Note that region is case-sensitive.
+
[,bash, indent]
----
gcloud kms keyrings create "redpanda-keyring" --location="{REGION}"
----
+
. Create a new key for the keyring in the same region as the bucket:
+
[,bash, indent]
----
gcloud kms keys create "redpanda-key" \
--location="{REGION}" \
--keyring="redpanda-keyring" \
--purpose="encryption"
----
+
. Get the key identifier:
+
[,bash]
----
gcloud kms keys list \
--location="REGION" \
--keyring="redpanda-keyring"
----
+
The result should look like the following. Be sure to record the name, as this is used to assign and manage the key, and use this as the {KEY_RESOURCE} placeholder in subsequent commands.
+
[,bash]
----
NAME PURPOSE ALGORITHM PROTECTION_LEVEL LABELS PRIMARY_ID PRIMARY_STATE
projects/{PROJECT_NAME}/locations/us/keyRings/redpanda-keyring/cryptoKeys/redpanda-key ENCRYPT_DECRYPT GOOGLE_SYMMETRIC_ENCRYPTION SOFTWARE 1 ENABLED
----
To configure the GCP bucket to use the KMS key:
. Assign the key to a service agent:
+
[,bash]
----
gcloud storage service-agent \
--project={PROJECT_ID_STORING_OBJECTS} \
--authorize-cmek={KEY_RESOURCE}
----
+
. Set the bucket default encryption key to the KMS key:
+
[,bash]
----
gcloud storage buckets update gs://{BUCKET_NAME} \
--default-encryption-key={KEY_RESOURCE}
----
(Optional) To re-encrypt existing data using the new KMS key:
Existing data in the bucket will continue to be read using the Google-managed key, while moving forward, new objects are encrypted using the new KMS key. If you wish to re-encrypt all data in the bucket to use the KMS key, run the following command:
[,bash]
----
gcloud storage objects update gs://{BUCKET_NAME}/ --recursive \
--encryption-key={KEY_RESOURCE}
----
--
Microsoft ABS/ADLS::
+
Expand Down

0 comments on commit e63834b

Please sign in to comment.