Support Vault via vault-k8s #846

ansd · 2021-09-17T17:07:32Z

Before this PR, the cluster-operator was always creating the credentials of the default (admin) user putting them into a K8s Secret object. TLS is supported by the user providing a K8s Secret object containing certificate and private key.

However, some users want to store credentials outside of K8s in external secret stores such as HashiCorp Vault, AWS Secrets Manager, Google Secrets Manager or Azure Key Vault. Benefits of doing so include:

all credentials in a single store
encrypt data at rest
auditing
third-party rotation features

As explained in this KubeCon talk there 4 different approaches in K8s to consume external secrets:

Direct API
Controller to mirrors secrets in K8s
Sidecar + MutatingWebhookConfiguration
Secrets Store CSI Driver

In this PR, we take the 3rd approach (Sidecar + MutatingWebhookConfiguration) integrating with Vault using vault-k8s. If spec.secretBackend.vault is configured in the RabbimqCluster CRD, no K8s Secret will be created for the default user credentials. Instead a Vault init container + sidecar fetch credentials from Vault.

In future, we might (no promises) support approaches 2. Controller to mirrors secrets in K8s as for example requested in #840 and 4. Secrets Store CSI Driver. The latter seems to be the future proof K8s native way of fetching external secrets.

This PR adds support for 3 features:

To create a cluster the default user's credentials should come from Vault's secret #822: When creating a RabbitmqCluster, the default user credentials are fetched from Vault. The assumption is that the credentials were already created in Vault (via some cluster-operator external mechanism). The cluster-operator will not create a K8s Secret for the default user. Instead, a Vault init container will fetch the credentials and mount them to be used in the RabbitMQ container.
Update default user's password in RabbitMQ when it has changed in Vault #823: Password rotation without restarting Pods. When the default user password changes in Vault, the Vault sidecar will update the password in the /etc/rabbitmq/conf.d/11-default_user.conf file. However, RabbitMQ cannot pick up config file changes on the fly. Therefore, some component need to update the password on the RabbitMQ server. The Vault sidecar container does not seem to contain the tools to HTTP PUT to the RabbitMQ Management API or to kubectl exec into the RabbitMQ container. Since we do not want to change the default Vault image, a 2nd side car can optionally be deployed. It contains a single Go binary that watches 11-default_user.conf for changes. If a new password is detected, it HTTP PUT to the RabbitMQ management API updating the password server side and copies the password into /var/lib/rabbitmq/.rabbitmqadmin.conf to be used for the rabbitmqadmin CLI. Username rotation is not supported.
To create a cluster with TLS enabled, the certificate and private key should come from Vault's secret #824: To enable TLS, RabbitMQ server certificates are provided on-demand by Vault PKI Engine instead of being pre-put into a K8s Secret. The RabbitMQ pod uses a short-lived leaf certificate (aligning with Vault's philosophy of short-lived secrets). After each RabbitMQ pod start-up, a new certificate is issued by Vault. The private key is never stored in Vault. Before the short-lived server certificate expires, vault sidecar container will request new certificate putting it into /etc/rabbitmq-tls/ where it will be picked up on-the-fly by the Erlang VM without the need to restart the pod.

No changes are done to the Erlang cookie: it will still be created by the cluster-operator and stored in a K8s Secret.

Tests:

Unit tests were added for the new admin-password-updater sidecar container.
Unit tests in the cluster-operator
End-to-end tests are provided in the docs/examples/vault-default-user and docs/examples/vault-tls directories covering the above 3 features and will be run in our Concourse pipeline. The setup.sh scripts give users a high level idea of what needs to be setup on Vault server side (although the examples use only Vault server dev mode).

Most of the work in this PR was done by @MarcialRosales 🙌

This PR is a draft because the following TODOs are left:

Get feedback: @twhite0 would you be interested in testing the changes?
Re-test everything.
Add docs: specifically READMEs in the docs/examples/vault-*/ directories.
Merge squashing all commits.
Merge Add Vault docs rabbitmq-website#1272

since it shouldn't be readable by everyone. Before this commit: ls -al /etc/rabbitmq/conf.d/ drwxrwsrwt 2 root rabbitmq 100 Sep 7 14:52 . drwxrwxrwx 1 rabbitmq rabbitmq 4096 Sep 7 14:30 .. -rw-r--r-- 1 root rabbitmq 604 Sep 7 14:29 10-operatorDefaults.conf -rw-r--r-- 1 _apt rabbitmq 45 Sep 7 14:52 11-default_user.conf -rw-r--r-- 1 root rabbitmq 51 Sep 7 14:29 90-userDefinedConfiguration.conf After this commit: ls -al /etc/rabbitmq/conf.d/ drwxrwsrwt 2 root rabbitmq 100 Sep 8 07:19 . drwxrwxrwx 1 rabbitmq rabbitmq 4096 Sep 8 07:19 .. -rw-r--r-- 1 root rabbitmq 604 Sep 8 07:18 10-operatorDefaults.conf -rw-r----- 1 rabbitmq rabbitmq 45 Sep 8 07:19 11-default_user.conf -rw-r--r-- 1 root rabbitmq 51 Sep 8 07:18 90-userDefinedConfiguration.conf

because this is needed when inter-node TLS is enabled or when scraping metrics via TLS. Although each RabbitMQ pod will request its own certificate, we still include all pod hostnames into every certificate because the index will only be known at runtime (not cluster-operator deploy time) and we can't use K8s downward API here since the labels must be set correctly when the vault-agent init container runs before all other containers.

Add unit tests for vault annotations TODO: add unit tests for vault commands and mounts

if default user secret is stored in Vault

Before this commit: 2021-09-09T13:27:27.312Z [INFO] (runner) rendered "(dynamic)" => "/etc/rabbitmq-tls//tls.key" 2021-09-09T13:27:27.313Z [INFO] (runner) rendered "(dynamic)" => "/etc/rabbitmq-tls//tls.crt"

Pending: Unit test

and fix mountpath

since it serves as kind of docs for our uses to give a high level idea of what needs to be set up on Vault side. Rename some fields to be more succinct.

Some K8s clusters (e.g. kind) require to set the issuer. Do not set the issuer on GKE.

ansd · 2021-09-21T11:11:02Z

@twhite0 thank you for your followups, these are excellent questions.

What was the driver for adding in the SecurityContext work in the Statefulset.

We set a PodSecurityContext to run as user 999 and FSGroup 999:

			SecurityContext: &corev1.PodSecurityContext{
				FSGroup:   &rabbitmqUID,
				RunAsUser: &rabbitmqUID,
			},

We just removed the other container SecurityContext via 6b529d9 since they are indeed not needed because they were only setting RunAsUser: &rabbitmqUID which was already set in the PodSecurityContext.

The driver to set the PodSecurityContext is to run processes as RabbitMQ user (as opposed to running as root user).
FSGroup ensures that mounted volumes belong to the RabbitMQ group.

For OpenShift, this is not needed since the runtime will assign arbitrary user IDs to the containers as documented in https://www.rabbitmq.com/kubernetes/operator/using-on-openshift.html#arbitrary-user-ids.

We found a couple more annotations which we needed to add into the Statefulset override section of our RabbitmqCluster definition to get things cooking.

You can now set arbitrary Vault annotations as documented in

cluster-operator/docs/examples/vault-default-user/rabbitmq.yaml

Lines 10 to 13 in 2f0e49c

    
           # Optionally, set Vault annotations as listed in 
        
           # https://www.vaultproject.io/docs/platform/k8s/injector/annotations 
        
           annotations: 
        
             vault.hashicorp.com/template-static-secret-render-interval: "15s"

User set Vault annotations will override cluster-operator-set Vault annotations.

twhite0 · 2021-09-21T19:42:20Z

@ansd: Thanks for the response and quick pivots as a result of my questions. We were able to pull the latest changes and see great results.

On OpenShift 4.6.26
Seeded our Vault environment with admin credentials + PKI backend and configured our vault injector.
Deployed a RMQ cluster pulling in both admin credentials and certs. ✔️
Deployed a RMQ cluster with the password rotation sidecar enabled. ✔️
We're still needing to work through how to automate the admin credential rotation process.
We're working through how to automate the Vault provisioning per cluster.

Let us know if there's anything else you'd like us to test or comment on.

ansd · 2021-09-22T07:30:03Z

Thanks a lot @twhite0 for your feedback. That's perfect!

@MarcialRosales had some good refactoring suggestions yesterday, he will change Vault field names slightly. Thereafter we should be ready to merge.

Vault attributes renaming: PathDefaultUser -> DefaultUserPath PathCertificate -> PKIIssuerPath Password rotation enabled by default: By default, the cluster operator deploys a sidecar container with a default image name It is possible to override the image by setting `DefaultUserUpdaterImage` Removed SecretBackend `CredentialUpdaterImage` in favor of having a dedicated image to control how to rotate passwords when Vault is enabled

api/v1beta1/rabbitmqcluster_types.go

config/crd/bases/rabbitmq.com_rabbitmqclusters.yaml

docs/api/rabbitmq.com.ref.asciidoc

docs/examples/vault-default-user/rabbitmq.yaml

internal/resource/statefulset_test.go

and allow to disable the sidecar container by setting image name to empty string.

twhite0 · 2021-09-23T20:33:10Z

@ansd:

We noticed the status is defaulting to supplying secretReference information.

credit: @ssheth1

…er resource if default user credentials come from Vault

controllers/rabbitmqcluster_controller_test.go

to rabbitmq/default-user-credential-updater to allow for easier and independant versioning from cluster-operator. We do not want to have the admin-password-updater image the same version as the cluster-operator image.

so that new repo name matches image name, container name, and entrypoint.

ansd · 2021-09-28T12:06:07Z

@ssheth1 @twhite0 very good spot, thank you :) This is fixed now.

The attributes were vault.defaultUserPath and vault.tls.pkiIssuerPath

ChunyiLyu and others added 30 commits September 17, 2021 16:27

Add vault properties to rmq spec

4da7dd9

Ensure vault anotations added when enabled

0e9ab2f

Append vault annotations right after adding other annotations

69164d9

Mount projected secrets for default-user only when vault is disabled

730ef6a

WIP Configure podspec with defaultuser and tls from vault

559f2fa

Pass failing test

8dcfa32

WIP initContainer should get credentials from vault

495ac1b

Update API docs

6bf6179

Make vault secrets available to setup-container and test it

5b8124c

Use positive condition first

4958f85

Add generated CRD changes

9d3b40f

WIP TLS Working and verified with example test

747509f

Remove duplicate code to create Service sub domain

32cfbae

Fix unit tests

7acf877

Add unit tests for vault annotations TODO: add unit tests for vault commands and mounts

Add vault unit tests

1186ec4

Add unit tests for cluster_types

16246ef

Add more Vault TLS tests

fa825ef

Test that default user K8s secret does not exist

2f46e3f

if default user secret is stored in Vault

Remove duplicate /

6617674

Before this commit: 2021-09-09T13:27:27.312Z [INFO] (runner) rendered "(dynamic)" => "/etc/rabbitmq-tls//tls.key" 2021-09-09T13:27:27.313Z [INFO] (runner) rendered "(dynamic)" => "/etc/rabbitmq-tls//tls.crt"

WIP Rotate defaultuser creds from a watcher container

a064548

Pending: Unit test

WIP: Use Golang image for password update sidecar

b941831

Support password update when TLS is enabled

2234927

Configure klog level

69d359e

and fix mountpath

Add test for password rotation

e861a5b

Move Vault setup scripts from pipeline repo to this repo

c1f1d9f

since it serves as kind of docs for our uses to give a high level idea of what needs to be set up on Vault side. Rename some fields to be more succinct.

Add admin-password-updater

e8aeae1

Fix typo

e130cf6

Add comment how to make example test succeed on kind

ecde1e0

Some K8s clusters (e.g. kind) require to set the issuer. Do not set the issuer on GKE.

Add annotations example

2f0e49c

ansd commented Sep 23, 2021

View reviewed changes

MarcialRosales and others added 5 commits September 23, 2021 09:15

Remove default image from the comments for maintainability reasons

9914d26

Correct docs: It is not possible to disable password rotation

75e167f

Set default user udpater image via kubebuilder tag

9b9a3d0

and allow to disable the sidecar container by setting image name to empty string.

Fix image pull secret docs

a231127

Add test for default image

4e6c70a

ansd marked this pull request as ready for review September 23, 2021 12:27

Do not expose default user credentials as status on the RabbitMQClust…

47291ea

…er resource if default user credentials come from Vault

ansd commented Sep 24, 2021

View reviewed changes

controllers/rabbitmqcluster_controller_test.go Outdated Show resolved Hide resolved

controllers/rabbitmqcluster_controller_test.go Outdated Show resolved Hide resolved

MarcialRosales and others added 4 commits September 24, 2021 17:15

nfocus two test cases

df7516a

Improve setup output

3b2e074

Move admin-password-updater to separate repo

11eb7e9

to rabbitmq/default-user-credential-updater to allow for easier and independant versioning from cluster-operator. We do not want to have the admin-password-updater image the same version as the cluster-operator image.

Rename admin-password-updater to default-user-credential-updater

199c105

so that new repo name matches image name, container name, and entrypoint.

Use same naming

0c2fb0e

MarcialRosales approved these changes Sep 28, 2021

View reviewed changes

ansd and others added 3 commits September 28, 2021 16:14

Remove unused function parameters

afbc078

Update docs & comments to match recently renamed CRD's attribute

e6fbefd

The attributes were vault.defaultUserPath and vault.tls.pkiIssuerPath

Small README changes

d151dc6

ansd merged commit e4e4271 into main Sep 28, 2021

ansd deleted the vault-integration branch September 28, 2021 16:18

This was referenced Sep 30, 2021

Update default user's password in RabbitMQ when it has changed in Vault #823

Closed

To create a cluster with TLS enabled, the certificate and private key should come from Vault's secret #824

Closed

yaronp68 mentioned this pull request Oct 1, 2021

Leverage CR secretbackend configuration to authenticate with cluster rabbitmq/messaging-topology-operator#239

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Vault via vault-k8s #846

Support Vault via vault-k8s #846

ansd commented Sep 17, 2021 •

edited

Loading

ansd commented Sep 21, 2021

twhite0 commented Sep 21, 2021

ansd commented Sep 22, 2021

twhite0 commented Sep 23, 2021

ansd commented Sep 28, 2021

Support Vault via vault-k8s #846

Support Vault via vault-k8s #846

Conversation

ansd commented Sep 17, 2021 • edited Loading

ansd commented Sep 21, 2021

twhite0 commented Sep 21, 2021

ansd commented Sep 22, 2021

twhite0 commented Sep 23, 2021

ansd commented Sep 28, 2021

ansd commented Sep 17, 2021 •

edited

Loading