Add support for secret data transformation

[benashz]

Extends VSO to support configuring Go templates that will be rendered to the K8s Secret data. Template specs can be configured directly in any syncable secret custom resource, or can be shared via references to an external SecretTransformation CR. Templates can be composed of other templates, and optionally rendered as a K8s Secret data value. The rendered template name or key is used as the key to the rendered data.

As inputs, the template has access to the source secret data, any related secret metadata, and a secret CR's labels and annotations.

Templating support also includes a large subset of the "hermetic" Sprig text functions that can be used to handle various encoding/decoding tasks, map manipulations, etc.

In addition to template support, it is now possible to filter top level secret data fields based on include/exclude regex patterns. Filters can be used in conjunction with templating, and are meant to augment the overall templating experience. Template fields are not considered during field filtering.

Example:

Given this vault secret data:

{
  "request_id": "xxxxxxxx-a84d-5f43-1d07-xxxxxxxx3d54",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "data": {
      "password": "secret",
      "some-data": "c2VjcmV0",
      "username": "alice"
    },
    "metadata": {
      "created_time": "2023-11-01T19:35:20.247974293Z",
      "custom_metadata": null,
      "deletion_time": "",
      "destroyed": false,
      "version": 2
    }
  },
  "warnings": null
}

and this VSS CR:

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vso-secret
  namespace: tenant1
spec:
  destination:
    create: true
    name: vso-secret
    transformation:
      templates:
        app.env:
          text: |-
            {{- range $key, $value := .Secrets }}
            {{- $envKey :=  printf "ENV_%s" $key | upper | replace "-" "_" -}}
            {{- if eq $key "some-data" -}}
            {{- printf "%s_DECODED=%s\n" $envKey ( $value | b64dec ) -}}
            {{- end -}}
            {{- printf "%s=%s\n" $envKey $value  -}}
            {{- end }}
      excludes:
        - .*
  hmacSecretData: true
  mount: kv-v2
  path: secrets
  refreshAfter: 60s
  type: kv-v2

Would result in this K8s Secret:

---
apiVersion: v1
kind: Secret
metadata:
  labels:
    app.kubernetes.io/component: secret-sync
    app.kubernetes.io/managed-by: hashicorp-vso
    app.kubernetes.io/name: vault-secrets-operator
    secrets.hashicorp.com/vso-ownerRefUID: xxxxxxxx-7dbf-4b6e-ab71-xxxxxxxx5b21
  name: vso-secret
  namespace: tenant1
  ownerReferences:
    - apiVersion: secrets.hashicorp.com/v1beta1
      kind: VaultStaticSecret
      name: vso-secret
      uid: xxxxxxxx-7dbf-4b6e-ab71-xxxxxxxx5b21
type: Opaque
data:
  # _raw: ....
  # app.env: ....
  app.env: |
    ENV_PASSWORD=secret
    ENV_SOME_DATA_DECODED=secret
    ENV_SOME_DATA=c2VjcmV0
    ENV_USERNAME=alice

Resolves #356, resolves #345, resolves #276, resolves #135

Relates to #562 #576 #546

[nia-potato]

hey @benashz thank you so much for working on this

Just a quick question, if i just wanted to make a secret in the vss config to be base64 decoded, is the below config correct?

I am assuming app.env is one of the key's in the secret (vso-secret)

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vso-secret
  namespace: tenant1
spec:
  destination:
    create: true
    name: vso-secret
    transformation:
      templateSpecs:
        - name: app.env
          text: |-
            {{- .Secrets | b64dec }}

  hmacSecretData: true
  mount: kv-v2
  path: secrets
  refreshAfter: 5s
  type: kv-v2

if the above is the case, im assuming we can first fork your branch to start testing this version of templating directly right?

[benashz]

Hey @nia-potato ,

Just a quick question, if i just wanted to make a secret in the vss config to be base64 decoded, is the below config correct?

.Secrets is a map containing the secret data, so you would need to base64 a specific key's value

I am assuming app.env is one of the key's in the secret (vso-secret)

app.env would be included in the K8s Secret vso-secret.

If you wanted to include all vault secrets and only decode a specific value you could use this template:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vso-secret
  namespace: tenant1
spec:
  destination:
    create: true
    name: vso-secret
    transformation:
      templateSpecs:
      - name: some-data
        text: '{{- get .Secrets "some-data" | b64dec -}}'
  hmacSecretData: true
  mount: kv-v2
  path: secrets
  refreshAfter: 5s
  type: kv-v2

some-data would be included in the k8s secret with its value base64 decoded.
E.g (values are base64 decoded):

{
  "_raw": "....",
  "password": "secret",
  "some-data": "secret",
  "username": "alice"
}

if the above is the case, im assuming we can first fork your branch to start testing this version of templating directly right?

Yes, you should be able to do some local testing, either from this repo or a fork. I can add some samples to this branch if you want to get up and running quickly. Otherwise you may be able to get going by running the following commands:

make setup-kind docker-build load-docker-image deploy setup-vault

Then you can access Vault by setting the following env vars:

export VAULT_TOKEN=root
export VAULT_ADDR=http://127.0.0.1:38300

Let me know if you encounter any issues.

[nia-potato]

Thanks @benashz

i will get back to you after i finish testing.

On a side note, since im assuming the official 0.4.0 will take a while to be released, i've currently just did a helm package chart/ && helm push <private-repo>

and on my dev cluster(non-local) i've tried to pull this via the private repo or directly via your local chart/ path

resource "helm_release" "vault-secrets-operator" {
  depends_on    = [kubernetes_namespace.ns] // Add this line
  wait          = true
  wait_for_jobs = true
  name          = "vault-secrets-operator"
  chart         = "./chart/"
  namespace     = kubernetes_namespace.ns.metadata.0.name

}

it seems to install just fine but when i try to apply the vault static secrets config as you showed me from above, i am getting Error from server (BadRequest): error when creating "config/samples/test-vss.yaml": VaultStaticSecret in version "v1beta1" cannot be handled as a VaultStaticSecret: strict decoding error: unknown field "spec.destination.transformation"

test-vss.yaml is basically the config you showed me from above.

Do you happen to know anyway around this other than just waiting for the official v1beta1 API to updated with your changes?

[benashz]

Thanks @benashz

i will get back to you after i finish testing.

👍

On a side note, since im assuming the official 0.4.0 will take a while to be released, i've currently just did a helm package chart/ && helm push <private-repo>

and on my dev cluster(non-local) i've tried to pull this via the private repo or directly via your local chart/ path

resource "helm_release" "vault-secrets-operator" {
  depends_on    = [kubernetes_namespace.ns] // Add this line
  wait          = true
  wait_for_jobs = true
  name          = "vault-secrets-operator"
  chart         = "./chart/"

Does chart directory contain the chart from this PR's branch?

namespace = kubernetes_namespace.ns.metadata.0.name

}

it seems to install just fine but when i try to apply the vault static secrets config as you showed me from above, i am getting `Error from server (BadRequest): error when creating "config/samples/test-vss.yaml": VaultStaticSecret in version "v1beta1" cannot be handled as a VaultStaticSecret: strict decoding error: unknown field "spec.destination.transformation"`

It looks like the CRDs do not match for some reason. Does running the following command from the repo root resolve the issue (make sure your repo is checked out this PR's branch, just in case):

kubectl apply --recursive -f chart/crds/.

test-vss.yaml is basically the config you showed me from above.

Do you happen to know anyway around this other than just waiting for the official v1beta1 API to updated with your changes?

I think running the kubectl command above should get the CRDs updated. Helm does not update/install CRDs during an upgrade, so that might explain the issue if this was not a fresh install.

Let me know if any of that helps.

[nia-potato]

Got the crds installing part resolved, thanks @benashz

However i am still unable to get the base64 decoding working, i feel i must be missing something obvious:

Given secret name: test with the keys below:

test0: test
test1: c2VjcmV0
test2: c2VjcmV0Mg==

i apply the config as discussed:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: test
  namespace: default
spec:
  destination:
    create: true
    name: test
    transformation:
      templateSpecs:
      - name: test1
        text: '{{- get .Secrets "test1" | b64dec -}}'
      - name: test2
        text: '{{- get .Secrets "test2" | b64dec -}}'
  hmacSecretData: true
  mount: secret
  path: path/to/test
  refreshAfter: 5s
  type: kv-v2

after running k get secrets test -o yaml i am still getting (base64(base64(value))) for test1 and test2.

apiVersion: v1
data:
  test: dGVzdA==
  test1: YzJWamNtVjA=
  test2: YzJWamNtVjBNZz09
kind: Secret
metadata:
  creationTimestamp: "2023-11-02T07:30:57Z"
  labels:
    app.kubernetes.io/component: secret-sync
    app.kubernetes.io/managed-by: hashicorp-vso
    app.kubernetes.io/name: vault-secrets-operator
    secrets.hashicorp.com/vso-ownerRefUID: 9bce9701-2b67-4aa9-81e7-XXXXXXXXXX
  name: test
  namespace: default
  ownerReferences:
  - apiVersion: secrets.hashicorp.com/v1beta1
    kind: VaultStaticSecret
    name: test
    uid: 9bce9701-2b67-4aa9-81e7-XXXXXXXXXXX
  resourceVersion: "363023982"
  uid: 4f4161c8-ffca-4ad9-8eaa-XXXXXXXXXX
type: Opaque

[benashz]

Hey @nia-potato , would you mind confirming that you are VSO is running from the dev Docker image? That could explain why are you not seeing the transformation directive being honored?

If you are testing with a Kind cluster can you run (from a checkout of this branch):

make docker-build load-docker-image rollout-restart-operator

Would you mind inspecting the VSO Deployment to ensure that your image is set to hashicorp/vault-secrets-operator:0.0.0-dev.

Can you run the following command to verify (you may need to adjust the namespace and name to match your setup):

kubectl get deployment -n vault-secrets-operator-system vault-secrets-operator-controller-manager -o=jsonpath='{$.spec.template.spec.containers[:].image}'

[nia-potato]

Hey @benashz
I was actually testing this on a live dev GKE cluster that previously had a working vso installation of 0.3.0. All i did was changed the installation method as shown in the above terraform to install locally from your chart/ directory that was checked out from this branch.
Then i manually forced an update of the crds as you suggested.

I see that the deployment does not contain X.X.X-dev as you mentioned, do you suspect that with my current method of installing the chart directly from your branch still does not contain your changes? which is causing the transformation be recognized by the updated crds during k apply -f vss-confg.yaml but not doing anything in practice?

k get deployment vault-secrets-operator-controller-manager -n vso -o=jsonpath='{$.spec.template.spec.containers[:].image}'
gcr.io/kubebuilder/kube-rbac-proxy:v0.14.4 hashicorp/vault-secrets-operator:0.3.4

a quick excerpt of the logs of the current running vso is as below, is the current testing method not available for a live cluster test in anyway but only for the local tests?

{"level":"info","ts":"2023-11-02T21:09:54Z","logger":"initCachingClientFactory","msg":"Initializing the CachingClientFactory"}
2023-11-02T21:09:55Z    INFO    setup   Starting manager        {"clientCachePersistenceModel": "none", "clientCacheSize": 10000}
2023-11-02T21:09:55Z    INFO    controller-runtime.metrics      Starting metrics server
2023-11-02T21:09:55Z    INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": "127.0.0.1:8080", "secure": false}
2023-11-02T21:09:55Z    INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
I1102 21:09:55.021867       1 leaderelection.go:250] attempting to acquire leader lease vso/b0d477c0.hashicorp.com...
I1102 21:10:10.953667       1 leaderelection.go:260] successfully acquired lease vso/b0d477c0.hashicorp.com
2023-11-02T21:10:10Z    DEBUG   events  vault-secrets-operator-controller-manager-85c6f7cc8-756f6_678b64e4-927b-40f9-98d7-1d50ec60fef3 became leader    {"type": "Normal", "object": {"kind":"Lease","namespace":"vso","name":"b0d477c0.hashicorp.com","uid":"e1b9e8d4-c7a2-4a03-b1c9-90f0bd7a5c95","apiVersion":"coordination.k8s.io/v1","resourceVersion":"363146383"}, "reason": "LeaderElection"}
2023-11-02T21:10:10Z    INFO    Starting EventSource    {"controller": "vaultstaticsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultStaticSecret", "source": "kind source: *v1beta1.VaultStaticSecret"}
2023-11-02T21:10:10Z    INFO    Starting Controller     {"controller": "vaultstaticsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultStaticSecret"}
2023-11-02T21:10:10Z    INFO    Starting EventSource    {"controller": "vaultauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultAuth", "source": "kind source: *v1beta1.VaultAuth"}
2023-11-02T21:10:10Z    INFO    Starting Controller     {"controller": "vaultauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultAuth"}
2023-11-02T21:10:10Z    INFO    Starting EventSource    {"controller": "vaultpkisecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultPKISecret", "source": "kind source: *v1beta1.VaultPKISecret"}
2023-11-02T21:10:10Z    INFO    Starting EventSource    {"controller": "vaultpkisecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultPKISecret", "source": "kind source: *v1beta1.VaultPKISecret"}
2023-11-02T21:10:10Z    INFO    Starting Controller     {"controller": "vaultpkisecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultPKISecret"}
2023-11-02T21:10:10Z    INFO    Starting EventSource    {"controller": "vaultconnection", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultConnection", "source": "kind source: *v1beta1.VaultConnection"}
2023-11-02T21:10:10Z    INFO    Starting Controller     {"controller": "vaultconnection", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultConnection"}
2023-11-02T21:10:10Z    INFO    Starting EventSource    {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVaultSecretsApp", "source": "kind source: *v1beta1.HCPVaultSecretsApp"}
2023-11-02T21:10:10Z    INFO    Starting Controller     {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVaultSecretsApp"}
2023-11-02T21:10:10Z    INFO    Starting EventSource    {"controller": "hcpauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPAuth", "source": "kind source: *v1beta1.HCPAuth"}
2023-11-02T21:10:10Z    INFO    Starting Controller     {"controller": "hcpauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPAuth"}
2023-11-02T21:10:10Z    INFO    Starting EventSource    {"controller": "vaultdynamicsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultDynamicSecret", "source": "kind source: *v1beta1.VaultDynamicSecret"}
2023-11-02T21:10:10Z    INFO    Starting Controller     {"controller": "vaultdynamicsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultDynamicSecret"}
2023-11-02T21:10:11Z    INFO    Starting workers        {"controller": "vaultstaticsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultStaticSecret", "worker count": 1}
2023-11-02T21:10:11Z    INFO    Starting workers        {"controller": "vaultpkisecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultPKISecret", "worker count": 1}
2023-11-02T21:10:11Z    INFO    Starting workers        {"controller": "vaultconnection", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultConnection", "worker count": 1}
2023-11-02T21:10:11Z    INFO    Starting workers        {"controller": "vaultauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultAuth", "worker count": 1}
2023-11-02T21:10:11Z    INFO    Starting workers        {"controller": "hcpauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPAuth", "worker count": 1}
2023-11-02T21:10:11Z    DEBUG   events  Successfully handled VaultAuth resource request {"type": "Normal", "object": {"kind":"VaultAuth","namespace":"vso","name":"default","uid":"97018d90-00f2-465c-a95f-a4064ba84241","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"362412968"}, "reason": "Accepted"}
2023-11-02T21:10:11Z    DEBUG   events  Successfully handled VaultAuth resource request {"type": "Normal", "object": {"kind":"VaultAuth","namespace":"default","name":"app-vault-auth","uid":"aed6df26-ef33-43d9-b913-2390c98776f3","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"278218924"}, "reason": "Accepted"}
2023-11-02T21:10:11Z    INFO    Starting workers        {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVaultSecretsApp", "worker count": 1}
2023-11-02T21:10:11Z    INFO    Starting workers        {"controller": "vaultdynamicsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultDynamicSecret", "worker count": 100}
2023-11-02T21:10:11Z    DEBUG   events  Secret synced   {"type": "Normal", "object": {"kind":"VaultStaticSecret","namespace":"demo","name":"auth-secret","uid":"56fd2eea-61f4-4377-9013-90dcc6aeec35","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"354653993"}, "reason": "SecretRotated"}

[benashz]

Hey @benashz I was actually testing this on a live dev GKE cluster that previously had a working vso installation of 0.3.0. All i did was changed the installation method as shown in the above terraform to install locally from your chart/ directory that was checked out from this branch. Then i manually forced an update of the crds as you suggested.

Ah that explains it. Your GKE cluster would need access to the development docker image some how. We do test on GKE. Maybe this bit could help: https://github.com/hashicorp/vault-secrets-operator/blob/1ad9b0ffda6415859a0397a08312d0be0aaf99d2/gcp.mk#L32C17-L35

Then you would need to set the image repo and tag to the dev image during your helm installation.

This article might help:

• https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app#pushing_the_docker_image_to
• https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app#building_the_container_image

I see that the deployment does not contain X.X.X-dev as you mentioned, do you suspect that with my current method of installing the chart directly from your branch still does not contain your changes? which is causing the transformation be recognized by the updated crds during k apply -f vss-confg.yaml but not doing anything in practice?

k get deployment vault-secrets-operator-controller-manager -n vso -o=jsonpath='{$.spec.template.spec.containers[:].image}'
gcr.io/kubebuilder/kube-rbac-proxy:v0.14.4 hashicorp/vault-secrets-operator:0.3.4

Thanks for the confirmation.

[benashz]

@nia-potato , Just an FYI: 9ab331c brings in some breaking API changes. I will update the main description with the details next week.

[nia-potato]

hey @benashz Thanks for the update, and i just confirmed that your patch prior of 9ab331c is working fine for encrypted secrets!

I can test again next week, and will start to experiment with your colleagues gcp auth with workload identity patch next week, so feel free to shoot over any experimental features you guys are working on, it seems like we are using most of your teams new features haha, have a good weekend!

[nia-potato]

hey @benashz lemme know when you need me to test

[benashz]

hey @benashz lemme know when you need me to test

Hey @nia-potato, you should be able to test any time against the latest commit on this branch. You will need to convert your *Secret CRs though: spec.destination.transformation.templateSpecs is no longer an array of maps, it is a map of template-names to a TemplateSpec. You will need to apply the updated CRDs as well.

If you were using the templateRef ConfigMap feature, please let me know, as that has been totally re-worked.

Let me know how it goes.

Thanks,

Ben

Example of the transformation API update (inline only):

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vso-secret
  namespace: tenant1
spec:
  destination:
    create: true
    name: vso-secret
    transformation:
      templateSpecs:
        app.env:
          text: |-
            {{- range $key, $value := .Secrets }}
            {{- $envKey :=  printf "ENV_%s" $key | upper | replace "-" "_" -}}
            {{- if eq $key "some-data" -}}
            {{- printf "%s_DECODED=%s\n" $envKey ( $value | b64dec ) -}}
            {{- end -}}
            {{- printf "%s=%s\n" $envKey $value  -}}
            {{- end }}
      fieldFilter:
        excludes:
          - .*
  hmacSecretData: true
  mount: kv-v2
  path: secrets
  refreshAfter: 60s
  type: kv-v2

[nia-potato]

@benashz

Thanks for the update

however i am getting this error after updating the crds with k apply --recursive -f chart/crds/. and can verify that the new secrettransformations.secrets.hashicorp.com has been added.
Failed setting up SecretRenderOption: failed to gather templates, duplicate template spec name "" {"type": "Warning", "object": {"kind":"VaultStaticSecret","namespace":"default","name":"test-secret","uid":"9bce9701-2b67-4aa9-81e7-21af70741a2f","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"363233457"}, "reason": "TransformationError"}

do you see any potential errors in the vss config?
Once again this is my current setup:

test-secret
test0: test
test1: c2VjcmV0
test2: c2VjcmV0Mg==

vss config

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: test-secret
  namespace: default
spec:
  destination:
    create: true
    name: test-secret
    transformation:
      templateSpecs:
        test1:
          text: '{{- get .Secrets "test1" | b64dec -}}'
        test2:
          text: '{{- get .Secrets "test2" | b64dec -}}'
  hmacSecretData: true
  mount: secret
  path: default/test-secret
  refreshAfter: 5s
  type: kv-v2

[benashz]

@benashz

Thanks for the update

however i am getting this error after updating the crds with k apply --recursive -f chart/crds/. and can verify that the new secrettransformations.secrets.hashicorp.com has been added. Failed setting up SecretRenderOption: failed to gather templates, duplicate template spec name "" {"type": "Warning", "object": {"kind":"VaultStaticSecret","namespace":"default","name":"test-secret","uid":"9bce9701-2b67-4aa9-81e7-21af70741a2f","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"363233457"}, "reason": "TransformationError"}

do you see any potential errors in the vss config? Once again this is my current setup:

test-secret
test0: test
test1: c2VjcmV0
test2: c2VjcmV0Mg==

vss config

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: test-secret
  namespace: default
spec:
  destination:
    create: true
    name: test-secret
    transformation:
      templateSpecs:
        test1:
          text: '{{- get .Secrets "test1" | b64dec -}}'
        test2:
          text: '{{- get .Secrets "test2" | b64dec -}}'
  hmacSecretData: true
  mount: secret
  path: default/test-secret
  refreshAfter: 5s
  type: kv-v2

The CR definition above looks correct to me for the latest change on this branch, but for some reason the template name is "". Just to confirm VSO was re-built from the same changeset as the CRDs?

[nia-potato]

hey @benashz it is now working, it was an error on my end.

Now that this is working, do you anticipate any more breaking changes regarding the transformation api before the 0.4.0 release? if not i will be using the version internally while i wait for the official 0.4.0 release.

Also within the commits, i also saw changes related to cert-manager im not sure if i understand the full scope of the commit since it is ongoing, but does that mean with your latest commit we can use cert-manager to fully manage certificates for vso on GKE? based on the annotation we need to add?

[kphunter]

Thank you for this! With templates that convert the default access_key to a more standard AWS_ACCESS_KEY_ID, et. al., we'll now be able to use the dynamic AWS secret engine...

[benashz]

@nia-potato

hey @benashz it is now working, it was an error on my end.

👍 Thanks for taking this out for a test spin!

Now that this is working, do you anticipate any more breaking changes regarding the transformation api before the 0.4.0 release? if not i will be using the version internally while i wait for the official 0.4.0 release.

I do not anticipate any further breaking changes, but as we are still fleshing out the overall secret transformation API, there is a possibility that some changes will be made that require a manual intervention. I will announce any breaking changes in this PR.

Also within the commits, i also saw changes related to cert-manager im not sure if i understand the full scope of the commit since it is ongoing, but does that mean with your latest commit we can use cert-manager to fully manage certificates for vso on GKE? based on the annotation we need to add?

I believe those references to cert-manager are boiler plate created by the operator-sdk. They are related to webhooks, which we are not currently using in VSO.

[AdamNuclear]

@benashz Great work! I am testing it in our lab and it works well.

But maybe one thing would be good to consider:

Example:
test data:

Key         Value
---         -----
data        map[test1:test test2:abcd]

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vso-secret
  namespace: vault-secrets-operator
spec:
  destination:
    create: true
    name: vso-secret
    transformation:
      templates:
        users:
          text: |-
             {{- range $value := .Secrets }}
             {{- printf "%s\n" $value  -}}
             {{- end -}}
  hmacSecretData: true
  mount: kv
  path: k8s/lab/adamtest
  refreshAfter: 5s
  type: kv-v2
  vaultAuthRef: vaultauth-aws-irsa

Output:

nuclear@teapot ~/ $ k get secret -o yaml vso-secret
apiVersion: v1
data:
  _raw: eyJkYXRhIjp7InRlc3QxIjoidGVzdCIsInRlc3QyIjoiYWJjZCJ9LCJtZXRhZGF0YSI6eyJjcmVhdGVkX3RpbWUiOiIyMDI0LTAyLTAxVDIxOjQ5OjI4Ljk2OTk2NzA0N1oiLCJjdXN0b21fbWV0YWRhdGEiOm51bGwsImRlbGV0aW9uX3RpbWUiOiIiLCJkZXN0cm95ZWQiOmZhbHNlLCJ2ZXJzaW9uIjo0fX0=
  test1: dGVzdA==
  test2: YWJjZA==
  users: dGVzdAphYmNkCg==

Expected output:

nuclear@teapot ~/ $ k get secret -o yaml vso-secret
apiVersion: v1
data:
  users: dGVzdAphYmNkCg==

Would it be possible to make some switch like "clean: true/false" or something like it? Once you are templating you know what you want in secret so _raw and key & value are redundant.

[benashz]

@benashz Great work! I am testing it in our lab and it works well.

@AdamNuclear Awesome, thanks!

Would it be possible to make some switch like "clean: true/false" or something like it? Once you are templating you know what you want in secret so _raw and key & value are redundant.

For excluding the _raw secret data, that work is being done in #546.

For controlling the inclusion of non-templated keys you can set the spec.destination.transformation.excludes to the regex pattern: [.*] I think that will give you want you want. The excludes and includes let you sort of mix and match templated and non-templated data.

An example of your VSS with excludes added:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vso-secret
  namespace: vault-secrets-operator
spec:
  destination:
    create: true
    name: vso-secret
    transformation:
      excludes:
        - .*
      templates:
        users:
          text: |-
             {{- range $value := .Secrets }}
             {{- printf "%s\n" $value  -}}
             {{- end -}}
  hmacSecretData: true
  mount: kv
  path: k8s/lab/adamtest
  refreshAfter: 5s
  type: kv-v2
  vaultAuthRef: vaultauth-aws-irsa

[AdamNuclear]

@benashz Ahh, that was just my misunderstanding regarding include/exclude. Now it does exactly what I was looking for.

Thank you for your explanation.

[tvoran]

A few questions here, but 👍

[kschoche]

Looks great, I left a few comments here and there but nothing blocking!

[duong-se]

hi @benashz do you have any document related to transformation template