Consul Templates Fail to Render when using KV2 secrets with deletion time

[Jasonrchoi]

Using the Vault Agent Injector in EKS Cluster A addressed to an external vault in EKS Cluster B, I have a test pod with the following annotations. The integration-app-test role was given a policy that grants access to both secret paths.

Command

        vault.hashicorp.com/agent-inject-secret-credentials.txt: secret/staging/app/secure #Generic backend
        vault.hashicorp.com/agent-inject-secret-secure: secretv2/data/application/staging/app #KV2 backend
        vault.hashicorp.com/agent-inject-template-secure: |
          {{ with secret "secretv2/data/application/staging/app" -}}
            {{ range $k, $v := .Data.metadata }}
              export {{ $k | toUpper }}={{ $v }}
            {{ end }}
          {{- end }}
        vault.hashicorp.com/auth-path: auth/kubernetes_integration
        vault.hashicorp.com/log-level: trace
        vault.hashicorp.com/role: integration-app-test

note `secret/` refers to a generic secrets backend, `secretv2/` refers to a 'kv2' backend

Expected behavior

What should have happened?

Secrets from both backends were rendered into the pod.

Actual behavior

We found that the "generic" secrets are rendered correctly, while the "kv2" secrets receive a no secret exists at {PATH} error and then eventually, Permission Denied. Because we were able to access secrets from our generic secret backend, it appeared we could rule out most auth/network issues.

Steps to reproduce

We think we discovered the cause which is:

consul-template/dependency/vault_read.go

Line 178 in e1484d3

func deletedKVv2(s *api.Secret) bool {

In the DeleteKV2v2 method, any Deletion_Time associated with the secret, other than null, will cause the method to return true, causing the Read to throw an error that the secret does not exist, even if it clearly does. Once we got rid of deletion time for our secrets, the kv2 secret was rendered correctly by the Vault Agent Injector.

References

It appears this issue was alluded to in vault-k8s but never resolved.

Are there any other GitHub issues (open or closed) that should
be linked here? For example:

• KV-v2 secrets with delete_version_after cannot be injected vault-k8s#176