KV-v2 secrets with delete_version_after cannot be injected

[leongyh]

Describe the bug
KV-v2 secrets with a "deletion_time" flag cannot be discovered and injected.

To Reproduce
Steps to reproduce the behavior:

1. Create a KV-v2 secret engine with delete_version_after set to a non default, such as '1hr'.
2. Create a secret in the mount.
3. Deploy your service with annotations referencing the newly created secret. The agent will fail to find and load the secret. Describing the secret:

> vault read kv-v2/test/secret2
Key      Value                                                                                                                           
data     {"hi":"bye"}                                                                                                                    
metadata {"created_time":"2020-09-02T22:07:35.707803228Z","deletion_time":"2020-09-02T23:07:35.707803228Z","destroyed":false,"version":1}

Here is the agent-injector-init logs:

2020/09/02 22:08:30.090179 [WARN] (view) vault.read(kv-v2/test/secret2): no secret exists at kv-v2/test/data/secret2 (retry attempt 8 after "32s"

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: "my-microservice"
 labels:
   app: nginx
spec:
 replicas: 1
 selector:
   matchLabels:
     app: nginx
 template:
   metadata:
     annotations:
       vault.hashicorp.com/agent-inject: "true"
       vault.hashicorp.com/role: "kubernetes-auth-vault-role"
       vault.hashicorp.com/agent-inject-secret-blah.txt: kv-v2/test/secret2
     labels:
       app: nginx
   spec:
     serviceAccountName: sample-service
     containers:
     - name: nginx
       image: nginx:1.14.2
       ports:
       - containerPort: 80

Expected behavior
The happy path where the secret did not have a deletion_time looks like this:

> vault read kv-v2/test/secret1
Key      Value                                                                                             
data     {"hey":"bae"}                                                                                     
metadata {"created_time":"2020-09-02T21:50:52.560873843Z","deletion_time":"","destroyed":false,"version":1}

The pod has the injected secret and all is well.

Environment

• Kubernetes version: 1.15
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): EKS
  • Other configuration options or runtime services (istio, etc.): Nothing worth mentioning
• vault-k8s version: 0.5.0

Additional context
We installed the injector via the vault-helm helm chart. Vault is an external instance.

[jasonodonnell]

Hi @leongyh, I think your path might be incorrect. Can you try this?

vault.hashicorp.com/agent-inject-secret-blah.txt: kv-v2/data/test/secret2

[leongyh]

@jasonodonnell That wasn't it.

I still get

2020/09/03 21:29:46.088463 [WARN] (view) vault.read(kv-v2/test/data/secret2): no secret exists at kv-v2/test/data/secret2 (retry attempt 8 after "32s")

FYI, my mount path is kv-v2/test.

[jasonodonnell]

I looked into this on my end and it's indeed a bug with Vault Agent/Consul Template. We haven't found the root cause yet but are looking closer. Thanks for reporting!

[jasonodonnell]

I think I figured out what is happening. When that flag is enabled, versioning is also enabled. So you need to specify the version in the template (I'm using a custom template here to do it):

vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/agent-inject-secret-kv-secret: "secret/data/hashiconf"
vault.hashicorp.com/agent-inject-template-kv-secret: |
    {{ with secret "secret/data/hashiconf" "version=1" }}{{ .Data | toJSONPretty }}{{ end }}

Can you give this a shot with your secret?

[leongyh]

Unfortunately, it did not work. Here are my annotations:

vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/log-level: "trace"
vault.hashicorp.com/role: "kubernetes-auth-vault-role"
vault.hashicorp.com/agent-inject-secret-blah: "kv-v2/test/secret2"
vault.hashicorp.com/agent-inject-template-blah: |
          {{- with secret "kv-v2/test/data/secret2" "version=6" -}}
            {{ .Data | toJSONPretty }}
          {{- end }}

When I tried this, I was surprised to see the agent asking for update/PUT permissions to Vault.

URL: PUT https://my.vault.com/v1/kv-v2/test/data/secret2
Code: 403. Errors:

* 1 error occurred:
	* permission denied

So I went ahead and gave it the update permissions in the Vault policy. The pod ran but it wrote to Vault instead.

This was injected into the pod's secret mount:

cat /vault/secrets/blah 
{
  "created_time": "2020-09-11T16:09:15.828326004Z",
  "deletion_time": "2020-09-16T07:09:15.828326004Z",
  "destroyed": false,
  "version": 13
}

[jasonodonnell]

For the permissions confusion: when Consul Template receives extra args in the with clause, it treats it like an update. It does this automatically behind the scenes.

I found the root cause of this bug in Consul Template. Since deletion_time, although it's in the future, is not an empty string, CT is just outright rejecting it.

We'll fix this up and get it fixed in a future version of Vault.

[dcatalano-figure]

running into same issue ... first off, thank you @leongyh for reporting this ... hard one to find (for me)

set delete_version_after to 0s, then recreated the secret, and it works again!

[nemethloci]

What's the status about this? We'd like to use versioning and delete_version_after option, but this combination doesn't seem to work with vault injector at the moment, unless one explicitly sets the version number in the vault injector config, which at least we would like to avoid.
@jasonodonnell : you meantioned, you found the issue in Consul Template: have a fix ever been created pushed upstream?

[samzurcher]

Any update or workaround on this issue? We just ran into this bug; unfortunately, it is not so easy to debug. Would be very grateful if you guys could look into this and fix it.

[VioletHynes]

Hey! Thanks for bearing with us. The issue has been identified and the bug has been fixed, and should be in the next release.

[VioletHynes]

Looks like I missed the boat slightly for 1.16.0, but this fix should be in 1.15.6 and 1.16.1.