Nomad fails to renew Vault token

[pkrolikowski]

It looks like Nomad is sending invalid request to Vault while renewing allocation token (see Vault audit log output)

Nomad version

Nomad v0.7.0

Vault version

Vault v0.8.1

Operating system and Environment details

Ubuntu 16.04.3 LTS

Nomad Client logs (if appropriate)

Dec 11 15:18:41 apps-cpu-dwnm wrap_nomad.sh[2137]: URL: PUT https://vault.query.consul:8200/v1/auth/token/renew-self
Dec 11 15:18:41 apps-cpu-dwnm wrap_nomad.sh[2137]: Code: 400. Errors:
Dec 11 15:18:41 apps-cpu-dwnm wrap_nomad.sh[2137]: * lease is not renewable

Job file (if appropriate)

Vault audit logs

{
  "time": "2017-12-11T15:18:41Z",
  "type": "response",
  "auth": {
    ...
    "policies": [
      "default",
      "test"
    ],
    "metadata": {
      "AllocationID": "a9cc4889-cd36-298c-c9db-8fe98fbac4d1",
      "NodeID": "xxx",
      "Task": "test"
    }
  },
  "request": {
    ...
    "path": "auth/token/renew-self",
    "data": {
      ...
  },
  "response": {
    "data": {
      "error": "hmac-sha256:7b613738279b913f0db829712e83feddd1914c051ccdbb7702efe4a566f2b85c"
    }
  },
  "error": "1 error occurred:\n\n* invalid request"
}

Token for this allocation has this attributes:

Key              	Value
---              	-----
accessor         	...
creation_time    	1512997082
creation_ttl     	259200
display_name     	token-xxx-test
expire_time      	2017-12-14T12:58:02.778658199Z
explicit_max_ttl 	0
id               	...
issue_time       	2017-12-11T12:58:02.673747189Z
last_renewal     	2017-12-11T12:58:02.778658343Z
last_renewal_time	1512997082
meta             	map[AllocationID:... Task:test]
num_uses         	0
orphan           	false
path             	auth/token/create/nomad-server
policies         	[default test]
renewable        	true
role             	nomad-server
ttl              	251578

renewable is set, so it can be renewed (i thought that's the case).

[dadgar]

1. Do you have multiple Vault clusters? Are the Nomad Server/Client talking to the same Vault server?
2. Does the policies default or test give the token auth/token/renew-self permissions as required here: https://www.nomadproject.io/docs/vault-integration/index.html#required-vault-policies

[pkrolikowski]

1. nope, nothin fancy; our setup: three VMs hosting vault and nomad servers plus set of nomad clients talking to this servers
2. yes, I was able to renew token manually (curl) by hitting auth/token/renew-self endpoint.

[dadgar]

@pkrolikowski Do you have reproduction steps?

[pkrolikowski]

Unfortunately not. It looks like it's correlated with high load on nomad client. Tomorrow I'll try to verify this.

[dadgar]

@pkrolikowski

1. Any update reproducing?
2. Can I see the full client logs
3. Is the token information you show the same as the token you are showing in the audit logs? The audit logs seems to be a different error than you are showing in the logs.
4. Can you show the output of nomad alloc-status for the allocation? Did it actually fail or did Nomad get it a new token? What I am driving at is, was this a transient or permanent error.

[dadgar]

@pkrolikowski I am going to close this till we get more information as it has not been reproducible

[c4milo]

@dadgar, FWIW. I just noticed nomad 0.8.3 does not update its vault token upon sending it a SIGHUP signal when reloading from systemd. I had to restart the service in order to get it to pick up a newly set vault token. I can create a new issue if you think that behavior is not expected.

[Tethik]

@c4milo I would assume it's not intended since I was just recommended to use that in #4372

[dadgar]

@c4milo Unfortunately a regression we will fix in 0.8.4: #4386

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.