Vault address is ignored in nomad clients on v1.7.0 #19380
Comments
|
Hi @L-P! Unfortunately I wasn't able to reproduce this. I used the following Vault listener config: listener "unix" {
address = "/home/tim/tmp/vault.sock"
}The following Nomad vault {
address = "unix:///home/tim/tmp/vault.sock"
token = "<redacted>"
enabled = true
}And just for good measure, I ensured that no traffic could be getting to the TCP listener As an aside, I noticed that you don't have a Another thing that you can check for me is the You can also look for a log line like this:
And that'll have a big ol' blob of JSON that's given to the template runner. In there should be a |
|
There's no Here are the requested info plus task logs, using the exact same nomad config, Nomad v1.6.4Task logs: Nomad v1.7.0Task logs: Nomad v1.7.1 behavior is slighly different (but |
|
Thanks @L-P! @lgfa29 and I are still working on a reproduction. Just one question for clarity: your job has a Another item that jumps out at me from your last post is this: And you're getting that for both 1.6.4 and 1.7.0, which means Nomad isn't fingerprinting Vault at all. Is the Any Vault agent logs you can provide from the Nomad agent startup, as well as Vault agent logs from the job starting up, would be very helpful here. |
|
Hi @L-P I think I've got a reproduction here and it's at the intersection of a missing I ran Vault in dev mode with the following configuration: listener "tcp" {
address = "127.0.0.1:8202"
cluster_address = "127.0.0.1:8203"
tls_disable = true
}I ran a Vault Agent with the following configuration: pid_file = "/home/tim/tmp/.vault-agent-pidfile"
vault {
address = "http://127.0.0.1:8202"
retry {
num_retries = 5
}
}
auto_auth {
method {
type = "token_file"
config = {
token_file_path = "/home/tim/.vault-token"
}
}
}
cache {}
api_proxy {
use_auto_auth_token = true
}
listener "unix" {
address = "/home/tim/tmp/vault.sock"
tls_disable = true
agent_api {
enable_quit = true
}
}My Nomad agent's vault {
address = "unix:///home/tim/tmp/vault.sock"
token = "hvs.<redacted>"
create_from_role = "nomad-cluster"
enabled = true
}Last, I configured an iptables rule to ensure that it was impossible for traffic on the default port to work: Then I ran through the Vault integration and retrieving dynamic secrets tutorial:
I've got a preliminary patch that seems to fix the problem by not requiring the diff --git a/client/allocrunner/taskrunner/template_hook.go b/client/allocrunner/taskrunner/template_hook.go
index 3824dd02a8..ea01687ad2 100644
--- a/client/allocrunner/taskrunner/template_hook.go
+++ b/client/allocrunner/taskrunner/template_hook.go
@@ -16,7 +16,6 @@ import (
cstructs "github.com/hashicorp/nomad/client/structs"
"github.com/hashicorp/nomad/client/taskenv"
"github.com/hashicorp/nomad/nomad/structs"
- structsc "github.com/hashicorp/nomad/nomad/structs/config"
)
const (
@@ -212,14 +211,11 @@ func (h *templateHook) Poststart(ctx context.Context, req *interfaces.TaskPostst
func (h *templateHook) newManager() (unblock chan struct{}, err error) {
unblock = make(chan struct{})
- var vaultConfig *structsc.VaultConfig
- if h.task.Vault != nil {
- vaultCluster := h.task.GetVaultClusterName()
- vaultConfig = h.config.clientConfig.GetVaultConfigs(h.logger)[vaultCluster]
+ vaultCluster := h.task.GetVaultClusterName()
+ vaultConfig := h.config.clientConfig.GetVaultConfigs(h.logger)[vaultCluster]
- if vaultConfig == nil {
- return nil, fmt.Errorf("Vault cluster %q is disabled or not configured", vaultCluster)
- }
+ if h.task.Vault != nil && vaultConfig == nil {
+ return nil, fmt.Errorf("Vault cluster %q is disabled or not configured", vaultCluster)
}
tg := h.config.alloc.Job.LookupTaskGroup(h.config.alloc.TaskGroup)Ordinarily leaving out the |
|
I can confirm I don't have a I tried adding one but it failed because So I don't seem to have a workaround. |
|
Ah, the docs could definitely be much more clear on this. The new
It'd be worth taking a look at the eval status of the eval created when you submit the job, as well as the debug-level logs on the scheduler for that eval. |
tgross
added
backport/1.7.x
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Operating system and Environment details
Running on AWS EC2 t3.small.
Issue
Nomad client attempts to reach local vault agent Vault using
127.0.0.1:8200,ignoring its configuration.
Downgrading the client to v1.6.4 fixes the issue.
Reproduction steps
configured to listen on a socket:
Expected Result
The value of
addressis used to connect to the vault agent.Actual Result
Another value is used, thus failing to reach the agent.
Job file (if appropriate)
Uses templates with
{{with secret "<path>"}}.Nomad Server logs (if appropriate)
n/a
Nomad Client logs (if appropriate)
The text was updated successfully, but these errors were encountered: