x509 certificate signed by unknown authority when Nomad secret enabled in Vault

[ericssen]

Hi all!
I am struggling with a certificate problem. I enabled Nomad secret engine in Vault to obtain Nomad ACL token for authenticate before submitting Nomad job. There is no problem if TLS encryption disabled but when enabled, I receive x509 message when trying to access Nomad ACL token. Is it possible to add cert to Vault's Nomad secret or add it to Vault's policy which is used to create Vault token? It is important to say that Nomad and can connect to Vault because certificates are signed with same CA but somehow, this request from Vault to Nomad doesn't pass due to mentioned error. Is it possible to specify which cert to use when creating vault write nomad/config/access
address=https://127.0.0.1:4646
token=adf4238a-882b-9ddc-4a9d-5b6758e4159e
from Hashicorp Vault documentation?
Error reading nomad/creds/monitoring: Error making API request.

URL: GET https://localhost:8200/v1/nomad/creds/monitoring
Code: 500. Errors:

• 1 error occurred:

• Put https://localhost:4646/v1/acl/token: x509: certificate signed by unknown authority

Thanks for help!

[ncabatoff]

Hi @ericssen,

No, there's no way to specify what cert Vault should use when speaking to Nomad. It sounds like you're using a self-signed certificate, and you haven't added the CA to your Vault server's system list of trusted CAs. Or possibly you simply haven't restarted Vault after doing so. You may read that a SIGHUP suffices to make Vault re-read certificates, but that doesn't apply to system CAs.

Updating the system CA list is OS-specific, you're probably looking for something like update-ca-certificates or update-ca-trust.

[nvx]

It looks like the Vault nomad backend uses the nomad SDK's DefaultConfig() option - https://github.com/hashicorp/vault/blob/v1.1.3/builtin/logical/nomad/backend.go#L57

This in turn allows setting the ca cert and other parameters directly via environment variables: https://github.com/hashicorp/nomad/blob/v0.9.3/api/api.go#L248

A workaround would be specifying eg NOMAD_CACERT environment variable to Vault. The downside of this is it can't be specified per-mount and requires a Vault server restart to achieve.

Ideally the other nomadConf parameters would be exposed to the mount config (although ideally being able to specify the PEM directly rather than referencing a file on the Vault servers disk, but even if it had to be a file on each Vault server that's better than nothing). This would be especially important when using client certificates for example.

Adding the Nomad CA to the system CAs list is less than ideal as the Nomad CA should be a private CA as per https://www.nomadproject.io/guides/security/securing-nomad.html so Vault ideally would not be trusting that root for anything other than Nomad use.