Consul integration upgrade based on Workload Identity

[mikenomitch]

Proposal

Once Workload Identity upgrades make it into Nomad, we can redo the Consul integration to use these tokens as the source of auth instead of manually provided Consul tokens.

Using these tokens, Nomad Users would have a one-time set up process to integrate Nomad workloads into Consul.

The general flow for setting up the Consul-Nomad integration would be:

• Set up Consul
  • Create a Policy for Nomad in Consul
  • Set up JWT Auth Method. As part of this setup, Consul is configured to use Nomad’s Public keys - either passing in the keys, a JWKS URL, or an OIDC Config URL
  • Create Binding Rule(s) for Consul to map Nomad workloads to Consul Roles and/or Services
• Set up Nomad
  • Pass a Consul URL into Nomad Server config in a new configuration block
    (Note: no token needed)
• Deploy Job
  • Job is configured to use new Consul integration
  • Nomad, recognizing that the new integration is being used, automatically requests a token for this job using the JWT auth method.

This would involve an up front cost to set up roles and binding rules in Consul, but after that no management of tokens would be needed.

Use Cases & Advantages

This would be advantageous in many ways:

• Nomad Users would not have to manage Consul token issuing, rotation, and revocation for Nomad clients.
• Nomad Users would not have to manage Consul tokens for each Nomad workload.
• Workload identity tokens could be time-bound and automatically rotated
• Workload identity tokens could provide fine-grained access at the task level
• Workload identity tokens could be automatically removed once the task has stopped
• Multiple Consul clusters could be configured to use the same token. This could allow Nomad to talk to multiple clusters at once and handle disaster recovery better. Failovers could happen without token rotation in Nomad.
• Consul could be more easily deployed as a Nomad job, as client tokens/config would not be needed.

[sofixa]

Something to be investigated: Will the new automatic rotation fall into the same issues as current manual rotation of Consul ACL tokens: hashicorp/consul#4372 (comment) , where after the ACL token has been rotated Nomad is unable to update the services which were registered by this token.

[tgross]

Shipped in Nomad 1.7.0-beta.1

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.