You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As part of #6701 Nomad Servers will need to be able to perform background Service Identity token reconciliation.
There is an inherent race condition where a Nomad Server successfully requests a new SI token from Consul, but then crashes / loses leadership before it's able to persist the details of that token into the raft log. This isn't particularly bad for Nomad, which can simply retry the request later on with a new leader, but it could cause orphaned tokens to accumulate in Consul. Since SI tokens are not periodic and will have no TTL, such tokens would linger forever. To avoid that, Nomad Servers can periodically request from Consul a list of every token it knows about, and compare that with every SI token a Nomad cluster knows about. If there are any SI tokens in Consul generated for the Nomad cluster (compared with metadata stored in the Description field), that Nomad is not aware of, request Consul to revoke them.
The text was updated successfully, but these errors were encountered:
Starting in Nomad 1.7.0-beta.1 we've deprecated the workflow where the servers mint SI tokens and need highly-privileged Consul tokens. That workflow will be removed in Nomad 1.9.
Ideally by that point we'll have some Consul-side sync (as Consul has for k8s). I'm going to keep this issue open in the meantime but rename it slightly.
tgross
changed the title
Enable Nomad Servers to perform background SI token reconciliation
Nomad-Consul sync for background SI token reconciliation
Nov 1, 2023
As part of #6701 Nomad Servers will need to be able to perform background Service Identity token reconciliation.
There is an inherent race condition where a Nomad Server successfully requests a new SI token from Consul, but then crashes / loses leadership before it's able to persist the details of that token into the raft log. This isn't particularly bad for Nomad, which can simply retry the request later on with a new leader, but it could cause orphaned tokens to accumulate in Consul. Since SI tokens are not periodic and will have no TTL, such tokens would linger forever. To avoid that, Nomad Servers can periodically request from Consul a list of every token it knows about, and compare that with every SI token a Nomad cluster knows about. If there are any SI tokens in Consul generated for the Nomad cluster (compared with metadata stored in the Description field), that Nomad is not aware of, request Consul to revoke them.
The text was updated successfully, but these errors were encountered: