Support uploading ACL Tokens

[carobme]

We were hit by #24411 recently, essentially having to recreate a cluster from scratch. While it was easy to apply jobs, namespaces, policies etc. again on the re-inizialized cluster, we were faced with having to create a bunch of new tokens, forcing people to adjust their deployment pipelines etc.

I'm fine with having to create a new bootstrap token, or management tokens in general, on a fresh cluster. But would it be possible to have an endpoint (i.e. POST /acl/token/:accessor_id) to upload/set specific (AccessorID, SecretID) pairs for client tokens? I'd rather prefer that over restoring a RAFT snapshot.

Nomad 1.9.1 (after downgrading from 1.9.2), we currently don't use Vault.

Thanks!

[tgross]

@carobme the bootstrap token already accepts an initial secret (because it's challenging to use otherwise), but this workflow is an awful corner case -- we generally expect that DR is going to happen via restoring from backup. Is there a particular reason that's a problem for your environment?

[carobme]

@tgross I always prefer a text based configuration that I can put in a Git repo over some blob (that is a blackbox to me). I already have everything that I need regarding Nomad in repo (agent config, jobs, policies etc.), except said tokens.

Aside from DR it's also useful for backup/restore: Let's say I accidentally delete a bunch of tokens, thinking that they're not needed anymore. Recognizing that a couple of hours later I want to restore them. I cannot just restore a snapshot from a few hours earlier, because that would erase everything that happened in between (e.g. new jobs that have been created since the snapshot has been taken).

[tgross]

Hi @carobme and thanks for the additional detail.

While I understand accidental deletion of ACL tokens is somewhat time consuming to recover from, we believe that recovering from security related incidents is a greater risk which is why Nomad does not allow creation of tokens by specifying the secret ID. Storing these IDs as plain text within GitHub repositories is something we strongly advise against and can easily lead to leaked data.

The Nomad Terraform provider, Nomad CLI, and Nomad API all provide convenient and scriptable methods for creating new ACL tokens quickly. Creating new tokens and updating their reference would be our initial suggesting outside of future infrastructure automation to utilize workload identity and short-lived rotating tokens.

That being said, after a quick internal chat there might be something to this idea in order to help out recovering from authoritative region failure. I'm going to mark it for further discussion and possible roadmapping.

[tgross]

It appears that the Consul Create Token API and Vault Create Token API already support this, which gives more evidence to the notion that this is probably fine to do.

[akamensky]

To add some additional point here. In our case we have some tooling (internally built) that is deployed alongside Nomad that needs to authenticate in it. We use Ansible for provisioning of everything (nomad and the tools), to be able to achieve reproducible environment and be able to restore quickly we need tokens to be setup from Ansible. In the current state it is a nightmare to setup tokens in Nomad with Ansible, therefore for our tooling we had to resort to using bootstrap token since it is the only one we can feed in from the Ansible (as opposed to generating new ones every time).

IMO there is a time and place to generating new tokens but forcing it for every case is not ideal. As noted - Consul doesn't have this problem and where we need we set tokens from provisioning systems, for everything else we can generate new ones every time.