Fixes Consul token checking when policies exist within namespaces

[t-davies]

Re: support case #122732

Fixes a bug where Nomad fails to fetch Consul policies whilst checking a Consul token presented when submitting a job, if the policies attached to that token exist within a namespace.

[ERROR] agent.http: Request error: method=GET url=/v1/acl/policy/83073614-xxxx-xxxx-xxxx-xxxxxxxxxxxx from=127.0.0.1:54668 error="Requested policy does not exist: ACL not found"

• When querying policies from Consul, use the namespace of the token rather than the default namespace.
  • This should be possible since rules and policies within a specific namespace are prevented from accessing resources in another namespace, therefore if the token is from a non-default namespace then all of it's policies must also exist within that namespace?
  • We cannot use the namespace from the job spec because tokens created in the "default" Consul namespace can have permissions granted in other namespaces - also job spec Consul namespaces are Enterprise-only.

[pkazmierczak]

Hey @t-davies, thanks for the bugfix, the code looks good to me!

FYI in Nomad 1.7 we will be introducing support for Workload Identity tokens, so that jobs will be able to login into Consul via JWT https://www.hashicorp.com/blog/nomad-1-7-improves-vault-and-consul-integrations-adds-numa-support

[shoenig]

The actual fix LGTM but I think the e2e test needs to account for the namespace'd token on deletion

=== RUN   TestE2E/ConnectACLs/*connect.ConnectACLsE2ETest/TestConnectACLsConnectDemoNamespaced
    acls.go:272: test register Connect job w/ ACLs enabled w/ operator token
    acls.go:278: created namespace: ns-a612e32f
    acls.go:286: created operator policy: 2797be55-60b8-8797-c2db-d119405a7342
    acls.go:290: created operator token: a9828584-2ec1-b352-d31d-c4d69ef40abb
    acls.go:307: connect legacy job with ACLs enable finished
    acls.go:67: cleanup: deregister nomad job id: connectfaaa960e
    acls.go:74: cleanup: delete consul token id: 4e616feb-0a73-40e0-966f-b8412a8d3954
    acls.go:76:
                Error Trace:    /home/shoenig/Go/nomad/e2e/connect/acls.go:76
                                                        /home/shoenig/Go/nomad/e2e/framework/framework.go:265
                                                        /home/shoenig/Go/nomad/e2e/framework/framework.go:271
                Error:          Received unexpected error:
                                Unexpected response code: 404 (Cannot find token to delete)
                Test:           TestE2E/ConnectACLs/*connect.ConnectACLsE2ETest/TestConnectACLsConnectDemoNamespaced

[shoenig]

.

[t-davies]

The actual fix LGTM but I think the e2e test needs to account for the namespace'd token on deletion

Thanks, good spot. I'm passing Namespace when deleting these (and the policies) now - also noticed we didn't clean up the namespaces either so getting rid of these too.

Once the tests run and are happy, I'll rebase the fixup!.

[shoenig]

LGTM; thanks @t-davies!