ACL tokens of type 'client' are unable to use the nomad logs command?

[henrikjohansen]

Nomad version

Output from nomad version

Nomad v0.9.1+ent

Issue

An ACL token with write access to a namespace can execute nomad fs against an allocation running in that namespace but nomad logs fails with a 403 error.

Reproduction steps

Create ACL policy :

$ nomad acl policy info testacl
Name        = testacl
Description = <none>
Rules       = namespace "testacl" {
  policy = "write"
}

# Allow read access to the Quota API
quota {
  policy = "read"
}
CreateIndex = 101545
ModifyIndex = 101545

Create ACL token :

$ nomad acl token create -policy=testacl
Accessor ID  = 494cf903-7c57-f8ea-fe73-87f7bd31c210
Secret ID    = e7636eb1-ac9f-39be-c7bc-6201c184e3d7
Name         = <none>
Type         = client
Global       = false
Policies     = [testacl]
Create Time  = 2019-05-22 19:39:19.625537207 +0000 UTC
Create Index = 103847
Modify Index = 103847 

Export ENV vars for the CLI :

$ export NOMAD_NAMESPACE=testacl
$ export NOMAD_TOKEN=e7636eb1-ac9f-39be-c7bc-6201c184e3d7

Run nomad commands

$ nomad status
ID    Type     Priority  Status   Submit Date
foo  service  50        running  2019-05-22T13:52:33+02:00

$ nomad status foo
ID            = foo
Name          = foo
Submit Date   = 2019-05-22T13:52:33+02:00
Type          = service
Priority      = 50
Datacenters   = XXXX
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group        Queued  Starting  Running  Failed  Complete  Lost
foo               0       0         1        0      2         0

Allocations
ID        Node ID   Task Group        Version  Desired  Status   Created    Modified
b13bef10  8ffa4632  foo               4        run      running  7h50m ago  7h49m ago

$ nomad fs b13bef10
Mode        Size     Modified Time         Name
drwxrwxrwx  4.0 KiB  2019-05-22T11:52:33Z  alloc/
drwxrwxrwx  4.0 KiB  2019-05-22T11:52:34Z  foo/

$ nomad logs b13bef10
Error reading file: Unexpected response code: 403 (Permission denied)

... even with capabilities = ["read-logs"] explicitly set in the ACL policy the above still fails ...
When changing to a 'management' token nomad logs works as expected.

[cgbaker]

Hi @henrikjohansen , I was able to reproduce this. Looking at the logs, it seems that the 403 is coming from inspection of the node; adding the following to your policy should allow the viewing of logs to proceed:

node { policy = "read" }

The fact that this ACL is needed for log viewing but not needed filesystem access feels buggy; we are going to look at the code and determine whether this is correct or not. I'll mark this as a bug for now and update the issue with our determination.

Update: This isn't a bug, it is a result of the CLI looking up the allocation Node info in order to stream directly from the node. A direct call for the logs using the client API does not have this problem:
https://www.nomadproject.io/api/client.html#stream-logs

[henrikjohansen]

@cgbaker In our case, granting read access to nodes is not really ideal. In a low-trust, multi-tenant setup tenants should have access to their own jobs but not much else ...

Update: I have opened #5761 to clarify the need to better granularity in ACL policies.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.