Purging jobs not working consistently between methods

[marknl]

Nomad version

1.5.6

Operating system and Environment details

Ubuntu 22.04

Issue

Purging a job is not working consistently when using the CLI and Web UI. There's also a difference when purging jobs within a namespace with a namespace specific token.

Reproduction steps

1. sudo nomad agent -dev -acl-enabled
2. nomad acl bootstrap
3. export NOMAD_TOKEN=<bootstrap_token>
4. nomad namespace apply app1
5. Create policy file policy-app1.hcl:
   namespace "app1" { policy = "write" }
6. nomad acl policy apply app1 policy-app1.hcl
7. nomad acl token create -name "app1" -policy app1
8. nomad job init
9. NOMAD_TOKEN=<app1_token> NOMAD_NAMESPACE=app1 nomad job run example.nomad.hcl

Expected Result

Whatever method I use to purge a job, it should not show up in the Web UI or job list after purging, nor should I get an ACL error when I purge a job in a namespace when that token has policy = "write" or submit-job in it.

Actual Result

There are 3 different outcomes:

1. Purge via CLI with app1 token

You run NOMAD_TOKEN=<app1_token> NOMAD_NAMESPACE=app1 nomad job stop -purge example and the job is stopped, purged and does not show up in the Web UI. All good, no errors.

2. Purge via Web UI with app1 token

1. NOMAD_TOKEN=<app1_token> nomad ui -authenticate
2. Go to: http://127.0.0.1:4646/ui/jobs/example@app1
3. Click on: "Stop Job", "Yes, Stop Job", "Purge Job", "Yes, Purge Job".

Result: Error purging job
Your ACL token does not grant permission to purge jobs.
Job is shown "Dead" in Web UI.

3. Purge via Web UI with management token

1. nomad ui -authenticate
2. Go to: http://127.0.0.1:4646/ui/jobs/example@app1
3. Click on: "Stop Job", "Yes, Stop Job", "Purge Job", "Yes, Purge Job".

Result: No ACL error. Job is shown "Dead" in Web UI, not purged.

Job file

You can use the example one that nomad gives you with nomad job init.

Nomad Server logs

(When pressing "Yes, Purge Job" with app1 token)
2023-05-31T12:08:24.368+0200 [DEBUG] http: request failed: method=DELETE path=/v1/job/example?namespace=app1?purge=true error="Permission denied" code=403

(When pressing "Yes, Purge Job" with management token)
2023-05-31T12:11:49.952+0200 [DEBUG] http: request complete: method=DELETE path=/v1/job/example?namespace=app1?purge=true duration="386.888µs"

Additional info

I can see the correct X-Nomad-Token being send with the purge request.

[lgfa29]

Thanks for the report @marknl.

Looking the server logs you provided it seems like the UI may be formatting the purge request incorrectly:

/v1/job/example?namespace=app1?purge=true 

Should've been:

/v1/job/example?namespace=app1&purge=true 

This may result in of these parameters to be ignored, which may explain the inconsistent results.

I have placed this in our roadmap for further triage and investigation.

[juberstine-godaddy]

Same issue. Confirmed same problems 1.5.6

[marknl]

I think #19139 fixes this issue.