Docker auth.config with basic auth not working in 1.9.3

[RafalGoslawski]

Nomad version

Nomad v1.9.3
BuildDate 2024-11-11T16:35:41Z
Revision d92bf1014886c0ff9f882f4a2691d5ae8ad8131c

Operating system and Environment details

Ubuntu 24.04.1 LTS, Hetzner Cloud

Issue

Pulling with docker on that host works fine, but Nomad fails with:

[ERROR] client.alloc_runner.task_runner: running driver failed: [...] task=server error="Failed to pull `[REDACTED]`: Error response from daemon: Head \"[REDACTED]/manifests/1.0.0\": no basic auth credentials"

# cat /etc/nomad.d/client.hcl 
[...]
plugin "docker" {
  auth {
    config = "/root/.docker/config.json"
  }
}

# cat /root/.docker/config.json 
{
  "auths": {
    "registry.service.consul:5000": {
      "auth": "[REDACTED]"
    }
  }
}

root@nomad-client-2:~# docker pull registry.service.consul:5000/rclone/rclone:1.68
1.68: Pulling from rclone/rclone
da9db072f522: Already exists 
27c3b6ae7a50: Pull complete 
a38d4e9e2bf7: Pull complete 
cbcea06882f6: Pull complete 
a9b7b82c3164: Pull complete 
Digest: sha256:74c51b8817e5431bd6d7ed27cb2a50d8ee78d77f6807b72a41ef6f898845942b
Status: Downloaded newer image for registry.service.consul:5000/rclone/rclone:1.68
registry.service.consul:5000/rclone/rclone:1.68

Reproduction steps

1. Setup nomad to use a private registry with basic auth as mentioned above.
2. Run a job that uses that registry without providing the auth block inside the job.

Expected Result

The job should work,

Actual Result

The job fails with "no basic auth credentials"

[ERROR] client.alloc_runner.task_runner: running driver failed: [...] task=server error="Failed to pull `[REDACTED]`: Error response from daemon: Head \"[REDACTED]/manifests/1.0.0\": no basic auth credentials"

[RafalGoslawski]

I've ran more tests and it doesn't matter if I use a private insecure registry with a port registry.service.consul:5000 or a secure registry with a public domain name and https registry.mydomain.com.

The basic auth doesn't work in either case when set through /root/.docker/config.json on the Nomad client.

[f3l1x]

We're running several Nomad cluster on 1.9.3 version with our internal Gitlab Registry and it's working just fine. I have exactly same configuration.

plugin "docker" {
  config {
    allow_privileged = true

    auth {
      config = "/root/.docker/config.json"
    }

    extra_labels = ["job_name", "job_id", "task_group_name", "task_name", "namespace", "node_name", "node_id"]

    volumes {
      enabled = true
    }
  }
}

It looks like you are using docker.auth.config instead of docker.config.auth.config.

https://developer.hashicorp.com/nomad/docs/drivers/docker#client-requirements

[RafalGoslawski]

@f3l1x Thanks, you're right.

Somehow I've placed auth outside of config... PEBKAC issue. ;)

Closing.

Edit: a warning on unrecognized options would prevent any confusion here.