Docker driver consistently fails with Unexpected response code: 500 to pull image from GCE repo, but docker pull succeeds

[denGitHub]

Nomad version

Output from nomad version
Nomad v0.7.0

Operating system and Environment details Vagrant box:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/xenial64"
  config.vm.provider "virtualbox"
  config.vm.hostname = "ubuntu"
  config.vm.provision "docker" # Just install it
end

Issue

I am trying to pull image from GCE in nomad task but receive the error:
"message":"unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"

I have the same issue with AWS ECR as well

Though pulling/pushing manually is working fine

Reproduction steps

1. Vagrant up
2. Vagrant ssh
3. Did steps from https://cloud.google.com/sdk/docs/
4. Did steps from https://cloud.google.com/container-registry/docs/advanced-authentication
5. Installed nomad from https://www.nomadproject.io/downloads.html , linux:64
6. nomad agent -dev -config=/etc/nomad.d/config.json
7. nomad run example.nomad

Nomad Server logs (if appropriate)

2018/04/26 17:18:06.049815 [ERR] driver.docker: failed pulling container gcr.io/masternodez-202313/webapp:latest: API error (500): {"message":"unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"}
   2018/04/26 17:18:06.049880 [WARN] client: error from prestart: failed to initialize task "redis" for alloc "35eae760-4716-2371-cd41-ac583e4b95e5": Failed to pull `gcr.io/masternodez-202313/webapp:latest`: API error (500): {"message":"unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"}
   2018/04/26 17:18:06.049932 [INFO] client: Restarting task "redis" for alloc "35eae760-4716-2371-cd41-ac583e4b95e5" in 25.362201382s
   2018/04/26 17:18:06.161119 [DEBUG] client: updated allocations at index 119 (total 3) (pulled 0) (filtered 3)
   2018/04/26 17:18:06.162067 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 3)
   2018/04/26 17:18:31.413077 [DEBUG] client: driver event for alloc "35eae760-4716-2371-cd41-ac583e4b95e5": Downloading image gcr.io/masternodez-202313/webapp:latest
   2018/04/26 17:18:31.562071 [DEBUG] client: updated allocations at index 120 (total 3) (pulled 0) (filtered 3)
   2018/04/26 17:18:31.563588 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 3)
   2018/04/26 17:18:33.275239 [ERR] driver.docker: failed pulling container gcr.io/masternodez-202313/webapp:latest: API error (500): {"message":"unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"}
   2018/04/26 17:18:33.275329 [WARN] client: error from prestart: failed to initialize task "redis" for alloc "35eae760-4716-2371-cd41-ac583e4b95e5": Failed to pull `gcr.io/masternodez-202313/webapp:latest`: API error (500): {"message":"unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"}
   2018/04/26 17:18:33.275404 [INFO] client: Restarting task "redis" for alloc "35eae760-4716-2371-cd41-ac583e4b95e5" in 26.065578779s
   2018/04/26 17:18:33.361236 [DEBUG] client: updated allocations at index 121 (total 3) (pulled 0) (filtered 3)
   2018/04/26 17:18:33.362100 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 3)
   2018/04/26 17:18:59.341996 [DEBUG] client: driver event for alloc "35eae760-4716-2371-cd41-ac583e4b95e5": Downloading image gcr.io/masternodez-202313/webapp:latest
   2018/04/26 17:18:59.560479 [DEBUG] client: updated allocations at index 123 (total 3) (pulled 0) (filtered 3)
   2018/04/26 17:18:59.560574 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 3)

### Nomad Client logs (if appropriate)
vagrant@ubuntu:~$ nomad logs example
Error querying allocation: Unexpected response code: 500 (alloc lookup failed: index error: Invalid UUID: encoding/hex: invalid byte: U+0078 'x')

Job file (if appropriate)

job "example" {
  datacenters = ["dc1"]
  type = "service"
  update {
    max_parallel = 1
    min_healthy_time = "10s"
    healthy_deadline = "3m"
    auto_revert = false
    canary = 0
  }

  group "cache" {
   count = 1
   restart {
      # The number of attempts to run the job within the specified interval.
      attempts = 10
      interval = "5m"
      delay = "25s"
      mode = "delay"
    }

    ephemeral_disk {
        size = 300
    }

    task "redis" {
      config {
        image = "gcr.io/masternodez-202313/webapp:latest"
	ssl = true
        auth {
              server_address = "https://gcr.io"
        }
        port_map {
          db = 6379
        }
      }
         resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
        network {
          mbits = 10
          port "db" {}
        }
      }

      service {
        name = "global-redis-check"
        tags = ["global", "cache"]
        port = "db"
        check {
          name     = "alive"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

Docker ~/.docker/config.json file (if appropriate)

{
  "auths": {
    "https://eu.gcr.io": {
      "email": "[email protected]", 
      "auth": "b2F1dGgyYWGSZXNzdG9rZW46eWEyOS5HbHVwQmJ6SzBuWks0S0k5Y3NTZklNSFVhOGo3YUNySWMydjdWTDJiVDI3NjBsQjZ1JncEFEUVhhaEc1Nl93aFF2U016cVF6VjkzYVBIMUtzcjlhand0Q01oMzBfMnBvMnB1V08tMFdTc3NoSko5YWh4Q042WFFk"
    }, 
    "https://staging-k8s.gcr.io": {
      "email": "[email protected]", 
      "auth": "b2F1dGgyYWNjZXNzdG9rZW46eWEyOS5HbHVwQmJ6SzBuWks0S0k5Y3NTZklNSFVhOGo3YUNySWMydjdWTDJiVDI3NjBsQjsdT1JncEFEUVhhaEc1Nl93aFF2U016cVF6VjkzYVBIMUtzcjlhand0Q01oMzBfMnBvMnB1V08tMFdTc3NoSko5YWh4Q042WFFk"
    }, 
    "https://asia.gcr.io": {
      "email": "[email protected]", 
      "auth": "b2F1dGgyYWNjZXNzdG9rZW46dfEyOS5HbHVwQmJ6SzBuWks0S0k5Y3NTZklNSFVhOGo3YUNySWMydjdWTDJiVDI3NjBsQjZFEUVhhaEc1Nl93aFF2U016cVF6VjkzYVBIMUtzcjlhand0Q01oMzBfMnBvMnB1V08tMFdTc3NoSko5YWh4Q042WFFk"
    }, 
    "https://gcr.io": {
      "email": "[email protected]", 
      "auth": "b2F1dGgyYWNjZXNzdG9rZW4xbWEyOS5HbHVwQmJ6SzBuWks0S0k5Y3NTZklNSFVhOGo3YUNySWMydjdWTDJiVDI3NjBsQjZyTcEFEUVhhaEc1Nl93aFF2U016cVF6VjkzYVBIMUtzcjlhand0Q01oMzBfMnBvMnB1V08tMFdTc3NoSko5YWh4Q042WFFk"
    }, 
    "https://us.gcr.io": {
      "email": "[email protected]", 
      "auth": "b2F1dGgyYWNjZXNzdG9rZW46eWEyOscHbHVwQmJ6SzBuWks0S0k5Y3NTZklNSFVhOGo3YUNySWMydjdWTDJiVDI3NjBsQjZyTl96N1JncVhhaEc1Nl93aFF2U016cVF6VjkzYVBIMUtzcjlhand0Q01oMzBfMnBvMnB1V08tMFdTc3NoSko5YWh4Q042WFFk"
    }
  }, 
  "HttpHeaders": {
    "User-Agent": "Docker-Client/18.03.1-ce (linux)"
  }
}

Nomad config /etc/nomad.d/config.json (if appropriate)

{
  "client": {
    "enabled": true,
    "options": {
      "docker.auth.config": "~/.docker/config.json",
      "docker.auth.helper": "gcr"
    }
}
}

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.