Nomad is no longer cleaning up unused Docker images.

[cconstantine]

Nomad version

Nomad v0.9.0+ent (dce6497)

Operating system and Environment details

Ubuntu 18.03 running in AWS, Based on AMI ami-22741f5a.
Docker Engine - Community 18.09.3

Issue

After upgrading from nomad 0.8.7+ent to 0.9.0+ent, nomad has stopped cleaning up unused docker images. We don't explicitly configure the cleanup behavior and instead rely on the defaults documented here: https://www.nomadproject.io/docs/drivers/docker.html#docker-cleanup-image.

Reproduction steps

Deploy an updated job with a new docker image.

Nomad Client logs (if appropriate)

root@nomad-client-g19-n1:/nomad/config# docker ps | grep redacted-docker-registry
74096a901e51        redacted-docker-registry:70c1bd3691f98d9e4d0353675dc11e1342f5a851-deploy          "puma -e staging"        40 minutes ago      Up 40 minutes       10.220.192.90:21051->3000/tcp, 10.220.192.90:21051->3000/udp           puma-8d6f9d0f-8ba8-13de-a2c1-b77b20792393
338e710b0001        redacted-docker-registry:d5d3a68c5ff928d922ed661b6d72d3d6d2b0a7d6-branch-deploy   "puma -e staging"        5 hours ago         Up 5 hours          10.220.192.90:21758->3000/tcp, 10.220.192.90:21758->3000/udp           puma-8e7a7ac1-6bde-f81a-469f-4f759ed245f8
root@nomad-client-g19-n1:/nomad/config# docker images | grep redacted-docker-registry
redacted-docker-registry            70c1bd3691f98d9e4d0353675dc11e1342f5a851-deploy          f4cb398834a0        About an hour ago   1.91GB
redacted-docker-registry            b6dfe18e8d26258a748fdb6b2b01521b0b748ea4-deploy          d2defb6e1908        About an hour ago   1.91GB
redacted-docker-registry            5b74917a455b20b0d0cb5a45f53a86110b542204-deploy          ecfeffb37770        2 hours ago         1.91GB
redacted-docker-registry            8cc5562d0f5d2d2a4b1ae5bb15e672b6e1c8c735-deploy          d291c965fd82        3 hours ago         1.91GB
redacted-docker-registry            f31c97b7f01f0d8210e1b06378d0d0a03034b38a-deploy          8f7c5e7d2991        3 hours ago         1.91GB
redacted-docker-registry            d5d3a68c5ff928d922ed661b6d72d3d6d2b0a7d6-branch-deploy   8eaa644db50f        5 hours ago         1.91GB
redacted-docker-registry            4564880a3c6dbc904c95bf7dd4f6d5ce05d700f2-deploy          b61eeef83a15        6 hours ago         1.91GB
redacted-docker-registry            9c843c5cb7e391b8d3f41d8c22e588fb9bc42e5f-deploy          bca9126d02a0        6 hours ago         1.91GB
redacted-docker-registry            e6261e3709d140f0c5d9fc963c51cc755a81091e-deploy          d90db05aaa2e        25 hours ago        1.9GB
redacted-docker-registry            195b9ab181d0398eda6c30fb0bacb8a95c53e77c-deploy          9b20c617884c        27 hours ago        1.9GB
redacted-docker-registry            f6b1c06d1b2fbdb663d839eac93bf52dd76cc93b-deploy          a0c5447f33f3        28 hours ago        1.9GB
redacted-docker-registry            5b4787470c54c92364de5099cd5652cb84854cb9-deploy          7f4de9c9c8be        28 hours ago        1.9GB
redacted-docker-registry            ab1327bc7324895c0e6c57c2b09edff9c7130b00-deploy          64c30e7cc385        29 hours ago        1.9GB
redacted-docker-registry            64a7faff1cf377f65ee6823ac16a6fc209dc6f37-deploy          19624ba6c1db        29 hours ago        1.9GB
redacted-docker-registry            4e66f35d5c7edea3bcc5809607fee7a1023c6555-deploy          395f9e62934b        31 hours ago        1.9GB
redacted-docker-registry            d297d28bd82f02529a83af7660f3a24527d3e31d-deploy          0eb1d8c0b912        4 days ago          1.91GB
redacted-docker-registry            b603673ea8c27534f260319e1aef3f85c5f53876-deploy          0e3af7b47cc0        4 days ago          1.92GB
redacted-docker-registry            f557ae78662ac2960592fa22b1d51a2960ebe61a-deploy          cbe208c9c48f        4 days ago          1.92GB
redacted-docker-registry            77f296565f739391859526401cfe468072e4cb6e-branch-deploy   73fb95e1d3a9        4 days ago          1.9GB
redacted-docker-registry            4e0b92f92cf5db14d22bf001ad059ce4720948d4-deploy          848c2e2edbb2        4 days ago          1.92GB
redacted-docker-registry            bcf62170eb637fd08120c14590aded47bb7f2532-branch-deploy   343ed1517b05        5 days ago          1.6GB

Nomad Server logs (if appropriate)

I don't see anything relevant to this issue in the server logs.

Nomad Client Config (Slightly Redacted)

data_dir  = "/nomad/data"

region = "staging"
datacenter = "staging"

telemetry {
  publish_allocation_metrics = true
  publish_node_metrics = true

  prometheus_metrics = true
}

advertise {
  <REDACTED>
}

client {
  enabled       = true

  max_kill_timeout = "24h"

  options {
    "docker.auth.config" = "/root/.docker/config.json"
  }
}

[schmichael]

Any chance you could try reproducing with 0.9.1? It contains some critical Docker related fixes.

[cconstantine]

Yes, but it's going to be hard to prioritize that over other work without a new feature we'd use, or a fix being called out in the changelog or resolved issue. Particularly with a 0.9.2 release pending.

[pkrolikowski]

Same here.

Nomad version:

Nomad v0.9.1 (4b2bdbd9ab68a27b10c2ee781cceaaf62e114399)

Docker version:

18.06.0-ce

OS:

Description:	Ubuntu 18.04.2 LTS

Nomad config

client {
       ...
       options = {
		"docker.auth.config"         = "/root/.docker/config.json"
		"docker.cleanup.image.delay" = "12h"
	}
}

[cgbaker]

I've also reproduced this on 0.9.1, as well as a recent 0.9.2-dev (fb38029)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.