Docker logs growing infinitely

[ygersie]

Nomad version

Nomad v0.9.1 (4b2bdbd)

Issue

Infinitely growing Docker logs. Docker stores the logs under the dockerd --data-root. Nomad seems to be collecting them separately and also performs the nice log rotation there. However, the actual source of the logs is never cleaned. We already have a file which grew to 116GB worth of access logs in a short time, here a du -h snippet:
117G /var/lib/docker/containers/dafbe8ab0f5aec724e49d80aee5ef77824838870460a0182e76c0924b45ae4ba/dafbe8ab0f5aec724e49d80aee5ef77824838870460a0182e76c0924b45ae4ba-json.log

Reproduction steps

Any job producing logs

[gpaggi]

This seems to be a side effect of the recent changes in how Nomad collects logs from the Docker driver.
It changed from syslog to json-file, without max-size/max-file configured.

For the time being I configured Docker to rotate its files in /etc/docker/daemon.json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "2"
  }
}

I know I can pass the logging config to the driver in the job file but it would be nice to be able to do so also in the client config, under the newly introduced plugin/docker/config stanza

[hsmade]

How to add it to a nomad task stanza:

     task "redis" {
      driver = "docker"
      config {
        image = "redis:3.2"
        port_map {
          db = 6379
        }
        logging {
          config {
            max-size= "10m",
            max-file= "10"
          }
        }
      }
....

[camerondavison]

does #5846 fix this issue?

[schmichael]

Oops, yes! Thanks for the reminder @camerondavison !

Please test out 0.9.4-rc1 if you were hit by this: https://groups.google.com/d/msg/nomad-tool/Zi2I7P-PWo0/yAPFPmgLDQAJ

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.