Variable substitution for arbitrary Consul keys in Docker image field

[eigengrau]

For a simple-and-stupid approach to CD, it would be convenient to give variable interpolation access to Consul keys. The use-case I’m interested in would place a reference to a Consul key within the Docker driver’s config.image. A CI job could then write Docker image hashes into that key and thus trigger a re-deployment, without needing to communicate with Nomad directly.

As a work-around, Consul Template may be sufficient, but nesting multiple levels of templating is a bit awkward even when different syntactic markers are being used.

This issue may overlap with #1185, but that ticket (at least the ticket topic) refers to expanding references using the local environment. Feel free to close as dupe if appropriate.

Thank you kindly for making Nomad!

[schmichael]

Hi @eigengrau -- great question!

Templating the Docker image field via a Consul template does work. It is a bit convoluted as you have to create environment variables from the template. Here's an example built from the nomad init example job:

job "example" {
  datacenters = ["dc1"]

  group "cache" {
    task "redis" {
      driver = "docker"

      config {
        image = "${IMAGE}"  # <--- Interpolate "IMAGE" env var created below here
        port_map {
          db = 6379
        }
      }

      template {
        data = <<EOF
IMAGE={{ key "image" }}      # <--- Pull Consul key "image" into env var "IMAGE" here
EOF

        # A destination file is written even for env templates to aide in debugging
        destination = "image.env"

        # env = true exposes the IMAGE variable inside the config stanza
        env = true
      }

    # resources and service stanzas removed for brevity
    }
  }
}

If you start the job before the image key exists in Consul, the task will block until it does exist. The task will be pending and you will see Task Events like:

Recent Events:
Time                       Type        Description
2019-01-16T07:24:50-08:00  Template    Missing: kv.block(image)
2019-01-16T07:24:47-08:00  Task Setup  Building Task Directory
2019-01-16T07:24:47-08:00  Received    Task received by client

If you run consul kv put image redis:3.2-alpine, the template will render and the task will start:

2019-01-16T07:25:47-08:00  Started     Task started by client

If you update the Consul key with consul kv put image redis:4.0-alpine, the template will rerender and the task will be restarted:

2019-01-16T07:32:23-08:00  Started     Task started by client
2019-01-16T07:32:22-08:00  Restarting  Task restarting in 0s
2019-01-16T07:32:22-08:00  Terminated  Exit Code: 0
2019-01-16T07:32:22-08:00  Restarting  Template with change_mode restart re-rendered

(Note that Task Events may change slightly between releases, so your output may not match this precisely.)

By default template's have a 5 second splay to prevent all of your tasks from being restarted simultaneously and thus causing an outage. You may need to increase that splay as tasks will be down while downloading the new Docker image.

Alternative: Deployments

There is a high potential for a service outage with the above approach due to 2 factors:

1. If the splay is not long enough, most or all of the tasks may be restarting simultaneously.
2. If there's a bug in the new image, there's no automated rollbacks to the old version.

Nomad Deployments are meant to address these two issues and are configured via the job's update stanza.

To use deployments with CI/CD you would use Consul template or a similar tool to template your job files before submitting them to Nomad. That way your CD tool controls when a deployment happens and what image is being deployed. The downside to this approach being that there are more services to configure than just Nomad and Consul.

I hope this helps!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.