[bug/regression] nomad 1.0.0 ignores nomad meta.connect.(sidecar|gateway)_image

[ygersie]

Nomad version: 1.0.0

The envoy proxy config set in the client meta are ignored since the introduction of the dynamic envoy version detection. This is not just an issue to configure a more recent envoy version but also because it now by default pulls envoy containers from the docker hub with rate limits.

This is the output from a nomad nodes status -self -verbose and has been unchanged since the 1.0.0 upgrade.

Meta
connect.gateway_image     = envoyproxy/envoy:v1.16.1
connect.log_level         = info
connect.proxy_concurrency = 1
connect.sidecar_image     = envoyproxy/envoy:v1.16.1

And after the nomad upgrade planning the countdash nomad job:

  +/- Task: "connect-proxy-count-dashboard" (forces create/destroy update)
    +/- Config {
          args[0]: "-c"
          args[1]: "${NOMAD_SECRETS_DIR}/envoy_bootstrap.json"
          args[2]: "-l"
          args[3]: "${meta.connect.log_level}"
      +/- args[4]: "--disable-hot-restart" => "--concurrency"
      +   args[5]: "${meta.connect.proxy_concurrency}"
      +   args[6]: "--disable-hot-restart"
      +/- image:   "${meta.connect.sidecar_image}" => "envoyproxy/envoy:v${NOMAD_envoy_version}"
        }

And the associated docker container:

9de76a58cd4b        envoyproxy/envoy:v1.16.0                      "/docker-entrypoint.…"   13 minutes ago      Up 13 minutes                           connect-proxy-count-dashboard-82594608-0a7e-4e8c-8000-2aa5f6ccdd59

[shoenig]

Thanks for reporting @ygersie , this does look like a regression.

As a workaround for sidecar proxies, sidecar_task can be specified with the desired image, e.g.

sidecar_task {
  config {
    image = "custom/envoy"
  }
}

but unfortunately that doesn't work yet for gateways (#9474).

I'll start working on a fix for this, but if the situation is urgent you may want to look into options like pre-pulling the images onto your hosts so they're available without going through docker's ratelimiting, or configuring your docker daemon's to default to an internal registry.

[idrennanvmware]

Thanks for the info @shoenig . We were just about to push out 1.0.0 and will hold off now as we don't want to get hit by this issue (we rely on the meta tag specifically to avoid the rate limiting issue).

Will wait on 1.0.1 :D

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.