Regression: ${HOST_IP} in ConsulResolver does not work without legacy mode

[lawliet89]

Describe the bug

If you configure a ConsulResolver with the HOST_IP environment variable as the address for Consul, Ambassador reports errors trying to resolve the ${HOST_IP} DNS name.

dial tcp: lookup ${HOST_IP}: no such host

Reverting back to legacy mode (by setting AMBASSADOR_LEGACY_MODE = "true") fixes it.

To Reproduce

Consider the following ConsulResolver:

apiVersion: getambassador.io/v2
kind: ConsulResolver
metadata:
  labels:
    app.kubernetes.io/instance: ambassador-internal
    app.kubernetes.io/managed-by: Terraform
    app.kubernetes.io/name: ambassador
    app.kubernetes.io/part-of: ambassador-internal
  name: ambassador-internal-consul
  namespace: core
spec:
  address: ${HOST_IP}:8501
  ambassador_id: internal
  datacenter: asia-southeast1

Ambassador boots with the error dial tcp: lookup ${HOST_IP}: no such host:

2021/01/28 05:33:10 Started Ambassador
2021/01/28 05:33:10 AMBASSADOR_CLUSTER_ID=9396113a-9118-53f1-9e25-45af3a51ca80
time="2021-01-28 05:33:10" level=info msg="[pid:14] started command []string{\"diagd\", \"/ambassador/snapshots\", \"/ambassador/bootstrap-ads.json\", \"/ambassador/envoy/envoy.json\", \"--notices\", \"/ambassador/notices.json\", \"--port\", \"8004\", \"--kick\", \"kill -HUP 1\"}" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/[email protected]/dexec/cmd.go:143" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-28 05:33:10" level=info msg="[pid:14] stdin  < not logging input read from file /dev/stdin" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/[email protected]/dexec/cmd.go:145" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-28 05:33:10" level=info msg="[pid:14] stdout > not logging output written to file /dev/stdout" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/[email protected]/dexec/cmd.go:148" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-28 05:33:10" level=info msg="[pid:14] stderr > not logging output written to file /dev/stderr" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/[email protected]/dexec/cmd.go:151" CMD=entrypoint PID=1 THREAD=/diagd
2021/01/28 05:33:10 Warning, unable to watch ingressclasses.networking.k8s.io, unknown kind.
2021-01-28 05:33:12 diagd 1.11.0 [P14TMainThread] INFO: AMBASSADOR_FAST_RECONFIGURE disabled, not initializing cache
2021-01-28 05:33:12 diagd 1.11.0 [P14TMainThread] INFO: WILL NOT update Mapping status
2021-01-28 05:33:12 diagd 1.11.0 [P14TMainThread] INFO: thread count 9, listening on 0.0.0.0:8004
2021-01-28 05:33:12 diagd 1.11.0 [P14TMainThread] INFO: Ambassador 1.11.0 booted
[2021-01-28 05:33:12 +0000] [14] [INFO] Starting gunicorn 20.0.4
[2021-01-28 05:33:12 +0000] [14] [INFO] Listening at: http://0.0.0.0:8004 (14)
[2021-01-28 05:33:12 +0000] [14] [INFO] Using worker: threads
[2021-01-28 05:33:12 +0000] [19] [INFO] Booting worker with pid: 19
2021-01-28 05:33:12 diagd 1.11.0 [P19TAEW] INFO: starting Scout checker and timer logger
2021-01-28 05:33:12 diagd 1.11.0 [P19TAEW] INFO: starting event watcher
2021/01/28 05:33:15 Taking K8sSecret snapshot.SecretRef{Namespace:"core", Name:"ambassador-consul-connector-internal"}
time="2021-01-28 05:33:15" level=info msg="[ERR] consul.watch: Watch (type: service) errored: Get \"https://$%7BHOST_IP%7D:8501/v1/health/service/www-sidecar-proxy?passing=1\": dial tcp: lookup ${HOST_IP}: no such host, retry in 5s" CMD=entrypoint PID=1 oops-i-did-not-pass-context-correctly=true
2021-01-28 05:33:19 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11eec63e80>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:33:19 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: 672D209A-FB6D-46A5-A95A-7129911E8857: 127.0.0.1 "GET /metrics" 12ms 200 success
2021/01/28 05:33:20 Memory Usage 0.28Gi (18%)
    PID 1, 0.26Gi: busyambassador entrypoint
    PID 14, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1
    PID 19, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1
time="2021-01-28 05:33:20" level=info msg="[ERR] consul.watch: Watch (type: service) errored: Get \"https://$%7BHOST_IP%7D:8501/v1/health/service/www-sidecar-proxy?passing=1\": dial tcp: lookup ${HOST_IP}: no such host, retry in 20s" CMD=entrypoint PID=1 oops-i-did-not-pass-context-correctly=true
2021-01-28 05:33:38 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11eec81c40>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:33:38 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: 42B69495-49D9-4E09-92CB-3EAFB2F64E9A: 127.0.0.1 "GET /metrics" 8ms 200 success
time="2021-01-28 05:33:40" level=info msg="[ERR] consul.watch: Watch (type: service) errored: Get \"https://$%7BHOST_IP%7D:8501/v1/health/service/www-sidecar-proxy?passing=1\": dial tcp: lookup ${HOST_IP}: no such host, retry in 45s" CMD=entrypoint PID=1 oops-i-did-not-pass-context-correctly=true
2021-01-28 05:33:42 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11eec81cd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:33:42 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: 8576C2E1-4B80-4531-9A5C-6479F087FC71: 127.0.0.1 "GET /metrics" 9ms 200 success
2021-01-28 05:33:49 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11eec88a90>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:33:49 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: FD6A8A3F-8878-41CB-B5FB-60D75B26E097: 127.0.0.1 "GET /metrics" 7ms 200 success
2021-01-28 05:34:08 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11eec928e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:34:08 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: 28D57CE9-15A1-4373-A1F6-1B2039767827: 127.0.0.1 "GET /metrics" 9ms 200 success
2021-01-28 05:34:11 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11eec928b0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:34:11 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: BFDDCBE7-E5CB-4B3B-9917-8DA376E193F4: 127.0.0.1 "GET /metrics" 7ms 200 success
2021-01-28 05:34:19 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11eec886a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:34:19 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: 8B199183-00E8-4FE2-8BFA-7F764307BA73: 127.0.0.1 "GET /metrics" 7ms 200 success
time="2021-01-28 05:34:25" level=info msg="[ERR] consul.watch: Watch (type: service) errored: Get \"https://$%7BHOST_IP%7D:8501/v1/health/service/www-sidecar-proxy?passing=1\": dial tcp: lookup ${HOST_IP}: no such host, retry in 1m20s" CMD=entrypoint PID=1 oops-i-did-not-pass-context-correctly=true
2021-01-28 05:34:38 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11eec63af0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:34:38 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: 6CDBB6C4-4AE1-40B5-BA6D-9475A936CE70: 127.0.0.1 "GET /metrics" 7ms 200 success
2021-01-28 05:34:41 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11ee812790>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:34:41 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: FD4D0846-078E-42BD-A89F-50C88EF20978: 127.0.0.1 "GET /metrics" 8ms 200 success
2021-01-28 05:34:49 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] WARNING: EnvoyStats.get_prometheus_state failed: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /stats/prometheus (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f11ee81a5b0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-01-28 05:34:49 diagd 1.11.0 [P19TThreadPoolExecutor-0_0] INFO: 82CDAC14-399F-4BC5-8B1B-FE5E21E255C4: 127.0.0.1 "GET /metrics" 8ms 200 success

Setting the environment variable AMBASSADOR_LEGACY_MODE = "true" fixes it.

Versions (please complete the following information):

• Ambassador: [e.g. 0.32.1] 1.11.0
• Kubernetes environment [e.g. Minikube, bare metal, Google Kubernetes Engine] GKE

Ambassador was deployed using the helm chart, and I have verified that the HOST_IP environment variable is set correctly on the pods.

[khussey]

This is fixed in Ambassador 1.13.0, which is now available.