-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setting envoy admin IP to anything besides 127.0.0.1 breaks prometheus metrics #10747
Comments
@alexdulin: thank you for raising this issue with such detail, including diagnosing the likely root cause! Your efforts will make this issue much easier to address. Assuming that hard-coded IP address you linked is the root cause, the bind address should be available in that context as Message to all: |
Configure the self_admin cluster to use the admin bind address provided when starting Envoy. Fixes #10747
Configure the self_admin cluster to use the admin bind address provided when starting Envoy. Fixes #10747
Configure the self_admin cluster to use the admin bind address provided when starting Envoy. Fixes #10747
Configure the self_admin cluster to use the admin bind address provided when starting Envoy. Fixes #10747
Configure the self_admin cluster to use the admin bind address provided when starting Envoy. Fixes #10747
Configure the self_admin cluster to use the admin bind address provided when starting Envoy. Fixes #10747
Configure the self_admin cluster to use the admin bind address provided when starting Envoy. Fixes #10747
Hi @alexdulin, thanks for reporting this issue. A fix has been merged and will be available in the next Consul 1.8, 1.9, and 1.10 patch releases toward the end of the month. |
Thanks for getting a fix in and following up on it. Much appreciated! Look forward to a new release. |
Configure the self_admin cluster to use the admin bind address provided when starting Envoy. Fixes #10747
Overview of the Issue
Running an envoy proxy with
consul connect envoy
and specifying the-admin-bind
to an IP that is not127.0.0.1
breaks prometheus metrics because theself_admin
cluster does not receive the correct IP for the admin listener - it will always be127.0.0.1
, regardless of what theconsul connect envoy
command specified. This makes it impossible to bind the admin listener to an IP other than127.0.0.1
and be able to correctly scrape prometheus metrics.My guess is because the IP is hard-coded into the bootstrap command and cannot be changed, regardless of what the admin bind flag was set to.
This problem was discovered due to a recent "bug fix" in Nomad that results in the admin listener for envoy sidecars to bind to
127.0.0.2
instead of127.0.0.1
: hashicorp/nomad#10883. The issue makes it impossible to use Nomad 1.1.3 and collect prometheus metrics from envoy.Reproduction Steps
curl -s 127.0.0.2:19002/listeners
. This should show it registered a prometheus listener with output like the followingself_admin
will have the wrong IP of127.0.0.1
, not127.0.0.2
. Runningcurl -s 127.0.0.2:19002/clusters | grep self_admin | sort
confirms this with output like the following:curl -s localhost:9114/metrics
results in a 503:Consul info for both Client and Server
Client info
Operating system and Environment details
envoy --version
The text was updated successfully, but these errors were encountered: