Prometheus Metric Hostnames could lead to DoS conditions?

[WillPlatnick]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):
No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):
metrics

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Hello,
In looking at my prometheus metrics, I currently have over 100,000. The reason for this is because of security scans that are hitting my servers with hostnames other than ones I have set in my ingress, which creates brand new sets of metrics that we don't care about.

This behavior is filling up my Prometheus server at an increasing rate because of junk data, and is causing my 3rd party scraper (datadog) to have issues scraping because there's so many metrics. I'm also wondering if this behavior could potentially be used to create a Denial of Service condition in either the ingress controller or prometheus installs that scrape controllers.

We have a source of truth about what hostnames we care about in our ingress declarations. Perhaps we should limit hostnames in metrics to those to make sure we don't get junk data?

[ElvinEfendi]

@WillPlatnick could this be related to the fact that host is a reserved key in DataDog and treated differently?

To get around this we use labels_mapper:

ad.datadoghq.com/nginx-ingress-controller.instances: '[{"prometheus_url" : "http://%%host%%:10254/metrics","namespace": "ingress_nginx","metrics": ["*"], "labels_mapper": {"host":"dest_host"}}]'

[WillPlatnick]

No, I mean this. blackbiz.ws isn't us, someone just tried to access our ingress controller using that hostname. nginx-ingress creates 65 unique metrics everytime a new hostname gets used. If they use HEAD, POST, OPTIONS, for the request it will generate even more.

nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="100"} 0
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="1000"} 2
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="10000"} 2
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="100000"} 2
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="1e+06"} 2
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="1e+07"} 2
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="+Inf"} 2
nginx_ingress_controller_bytes_sent_sum{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 507
nginx_ingress_controller_bytes_sent_count{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.005"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.01"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.025"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.05"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.1"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.25"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.5"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="1"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="2.5"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="5"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="10"} 2
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="+Inf"} 2
nginx_ingress_controller_request_duration_seconds_sum{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 0.003
nginx_ingress_controller_request_duration_seconds_count{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 2
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="10"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="20"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="30"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="40"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="50"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="60"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="70"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="80"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="90"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="100"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="+Inf"} 2
nginx_ingress_controller_request_size_sum{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 269
nginx_ingress_controller_request_size_count{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.005"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.01"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.025"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.05"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.1"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.25"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.5"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="1"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="2.5"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="5"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="10"} 2
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="+Inf"} 2
nginx_ingress_controller_response_duration_seconds_sum{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 0
nginx_ingress_controller_response_duration_seconds_count{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 2
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.005"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.01"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.025"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.05"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.1"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.25"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="0.5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="1"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="2.5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="10"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404",le="+Inf"} 2
nginx_ingress_controller_response_size_sum{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 507
nginx_ingress_controller_response_size_count{controller_class="nginx",controller_namespace="nginx-ingress",controller_pod="nginx-ingress-lb-hnktb",host="blackbiz.ws",ingress="",method="GET",namespace="",path="/",service="",status="404"} 2

[aledbf]

@WillPlatnick thank you for the report. To answer your question, yes the ingress controller doesn't filter the traffic stats we send to prometheus. Same thing for the logs in nginx.
As you said, this could lead to a DOS of the prometheus server.
I'll see how to avoid this, adding additional information in the log about this anomaly

[aledbf]

@WillPlatnick while #3118 is merged and 0.20 is released, you can use quay.io/kubernetes-ingress-controller/nginx-ingress-controller:dev to test the fix

[towolf]

Also, the histograms for the bytes-based metrics do not make sense. That would reduce this problem by 2/3.

[aledbf]

@towolf thank you for your feedback. Providing metrics for something generic like the ingress controller is one of those situations where we can say one size fits all is hard.
We are exploring how to improve this feature, giving to the users the tools to customize what should be exposed.