Few metrics are missing in Ingress nginx #5755

thamaraiselvam · 2020-06-23T06:39:05Z

NGINX Ingress controller version: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0

Kubernetes version (use kubectl version): Server Version: v1.14.10-gke.27

Environment:

Cloud provider or hardware configuration: GCP
OS (e.g. from /etc/os-release): Container OS
Kernel (e.g. uname -a): Linux nginx-ingress-controller-5485cb45c7-48qfp 4.14.138+ Basic structure #1 SMP Tue Sep 3 02:58:08 PDT 2019 x86_64 Linux

What happened:

After fresh installed nginx ingress controller with metrics enabled. we are not able to see few metrics.

go_goroutines Number of goroutines that currently exist.
go_goroutines gauge
nginx_ingress_controller_bytes_sent The number of bytes sent to a client
nginx_ingress_controller_bytes_sent histogram
nginx_ingress_controller_ingress_upstream_latency_seconds Upstream service latency per Ingress
nginx_ingress_controller_ingress_upstream_latency_seconds summary
nginx_ingress_controller_request_duration_seconds The request processing time in milliseconds
nginx_ingress_controller_request_duration_seconds histogram
nginx_ingress_controller_request_size The request length (including request line, header, and request body)
nginx_ingress_controller_request_size histogram
nginx_ingress_controller_requests The total number of client requests.
nginx_ingress_controller_requests counter
nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server
nginx_ingress_controller_response_duration_seconds histogram
nginx_ingress_controller_response_size The response length (including request line, header, and request body)
nginx_ingress_controller_response_size histogram

What you expected to happen:

We are able to see above metrics in different cluster with same ingress nginx controller version, we are not sure why in this fresh install those are metrics are missing.

We also verified deployment, configmap and other setting, all are same for both installation but still it is not working for new installer.

How to reproduce it:

Install following helm chart with version 1.36.2

helm install stable/nginx-ingress --name ingress --set controller.metrics.enabled=true --version 1.36.2

after installed, go into ingress controller and check

curl localhost:10254/metrics

Additional information:

We also put load for ingress controller but still could not see those metrics.

The text was updated successfully, but these errors were encountered:

kundan2707 · 2020-06-26T02:51:04Z

/assign

Bo0km4n · 2020-06-26T03:49:51Z

@thamaraiselvam Can you show me applied ingress yaml?
I guess may be you didn't specify ingress host.

Bo0km4n · 2020-06-26T03:55:51Z

If you didn't specify ingress host, ingress status is as below:

$ kubectl get ingress
NAME                     CLASS     HOSTS              ADDRESS        PORTS   AGE
example-ingress   <none>   *                         10.8.201.111   80      5m36s

When HOSTS parameter is *,
execute this if statement:

ingress-nginx/internal/ingress/metric/collectors/socket.go

Lines 233 to 235 in d825859

    
           if !sc.hosts.Has(stats.Host) { 
        
           	klog.V(3).Infof("skiping metric for host %v that is not being served", stats.Host) 
        
           	continue

So you can't see some metrics such as nginx_ingress_controller_request

aedorado · 2020-06-26T04:55:01Z

Hi @Bo0km4n this is a sample ingress in my minikube

web-ingress   <none>   hello-world.info             80      37h

And I also verified that all our ingress in the production cluster have HOSTS defined. However in both the cases, I am getting the same issue that some of these metrics are missing.

Bo0km4n · 2020-06-26T06:25:38Z

@aedorado I see.
Did you exec curl command like as: curl localhost ?
You should specify host of header like this: curl -H "host: hello-world.info" localhost.
Therefore, nginx process write metrics data about request to unix domain socket.
Ingress controller read this data from that socket, and then it outputs some metrics to user.
Please try it.

Bo0km4n · 2020-06-26T06:35:04Z

/assign

aedorado · 2020-06-28T16:17:33Z

I get the response below:

$ curl -H "host: hello-world.info" localhost
curl: (7) Failed to connect to localhost port 80: Connection refused

Earlier I was curling it as curl hello-world.info

Besides, generally we open the URLs in browser or make API calls using the ingress, so would expect the ingress to send metrics for all the scenarios.

Bo0km4n · 2020-06-29T00:57:08Z

@aedorado localhost address is example and depends on your environment.
Please specify ingress IP address that allocated by your Kubernetes cluster.

aedorado · 2020-06-29T05:23:33Z

Thank you @Bo0km4n

In this case address is not there

➜  ~ k get ingress
NAME          CLASS    HOSTS              ADDRESS   PORTS   AGE
web-ingress   <none>   hello-world.info             80      4d13h

Bo0km4n · 2020-06-29T18:05:26Z

@danderson Please note your environment and manifests(Service, Deployment, Ingress)

aedorado · 2020-07-01T05:56:33Z

Hey @Bo0km4n can you be little more elaborate?

Bo0km4n · 2020-07-01T07:07:41Z

@aedorado Ok,
Tell me which you are using kubernetes environment such as microk8s, minikube, kind.
I want to see yaml manifests of Deployment resource and Ingress resource you applied.

aedorado · 2020-07-02T05:05:28Z

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2020-06-24T15:01:54Z"
  generation: 1
  labels:
    app: web
  managedFields:
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:app: {}
      f:spec:
        f:progressDeadlineSeconds: {}
        f:replicas: {}
        f:revisionHistoryLimit: {}
        f:selector:
          f:matchLabels:
            .: {}
            f:app: {}
        f:strategy:
          f:rollingUpdate:
            .: {}
            f:maxSurge: {}
            f:maxUnavailable: {}
          f:type: {}
        f:template:
          f:metadata:
            f:labels:
              .: {}
              f:app: {}
          f:spec:
            f:containers:
              k:{"name":"hello-app"}:
                .: {}
                f:image: {}
                f:imagePullPolicy: {}
                f:name: {}
                f:resources: {}
                f:terminationMessagePath: {}
                f:terminationMessagePolicy: {}
            f:dnsPolicy: {}
            f:restartPolicy: {}
            f:schedulerName: {}
            f:securityContext: {}
            f:terminationGracePeriodSeconds: {}
    manager: kubectl
    operation: Update
    time: "2020-06-24T15:01:54Z"
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:deployment.kubernetes.io/revision: {}
      f:status:
        f:availableReplicas: {}
        f:conditions:
          .: {}
          k:{"type":"Available"}:
            .: {}
            f:lastTransitionTime: {}
            f:lastUpdateTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"Progressing"}:
            .: {}
            f:lastTransitionTime: {}
            f:lastUpdateTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:observedGeneration: {}
        f:readyReplicas: {}
        f:replicas: {}
        f:updatedReplicas: {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-07-01T18:54:26Z"
  name: web
  namespace: default
  resourceVersion: "499938"
  selfLink: /apis/apps/v1/namespaces/default/deployments/web
  uid: 31358c78-9f52-4eec-bb33-fe03f9bee183
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: web
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: web
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:1.0
        imagePullPolicy: IfNotPresent
        name: hello-app
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-06-24T15:01:54Z"
    lastUpdateTime: "2020-06-24T15:02:01Z"
    message: ReplicaSet "web-6785d44d5" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-07-01T18:54:26Z"
    lastUpdateTime: "2020-07-01T18:54:26Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

aedorado · 2020-07-02T05:06:02Z

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2020-06-24T15:20:44Z"
  labels:
    app: web
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:app: {}
      f:spec:
        f:externalTrafficPolicy: {}
        f:ports:
          .: {}
          k:{"port":8080,"protocol":"TCP"}:
            .: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl
    operation: Update
    time: "2020-06-24T15:20:44Z"
  name: web
  namespace: default
  resourceVersion: "3421"
  selfLink: /api/v1/namespaces/default/services/web
  uid: 8484b1c9-b156-4677-aa4f-b3b7e0b64ffc
spec:
  clusterIP: 10.99.89.90
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 31469
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: web
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

aedorado · 2020-07-02T05:06:35Z

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
  creationTimestamp: "2020-06-24T15:22:36Z"
  generation: 1
  managedFields:
  - apiVersion: networking.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:nginx.ingress.kubernetes.io/rewrite-target: {}
      f:spec:
        f:rules: {}
    manager: kubectl
    operation: Update
    time: "2020-06-24T15:22:36Z"
  name: web-ingress
  namespace: default
  resourceVersion: "504590"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/web-ingress
  uid: 79444cf1-191a-4134-bff2-7bc3764fe64e
spec:
  rules:
  - host: hello-world.info
    http:
      paths:
      - backend:
          serviceName: web
          servicePort: 8080
        path: /
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

aedorado · 2020-07-02T05:09:20Z

All of the above are from minikube, but we are having the same problem on our GKE clusters as well.

Bo0km4n · 2020-07-03T02:28:14Z

@aedorado I confirmed some metrics you want to see in minikube.

I note steps taken below:

Set up ingress

minikube start
minikube addons enable ingress

deploy service and ingress

$ echo '---
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: hello
  name: hello
spec:
  containers:
  - image: gcr.io/google-samples/hello-app:1.0
    name: hello-app
    ports:
    - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-app
  name: hello-app
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    run: hello
  type: NodePort' | kubectl apply -f -

echo 'apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web-ingress-1
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - host: hello-world.info
    http:
      paths:
      - path: /
        backend:
          serviceName: hello-app
          servicePort: 8080' | kubectl apply -f -

And then, exec curl command for expose some metrics

curl http://`minikube ip` -H 'Host: hello-world.info'

To confirm nginx metrics, enter shell of ingress controller

kubectl get po -A
NAMESPACE     NAME                                        READY   STATUS      RESTARTS   AGE
default       hello                                       1/1     Running     0          11m
kube-system   coredns-66bff467f8-l6szq                    1/1     Running     0          17h
kube-system   coredns-66bff467f8-xvktx                    1/1     Running     0          17h
kube-system   etcd-minikube                               1/1     Running     0          17h
kube-system   ingress-nginx-admission-create-t2jdq        0/1     Completed   0          17h
kube-system   ingress-nginx-admission-patch-s6jkh         0/1     Completed   1          17h
kube-system   ingress-nginx-controller-7bb4c67d67-zn5m7   1/1     Running     0          17h
kube-system   kube-apiserver-minikube                     1/1     Running     0          17h
kube-system   kube-controller-manager-minikube            1/1     Running     0          17h
kube-system   kube-proxy-mvcrv                            1/1     Running     0          17h
kube-system   kube-scheduler-minikube                     1/1     Running     0          17h
kube-system   metrics-server-7bc6d75975-lnkd7             1/1     Running     0          17h
kube-system   storage-provisioner                         1/1     Running     0          17h

kubectl exec -it ingress-nginx-controller-7bb4c67d67-zn5m7 -n kube-system sh

In the ingress nginx controller, access to metrics path

/etc/nginx $ curl localhost:10254/metrics | grep nginx_ingress_controller_request
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP nginx_ingress_controller_request_duration_seconds The request processing time in milliseconds
# TYPE nginx_ingress_controller_request_duration_seconds histogram
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="0.005"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="0.01"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="0.025"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="0.05"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="0.1"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="0.25"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="0.5"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="1"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="2.5"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="5"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="10"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="+Inf"} 4
nginx_ingress_controller_request_duration_seconds_sum{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200"} 0.004
nginx_ingress_controller_request_duration_seconds_count{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200"} 4
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="0.005"} 0
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="0.01"} 0
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="0.025"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="0.05"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="0.1"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="0.25"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="0.5"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="defau100 55368    0 55368   ello-app",status="200",le="1"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="2.5"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="5"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="10"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="+Inf"} 1 0     0  2457k      0 --:--:-- --:--:-- --:--:-- 2457k

nginx_ingress_controller_request_duration_seconds_sum{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200"} 0.012
nginx_ingress_controller_request_duration_seconds_count{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200"} 1
# HELP nginx_ingress_controller_request_size The request length (including request line, header, and request body)
# TYPE nginx_ingress_controller_request_size histogram
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="10"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="20"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="30"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="40"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="50"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="60"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="70"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="80"} 4
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="90"} 4
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="100"} 4
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200",le="+Inf"} 4
nginx_ingress_controller_request_size_sum{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200"} 320
nginx_ingress_controller_request_size_count{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="GET",namespace="default",path="/",service="hello-app",status="200"} 4
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="10"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="20"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="30"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="40"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="50"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="60"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="70"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="80"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="90"} 1
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="100"} 1
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200",le="+Inf"} 1
nginx_ingress_controller_request_size_sum{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200"} 81
nginx_ingress_controller_request_size_count{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",host="hello-world.info",ingress="web-ingress-1",method="HEAD",namespace="default",path="/",service="hello-app",status="200"} 1
# HELP nginx_ingress_controller_requests The total number of client requests.
# TYPE nginx_ingress_controller_requests counter
nginx_ingress_controller_requests{controller_class="nginx",controller_namespace="kube-system",controller_pod="ingress-nginx-controller-7bb4c67d67-zn5m7",ingress="web-ingress-1",namespace="default",service="hello-app",status="200"} 5

kundan2707 · 2020-07-03T04:09:34Z

/unassign

kingli-crypto · 2020-07-27T10:02:43Z

If you didn't specify ingress host, ingress status is as below:
$ kubectl get ingress
NAME                     CLASS     HOSTS              ADDRESS        PORTS   AGE
example-ingress   <none>   *                         10.8.201.111   80      5m36s
When HOSTS parameter is *,
execute this if statement:

ingress-nginx/internal/ingress/metric/collectors/socket.go

Lines 233 to 235 in d825859

if !sc.hosts.Has(stats.Host) {

klog.V(3).Infof("skiping metric for host %v that is not being served", stats.Host)

continue

So you can't see some metrics such as nginx_ingress_controller_request

@Bo0km4n Why are metrics filtered out in this case. I have an ingress where host is not specified (empty key) and I am not getting these metrics.

jaimehrubiks · 2020-07-28T05:06:46Z

Same issue here, we only use one host and we don't specify its name.

Bo0km4n · 2020-07-28T05:47:04Z

If you don't specify the host name, nginx convert destination host parameter to Pod IP.
This means that the number of time series data will not be duplicated by the host, and the number of time series data will explode.

Maybe, I guess above the code is filtering metrics by host parameter for those reason.
If you want get those metrics, you should specify host parameter. Even if there is only one host in your cluster.

kingli-crypto · 2020-07-29T04:15:37Z

@Bo0km4n This PR has made per host metrics optional #3594. So I think even if we remove the restriction for mis-matched hostname, the number of series will not explode.

Please correct me if I am wrong, I am new to all this.

Bo0km4n · 2020-07-30T17:34:53Z

@kingli-crypto Thanks for giving information. In case of disabling metrics-per-host option, you may be able to show some metrics removed by above codes without indicating host parameter.

Please try it.

kingli-crypto · 2020-07-31T08:01:34Z

@Bo0km4n No that's not the case. I disabled metrics-per-host and the metrics does not show up.

kingli-crypto · 2020-07-31T12:44:05Z

Turns out there is a existing PR for this issue #4139

fejta-bot · 2020-12-02T12:33:19Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2021-01-01T13:17:47Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2021-01-31T14:01:27Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2021-01-31T14:01:37Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

headyj · 2023-06-02T07:24:11Z

@kingli-crypto did you manage to solve that issue? I have the exact same issue and disabling metrics-per-host doesn't solve the issue

iamkinchit · 2024-12-18T07:57:33Z

@thamaraiselvam are you able to fix it.
/reopen

k8s-ci-robot · 2024-12-18T07:57:38Z

@iamkinchit: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

@thamaraiselvam are you able to fix it.
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

thamaraiselvam added the kind/bug Categorizes issue or PR as related to a bug. label Jun 23, 2020

k8s-ci-robot assigned kundan2707 Jun 26, 2020

k8s-ci-robot assigned Bo0km4n Jun 26, 2020

k8s-ci-robot unassigned kundan2707 Jul 3, 2020

kingli-crypto mentioned this issue Jul 31, 2020

Skip host metrics only if --metrics-per-host is enabled #5961

Closed

8 tasks

Bo0km4n removed their assignment Sep 3, 2020

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 2, 2020

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 1, 2021

k8s-ci-robot closed this as completed Jan 31, 2021

vast0906 mentioned this issue Nov 17, 2021

metrics not some metrics #7933

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Few metrics are missing in Ingress nginx #5755

Few metrics are missing in Ingress nginx #5755

thamaraiselvam commented Jun 23, 2020

kundan2707 commented Jun 26, 2020

Bo0km4n commented Jun 26, 2020 •

edited

Loading

Bo0km4n commented Jun 26, 2020

aedorado commented Jun 26, 2020

Bo0km4n commented Jun 26, 2020 •

edited

Loading

Bo0km4n commented Jun 26, 2020

aedorado commented Jun 28, 2020 •

edited

Loading

Bo0km4n commented Jun 29, 2020

aedorado commented Jun 29, 2020

Bo0km4n commented Jun 29, 2020

aedorado commented Jul 1, 2020

Bo0km4n commented Jul 1, 2020

aedorado commented Jul 2, 2020

aedorado commented Jul 2, 2020

aedorado commented Jul 2, 2020

aedorado commented Jul 2, 2020

Bo0km4n commented Jul 3, 2020 •

edited

Loading

kundan2707 commented Jul 3, 2020

kingli-crypto commented Jul 27, 2020

jaimehrubiks commented Jul 28, 2020

Bo0km4n commented Jul 28, 2020

kingli-crypto commented Jul 29, 2020

Bo0km4n commented Jul 30, 2020 •

edited

Loading

kingli-crypto commented Jul 31, 2020

kingli-crypto commented Jul 31, 2020

fejta-bot commented Dec 2, 2020

fejta-bot commented Jan 1, 2021

fejta-bot commented Jan 31, 2021

k8s-ci-robot commented Jan 31, 2021

headyj commented Jun 2, 2023

iamkinchit commented Dec 18, 2024

k8s-ci-robot commented Dec 18, 2024

Few metrics are missing in Ingress nginx #5755

Few metrics are missing in Ingress nginx #5755

Comments

thamaraiselvam commented Jun 23, 2020

kundan2707 commented Jun 26, 2020

Bo0km4n commented Jun 26, 2020 • edited Loading

Bo0km4n commented Jun 26, 2020

aedorado commented Jun 26, 2020

Bo0km4n commented Jun 26, 2020 • edited Loading

Bo0km4n commented Jun 26, 2020

aedorado commented Jun 28, 2020 • edited Loading

Bo0km4n commented Jun 29, 2020

aedorado commented Jun 29, 2020

Bo0km4n commented Jun 29, 2020

aedorado commented Jul 1, 2020

Bo0km4n commented Jul 1, 2020

aedorado commented Jul 2, 2020

aedorado commented Jul 2, 2020

aedorado commented Jul 2, 2020

aedorado commented Jul 2, 2020

Bo0km4n commented Jul 3, 2020 • edited Loading

Set up ingress

kundan2707 commented Jul 3, 2020

kingli-crypto commented Jul 27, 2020

jaimehrubiks commented Jul 28, 2020

Bo0km4n commented Jul 28, 2020

kingli-crypto commented Jul 29, 2020

Bo0km4n commented Jul 30, 2020 • edited Loading

kingli-crypto commented Jul 31, 2020

kingli-crypto commented Jul 31, 2020

fejta-bot commented Dec 2, 2020

fejta-bot commented Jan 1, 2021

fejta-bot commented Jan 31, 2021

k8s-ci-robot commented Jan 31, 2021

headyj commented Jun 2, 2023

iamkinchit commented Dec 18, 2024

k8s-ci-robot commented Dec 18, 2024

Bo0km4n commented Jun 26, 2020 •

edited

Loading

Bo0km4n commented Jun 26, 2020 •

edited

Loading

aedorado commented Jun 28, 2020 •

edited

Loading

Bo0km4n commented Jul 3, 2020 •

edited

Loading

Bo0km4n commented Jul 30, 2020 •

edited

Loading