ingress-controller metric: nginx_ingress_controller_requests is always 0

[PoojaGidaveer]

We are running Performance Tests and we are forwarding the traffic via ingress resource but all of the nginx-ingress-controller metrics value are always 0.

The configuration on which nginx-ingress-controller binary is running :

 - args:
    - /nginx-ingress-controller
    - --default-backend-service=kube-system/nginx-ingress-controller-default-backend
    - --election-id=ingress-controller-leader
    - --ingress-class=nginx
    - --configmap=kube-system/nginx-ingress-controller
    - --enable-ssl-passthrough

Ingress-resource annotations:

metadata:
  annotations:
    controller.metrics.enabled: "true"
    ingress.kubernetes.io/proxy-connect-timeout: "10"
    ingress.kubernetes.io/proxy-read-timeout: "120"
    ingress.kubernetes.io/proxy-send-timeout: "120"
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: connectivity-proxy
    meta.helm.sh/release-namespace: ngptt-184409
    metrics.enabled: "true"
    nginx.ingress.kubernetes.io/auth-tls-secret: <secret>
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Forwarded-Client-Cert $ssl_client_escaped_cert;
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "10"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-ssl-name: <domain>
    nginx.ingress.kubernetes.io/proxy-ssl-secret: <secret>
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: "1"
    nginx.org/websocket-services: connectivity-proxy

We followed this link and added --metrics-per-host=false still we didnot see any metrics
Prometheus nginx-ingress-controller target is reachable.

NGINX Ingress controller version:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:19:55Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}

Environment: GCP

/kind bug

[longwuyuan]

We are working on a fix for the docs
Have you tried this https://dev.to/xaviergeerinck/monitoring-the-kubernetes-nginx-ingress-controller-with-prometheus-and-grafana-35gi

/assign

[longwuyuan]

/area docs

[longwuyuan]

/remove-kind bug

[PavanVasisht29]

@longwuyuan we had already exposed the 10254 port and scraping the nginx metrics, still few of them weren't being captured, as mentioned above.

[longwuyuan]

are you talking about the same cluster as the user who created this issue ?

[PavanVasisht29]

Yuppee.

[longwuyuan]

show

date ; time ; hostname ; kubectl -n <controllernamespace> describe deploy ingcontroller0-ingress-nginx-controller

[PavanVasisht29]

Name:                   nginx-ingress-ingress-nginx-controller
Namespace:              kube-system
CreationTimestamp:      Fri, 25 Jun 2021 16:31:35 +0530
Labels:                 app.kubernetes.io/component=controller
                        app.kubernetes.io/instance=nginx-ingress
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=ingress-nginx
                        app.kubernetes.io/version=0.47.0
                        helm.sh/chart=ingress-nginx-3.34.0
Annotations:            deployment.kubernetes.io/revision: 4
                        meta.helm.sh/release-name: nginx-ingress
                        meta.helm.sh/release-namespace: kube-system
Selector:               app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/component=controller
                    app.kubernetes.io/instance=nginx-ingress
                    app.kubernetes.io/name=ingress-nginx
  Annotations:      prometheus.io/port: 10254
                    prometheus.io/scrape: true
  Service Account:  nginx-ingress-ingress-nginx
  Containers:
   controller:
    Image:       k8s.gcr.io/ingress-nginx/controller:v0.47.0@sha256:a1e4efc107be0bb78f32eaec37bef17d7a0c81bec8066cdf2572508d21351d0b
    Ports:       80/TCP, 443/TCP, 8443/TCP, 10254/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
      --election-id=ingress-controller-leader
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    Requests:
      cpu:      1
      memory:   900Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
  Volumes:
   webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-ingress-nginx-admission
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-ingress-ingress-nginx-controller-b5f5897fc (3/3 replicas created)
Events:          <none>

[longwuyuan]

ok .. add other data related like prometheus details etc. describe what you expect to see and what is missing

[PoojaGidaveer]

@longwuyuan we are running Performance Test. The flow is like below-
Load_Generator -> Connectivity-Proxy_Pod -> connectivity-proxy_ingress -> backend

So during the load we could see metrics of backend_ingress, but some how we are not seeing metrics of connectivity-proxy_ingress

for example- when queried round(sum(irate(nginx_ingress_controller_requests[2m])) by (ingress), 0.001) we can see backend_ingress metrics but connectivity-proxy_ingress metric is always 0

above we shared the configurations used for connectivity-proxy_ingress.
please let us know how do we solve it.

[longwuyuan]

Is that a custom metric or is that something you are computing from available metrics ?

I see this on that query. What is the value you see

[PoojaGidaveer]

Please refer to below screenshot

[longwuyuan]

Change date range because I get non zero valuesThanks,  -- ; Long 15 Jul 2021, 14:40 by ***@***.***:

…

Please refer to below screenshot <https://user-images.githubusercontent.com/33653955/125762049-c6476f43-e79f-486c-bd73-f54d7fe3006e.png> — You are receiving this because you were mentioned. Reply to this email directly, > view it on GitHub <#7285 (comment)>> , or > unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWW3BJLTVXPT76C75QTTX2QX3ANCNFSM47JQQDHA>> .

[PoojaGidaveer]

As we said above we are running load test which is passing via connectivity_proxy ingress and reaching to nodecap_ingress. But we are only seeing metrics of nodecap ingress.. That's the reason we have raised this issue to understand if there is any missing configurations

[longwuyuan]

Ah apologies, I don't even know what those 2 types of ingresses are. Can you point me to some documentation that describes what those 2 types of ingresses are ?

Also, would you care to share screen on zoom and show what you are talking about ?

[PoojaGidaveer]

Kindly share your email id so that we can send MS Teams blocker and have a discussion over the call.. please let us know your convenient time

[longwuyuan]

I don't have msteams. Zoom works for me. Timing is GMT +5.30 12Noon to 6PM for 30 minute or more as per needThanks,  -- ; Long 15 Jul 2021, 17:36 by ***@***.***:

…

Kindly share your email id so that we can send MS Teams blocker and have a discussion over the call.. please let us know your convenient time — You are receiving this because you were mentioned. Reply to this email directly, > view it on GitHub <#7285 (comment)>> , or > unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWTZWBZZJM3ZQW7S2XDTX3FKZANCNFSM47JQQDHA>> .