Enhancement: Enable DNS Metrics

[malachma]

This is an Enhancement/Feature Request

Currently kube-dns does not expose the dns-metrics, either via the Standard port 10055 or any other port.
Due to the missing dns-metrics port external monitoring Solutions like datadog are not able to be used, as they expect the metrics port open and available.

[slack]

This is a good call, I've created a ticket in our backlog to track.

[ghost]

+1

Having these metrics would be quite useful.

[mfpierre]

+1 on this as well.

If would be great if this was on the standard 10055 port, if not could you please notify us (Datadog)? 🙇

[jalberto]

any progress on this?

[NicolaiPetri]

Hi.
I got this partially working on my aks cluster by doing the following:

kubectl edit deployment -n kube-system kube-dns-v20

and then inserted the following setting

        env:
        - name: PROMETHEUS_PORT
          value: "10055"

This makes polling of port 10055 work.

[StianOvrevage]

@NicolaiPetri s workaround works here. I also added the sidecar container that exposes dnsmasq metrics.

Here is how to apply both workarounds and this makes default prometheus-operator setup to automatically start scraping both metrics. Save the following file as kube-dns-metrics-patch.yaml and run kubectl patch deployment -n kube-system kube-dns-v20 --patch "$(cat kube-dns-metrics-patch.yaml)" to apply the changes.

spec:
  template:
    spec:
      containers:
      - name: kubedns
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
      - name: sidecar
        image: k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.10
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --v=2
        - --logtostderr
        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local
        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 20Mi
            cpu: 10m

[larsduelfer]

Would be awesome if this is provided out-of-the-box in the future.

[jnoller]

This was rolled out in the January 2019 releases. Customers who upgrade their cluster to a new version (major/minor/patch) will see that kube-dns/coreDNS both have the autoscaler enabled and DNS metrics are now exposed as well.

[palmerabollo]

@jnoller I think there is a missing "prometheus.io/scrape" annotation in the coredns deployment (at least with k8s 1.16.x). Should I open a separate issue?