prometheus-adapter not working in GKE

[kehao95]

Summary

I've been wanting to setup prometheus-adapter for a while and couldn't get it right. Getting following errors:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
Error from server (ServiceUnavailable): the server is currently unable to handle the request

prometheus-adapter seem to be running with little helpful logs
apiServices checked failed with message:

failing or missing response from https://10.6.7.183:6443/apis/custom.metrics.k8s.io/v1beta1: Get https://10.6.7.183:6443/apis/custom.metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Didn't find much useful clues. So wish I can get some help from the community.

My environment

• So I'm using the HEAD of the kube-prometheus project,
• with zero customization of prometheus-adapter related configs.
• Using GKE cluster v1.18.9-gke.801
• directxman12/k8s-prometheus-adapter:v0.8.0

Details:

Saw apiServices registrations for metrics.k8s.io and custom.metrics.k8s.io

So after I generated the manifest I see following related yamls:

* prometheus-adapter-apiService.yaml
prometheus-adapter-clusterRole.yaml
prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
prometheus-adapter-clusterRoleBinding.yaml
prometheus-adapter-clusterRoleBindingDelegator.yaml
prometheus-adapter-clusterRoleServerResources.yaml
prometheus-adapter-configMap.yaml
* prometheus-adapter-customMetricsApiService.yaml
* prometheus-adapter-customMetricsApiServiceV1Beta2.yaml
prometheus-adapter-customMetricsClusterRoleBindingHPA.yaml
prometheus-adapter-customMetricsClusterRoleBindingServerResources.yaml
prometheus-adapter-customMetricsClusterRoleServerResources.yaml
prometheus-adapter-deployment.yaml
prometheus-adapter-roleBindingAuthReader.yaml
prometheus-adapter-service.yaml
prometheus-adapter-serviceAccount.yaml
prometheus-adapter-serviceMonitor.yaml

Among them I saw 3 apiServices registrations for v1beta1.metrics.k8s.io v1beta1.custom.metrics.k8s.io & v1beta2.custom.metrics.k8s.io

But apiservices failed to pass check
After apply them I saw:

$ kubectl get apiservices
NAME                                   SERVICE                         AVAILABLE                      AGE
v1beta1.custom.metrics.k8s.io          monitoring/prometheus-adapter   False (FailedDiscoveryCheck)   25d
v1beta1.metrics.k8s.io                 monitoring/prometheus-adapter   False (FailedDiscoveryCheck)   34d
v1beta2.custom.metrics.k8s.io          monitoring/prometheus-adapter   False (FailedDiscoveryCheck)   34d

3 apis are using the monitoring/prometheus-adapter but FailedDiscoveryCheck
(v1beta1.metrics.k8s.io changed back to kube-system/metrics-server a while later)

Check details show failing or missing response

$ kubectl describe APIService v1beta1.custom.metrics.k8s.io
Name:         v1beta1.custom.metrics.k8s.io
...
Spec:
  Group:                     custom.metrics.k8s.io
...
Status:
  Conditions:
    Message:               
      failing or missing response from https://10.6.7.183:6443/apis/custom.metrics.k8s.io/v1beta1: 
      Get https://10.6.7.183:6443/apis/custom.metrics.k8s.io/v1beta1: net/http: 
      request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
...

However the prometheus-adapter is not complete dead
I tried HTTP get the https://10.6.7.183:6443/apis/custom.metrics.k8s.io/v1beta1 got 403 error (don't know how to mock an valid API request yet)

And the prometheus-adapter logs

I1102 00:30:13.273064       1 adapter.go:98] successfully using in-cluster auth
I1102 00:30:14.474853       1 request.go:621] Throttling request took 1.173651213s, request: GET:https://10.7.0.1:443/apis/apiextensions.k8s.io/v1?timeout=32s
I1102 00:30:19.041565       1 serving.go:307] Generated self-signed cert (/var/run/serving-cert/apiserver.crt, /var/run/serving-cert/apiserver.key)
I1102 00:30:19.880928       1 secure_serving.go:178] Serving securely on [::]:6443
I1102 00:30:19.882292       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1102 00:30:19.882541       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1102 00:30:19.882614       1 dynamic_serving_content.go:130] Starting serving-cert::/var/run/serving-cert/apiserver.crt::/var/run/serving-cert/apiserver.key
I1102 00:30:19.882693       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I1102 00:30:19.886125       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1102 00:30:19.886141       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1102 00:30:19.991345       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1102 00:30:19.991360       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file

[brancz]

cc @s-urbaniak

[kehao95]

Answer my own question here:

It's stupid simple for the root cause. But was hard to find it until i tired to install another API server and encountered the similar issue: Everything looks good but apiservices failed to pass check.

It turns out that it was just a firewall issue. And it has been mentioned in GKE Doc.
https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules

By default, firewall rules restrict your cluster control plane to only initiate TCP connections to your nodes on ports 443 (HTTPS) and 10250 (kubelet).
For some Kubernetes features, you might need to add firewall rules to allow access on additional ports.
Kubernetes features that require additional firewall rules include:
-Admission webhooks
-Aggregated API servers
-Webhook conversion
-Dynamic audit configuration

So after I add 6443(prometheus-adapter) port to the firewall rules, everything worked fine.