x509: certificate signed by unknown authority #146

max201920 · 2018-10-01T06:39:30Z

In Metric server pod getting following error.

[restful] 2018/10/01 06:35:24 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1001 06:35:24.584055 1 serve.go:96] Serving securely on [::]:443
E1001 06:36:24.615237 1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:kube: unable to fetch metrics from Kubelet kube (kube): Get https://kube:10250/stats/summary/: x509: certificate signed by unknown authority

E1001 06:37:24.543263 1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:kube: unable to fetch metrics from Kubelet kube (kube): Get https://kube:10250/stats/summary/: x509: certificate signed by unknown authority

DirectXMan12 · 2018-10-02T15:32:13Z

It looks like you're trying to validate the kubelet serving certs without having them be signed by the main Kubernetes CA. If you're on metrics-server from before v0.3.0, this can be confirmed with the insecure=true source option (otherwise, you can pass a similar command line flag to v0.3.0+),

max201920 · 2018-10-03T06:29:16Z

I deployed new kubernetes server 1.12 using kubeadm and then using kubectl create -f deploy/1.8+/ I created metric server , no custom changes were done in either kubelet or metric server of kubernetes API.

DirectXMan12 · 2018-10-03T15:18:00Z

Please try using the --kubelet-insecure-tls flag for metrics-server, and see if that helps. Some of the installation tools don't set up certificates properly yet.

albrr · 2018-10-09T16:23:23Z

How and where do I use those flags? I have the same problem but can't find any information on how to use this

k0nstantinv · 2018-10-17T04:14:15Z

@albrr your metrics-server Deployment should be modified like this:

      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
        command:
        - /metrics-server
        - --kubelet-insecure-tls

albrr · 2018-10-17T12:15:50Z

thanks @k0nstantinv :)

towolf · 2018-10-28T15:49:43Z

@DirectXMan12 following recommendations I am employing a Kubelet CA separate from the main CA.

But now I get this all the time:

E1028 15:43:03.202456       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:kvm336: unable to fetch metrics from Kubelet kvm336 (10.109.0.3): Get https://10.109.0.3:10250/stats/summary/: x509: certificate signed by unknown authority,  ... ... ...]

The Apiserver itself can be configured to verify using the Kubelet CA.

--kubelet-certificate-authority=/etc/kubernetes/pki/kubelet-ca.pem

I suppose for metrics-server this would be possible too?

rayanebel · 2018-11-05T18:39:39Z

+1

DirectXMan12 · 2018-11-07T23:38:23Z

Yes, we need a feature to support a separate kubelet CA.

@brancz this is becoming a decently common request

/kind feature
/priority important-soon

rayanebel · 2018-11-14T21:31:24Z

While waiting this new feature. It’s possible to mount the certificate inside the pod using a volume or something else ?

brancz · 2018-11-15T10:06:54Z

@rayanebel I don't think so as the same cert that is currently used by default for the kubelet is also used (and there it is valid) for the apiserver.

If anyone wants to give this a go, please feel free to open a PR 🙂 .

metalmatze · 2018-11-15T14:27:27Z

Just for completeness: We also need the same thing in the Prometheus Adapter.

brancz · 2018-11-15T14:40:18Z

@metalmatze feel free to open an issue there.

Aiqbal1234 · 2018-11-27T08:48:10Z

Has any progress been made on this?
by setting the --kubelet-insecure-tls flag i can get autoscaling to work. But i still get the logs related to CA and not all metrics are available.

Do/should i need to regenerate the kubelet certificates (we use AWS and Kubeadm)? I cant see why i would need to do that, our pods are being deployed to the nodes... So the kubelet certificates should exist i believe.

btw i already have apiserver-kubelet-client.key and apiserver-kubelet-client.crt on master

DirectXMan12 · 2018-11-28T18:15:10Z

there's hasn't been any PR for it yet, IIRC. It's on the list of things to do soon, though.

desmatz · 2019-01-31T07:08:27Z

I did the following (kubeadm ha cluster):
in accordance with:
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

Create a Certificate Signing Request

cat <<EOF | cfssl genkey - | cfssljson -bare kubelet-server
{
  "hosts": [
    "node-name-1",
    "node-name-2",
    ""...",
    "node-ip-1",
    "node-ip-2",
    "..."
  ],
  "CN": "kubelet-server",
  "key": {
    "algo": "ecdsa",
    "size": 256
  }
}
EOF

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: kubelet-server
spec:
  groups:
  - system:nodes
  - system:authenticated
  request: $(cat kubelet-server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

kubectl describe csr kubelet-server
kubectl certificate approve kubelet-server
kubectl get csr kubelet-server -o jsonpath='{.status.certificate}' | base64 --decode > kubelet-server.pem
Copy to all nodes in /var/lib/kubelet/pki/
Add to kubelet config.yaml tls cert & key
see: https://godoc.org/k8s.io/kubernetes/pkg/kubelet/apis/config#KubeletConfiguration
echo "tlsPrivateKeyFile: /var/lib/kubelet/pki/kubelet-server-key.pem" >> /var/lib/kubelet/config.yaml
echo "tlsCertFile: /var/lib/kubelet/pki/kubelet-server.pem" >> /var/lib/kubelet/config.yaml
restart kubelet

systemctl daemon-reload && \
systemctl restart kubelet && \
systemctl status kubelet

view metrics-server logs. If ok
kubectl get hpa

sarneaud · 2019-02-03T22:15:29Z

Another kubeadm user here. I solved the problem by doing the following before running kubeadm init:

kubeadm --config config.yaml init phase certs all
Creating an X509 key in /var/lib/kubelet/pki/kubelet.key
Creating a cert signed by /etc/kubernetes/pki/ca.key, putting it in /var/lib/kubelet/pki/kubelet.crt

It's basically the same as @desmatz solution, but using the CA key directly instead of using the cert signing API.

ghost · 2019-02-14T10:54:18Z

@albrr your metrics-server Deployment should be modified like this:

      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
        command:
        - /metrics-server
        - --kubelet-insecure-tls

thank you so much!
Костя ты красавчик ;))

ioanc · 2019-03-04T16:34:21Z

security is a hot topic these days.
offering a possible solution by disabling security, using --kubelet-insecure-tls is not the right way to move forward.

@desmatz and @sarneaud solutions are the proper way to fix the issue.
Thank you guys !!!

paphillon · 2019-03-08T05:12:56Z

@ioanc From what I understood, these solutions intend to be planned in advance for use of metrics server, existing clusters with a separate CA for kubelet and or not using kubeadm do need support to specify kubelet CA public key as suggested by @DirectXMan12

charlesguoooo · 2019-03-13T13:17:10Z

when you use kubeadm deploy metrics-server, you can add serverTLSBootstrap: true to /var/lib/kubelet/config.yml ,it works for me. remember restart kubelet!

Baykonur · 2019-03-13T15:06:35Z

@DirectXMan12

It looks like you're trying to validate the kubelet serving certs without having them be signed by the main Kubernetes CA. If you're on metrics-server from before v0.3.0, this can be confirmed with the insecure=true source option (otherwise, you can pass a similar command line flag to v0.3.0+),

I have the same error:

manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cored01: unable to fetch metrics from Kubelet cored01 (cored01): Get https://cored01:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:cored02: unable to fetch metrics from Kubelet cored02 (cored02): Get https://cored02:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:cored03: unable to fetch metrics from Kubelet cored03 (cored03): Get https://cored03:10250/stats/summary/: x509: certificate signed by unknown authority]

And the kubelet client certificate is signed by the main CA (kube-apiserver, or are we talking about another one?)

cored02 kubernetes # openssl verify -CAfile /etc/kubernetes/pki/ca.crt /var/lib/kubelet/pki/kubelet-client-current.pem
/var/lib/kubelet/pki/kubelet-client-current.pem: OK

If I pass the --kubelet-insecure-tls, all works fine.

I used kubeadm to bootstrap the cluster (single master, 2 workers), no custom setup whatsoever, kubernetes version is 1.13.4.

on the master:

cored01 pki # ps -ef | grep kube-apiserver
root      1329  1310  2 Mar08 ?        02:33:09 kube-apiserver --authorization-mode=Node,RBAC --advertise-address=10.14.10.116 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=172.17.224.0/19 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

cored01 pki # openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Mar  5 06:56:50 2019 GMT
            Not After : Mar  2 06:56:50 2029 GMT
        Subject: CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d9:8f:9d:04:de:50:ee:73:24:28:50:1d:98:e7:
                    72:e0:5f:78:44:be:47:b3:f8:e6:fc:8e:08:f2:fd:
                    da:33:a3:2e:c2:7b:d1:1e:00:db:13:e6:24:68:3b:
                    be:8d:ea:31:ff:37:ee:59:68:03:6f:08:73:94:cc:
                    60:69:e1:0b:67:08:25:2b:f3:b9:f4:6e:de:2d:86:
                    27:3a:9c:7d:06:84:af:f1:09:e5:5b:fd:7f:4f:4d:
                    50:c3:6b:d5:91:48:f5:ad:59:56:ff:d3:0c:3f:c0:
                    fa:ef:29:5f:da:21:8d:5b:44:98:45:e6:7d:78:0b:
                    62:ba:ed:1c:0b:71:cd:86:e4:20:e7:b3:ab:72:a9:
                    64:93:c4:0f:32:aa:01:61:70:f7:8e:4f:c1:ce:63:
                    1d:a4:2c:36:6b:af:16:6e:16:f3:8f:43:55:80:6c:
                    c3:6e:d0:94:65:92:bd:f1:29:e3:1a:be:60:42:64:
                    a9:84:45:b7:6f:00:b6:8c:fb:ae:7a:90:28:08:d4:
                    6a:58:fb:4e:63:79:0f:01:a1:79:61:ea:04:8d:5c:
                    f1:59:fa:a6:6e:f4:93:06:9f:3c:0f:5c:26:53:f5:
                    7e:f6:9a:2b:f7:e9:d8:43:66:b1:09:0f:5c:62:c0:
                    51:0b:7c:cd:6b:3b:07:d1:0e:fb:4c:05:55:ab:8f:
                    db:7f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         5b:ef:38:c4:5a:c8:95:a3:71:e4:3b:30:e4:33:79:42:6d:fc:
         02:f6:dd:d5:0e:e2:d2:83:aa:6b:3a:3b:74:04:c4:06:61:41:
         94:ad:4f:88:03:0a:09:b8:59:ad:30:b2:38:0b:4b:11:00:64:
         62:17:50:2f:ce:05:49:c2:99:3a:fb:11:d3:79:9b:2b:35:4f:
         5d:67:73:2d:a1:f7:64:47:a5:35:1c:ea:4f:cf:d0:be:79:9b:
         9c:a4:32:fd:8f:70:25:98:a8:b9:f0:d2:f7:7e:f9:2b:06:b3:
         6f:ab:ab:f7:f3:63:12:eb:2f:c9:13:b0:f1:b5:4f:aa:56:9a:
         3a:3f:2d:51:4a:68:0f:39:bc:2a:ee:91:a0:7e:ec:ea:cd:b9:
         81:d7:4c:90:af:48:0a:ec:e7:84:8a:3d:84:71:31:40:98:f1:
         4a:71:bc:29:0f:2c:58:da:c6:bd:4e:c2:fb:9a:bc:41:3a:0a:
         55:df:13:98:27:52:e7:7f:5b:5f:a2:12:51:fc:08:6c:64:54:
         13:6e:b4:7b:c6:d8:1c:aa:00:a8:55:5b:da:67:e5:8e:d9:a4:
         79:6c:e9:35:dc:63:cf:25:48:47:2a:ee:ac:fe:5e:58:2b:89:
         ba:0a:3f:78:3e:a8:e9:7a:9b:d4:5c:1b:af:a3:29:d9:c1:d9:
         49:b2:d5:ee

on the worker:

cored02 kubernetes # openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Mar  5 06:56:50 2019 GMT
            Not After : Mar  2 06:56:50 2029 GMT
        Subject: CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d9:8f:9d:04:de:50:ee:73:24:28:50:1d:98:e7:
                    72:e0:5f:78:44:be:47:b3:f8:e6:fc:8e:08:f2:fd:
                    da:33:a3:2e:c2:7b:d1:1e:00:db:13:e6:24:68:3b:
                    be:8d:ea:31:ff:37:ee:59:68:03:6f:08:73:94:cc:
                    60:69:e1:0b:67:08:25:2b:f3:b9:f4:6e:de:2d:86:
                    27:3a:9c:7d:06:84:af:f1:09:e5:5b:fd:7f:4f:4d:
                    50:c3:6b:d5:91:48:f5:ad:59:56:ff:d3:0c:3f:c0:
                    fa:ef:29:5f:da:21:8d:5b:44:98:45:e6:7d:78:0b:
                    62:ba:ed:1c:0b:71:cd:86:e4:20:e7:b3:ab:72:a9:
                    64:93:c4:0f:32:aa:01:61:70:f7:8e:4f:c1:ce:63:
                    1d:a4:2c:36:6b:af:16:6e:16:f3:8f:43:55:80:6c:
                    c3:6e:d0:94:65:92:bd:f1:29:e3:1a:be:60:42:64:
                    a9:84:45:b7:6f:00:b6:8c:fb:ae:7a:90:28:08:d4:
                    6a:58:fb:4e:63:79:0f:01:a1:79:61:ea:04:8d:5c:
                    f1:59:fa:a6:6e:f4:93:06:9f:3c:0f:5c:26:53:f5:
                    7e:f6:9a:2b:f7:e9:d8:43:66:b1:09:0f:5c:62:c0:
                    51:0b:7c:cd:6b:3b:07:d1:0e:fb:4c:05:55:ab:8f:
                    db:7f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         5b:ef:38:c4:5a:c8:95:a3:71:e4:3b:30:e4:33:79:42:6d:fc:
         02:f6:dd:d5:0e:e2:d2:83:aa:6b:3a:3b:74:04:c4:06:61:41:
         94:ad:4f:88:03:0a:09:b8:59:ad:30:b2:38:0b:4b:11:00:64:
         62:17:50:2f:ce:05:49:c2:99:3a:fb:11:d3:79:9b:2b:35:4f:
         5d:67:73:2d:a1:f7:64:47:a5:35:1c:ea:4f:cf:d0:be:79:9b:
         9c:a4:32:fd:8f:70:25:98:a8:b9:f0:d2:f7:7e:f9:2b:06:b3:
         6f:ab:ab:f7:f3:63:12:eb:2f:c9:13:b0:f1:b5:4f:aa:56:9a:
         3a:3f:2d:51:4a:68:0f:39:bc:2a:ee:91:a0:7e:ec:ea:cd:b9:
         81:d7:4c:90:af:48:0a:ec:e7:84:8a:3d:84:71:31:40:98:f1:
         4a:71:bc:29:0f:2c:58:da:c6:bd:4e:c2:fb:9a:bc:41:3a:0a:
         55:df:13:98:27:52:e7:7f:5b:5f:a2:12:51:fc:08:6c:64:54:
         13:6e:b4:7b:c6:d8:1c:aa:00:a8:55:5b:da:67:e5:8e:d9:a4:
         79:6c:e9:35:dc:63:cf:25:48:47:2a:ee:ac:fe:5e:58:2b:89:
         ba:0a:3f:78:3e:a8:e9:7a:9b:d4:5c:1b:af:a3:29:d9:c1:d9:
         49:b2:d5:ee

metrics-server deployment.yaml:

...
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
        imagePullPolicy: Always
        command:
        - /metrics-server
        - --logtostderr
        - --kubelet-preferred-address-types=Hostname
...

If you require I can increase the log level.

I would appreciate if you can take your time to explain what the issue here is and how I can solve it.

sarneaud · 2019-03-14T00:16:36Z

@charlesguoooo

when you use kubeadm deploy metrics-server, you can add serverTLSBootstrap: true to /var/lib/kubelet/config.yml ,it works for me. remember restart kubelet!

That works for me too.

I added this to my kubeadm.conf:

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
serverTLSBootstrap: true

After starting the master or joining any nodes, the CSRs need to be approved. kubectl get csr gets a list. The pending ones for the nodes need to be approved with kubectl certificate approve $CSR_NAME. After a few minutes, certs are all good.

Baykonur · 2019-03-14T09:01:42Z

@sarneaud , thanks a lot. Your last comment saved me. @DirectXMan12 , you can ignore my comment.

What missing was kubelet-server-[datetime].pem under /var/lib/kubelet/pki in the worker node (also in master obviously), apparently metrics-server needs that certificate and it has to be signed by the /etc/kubernetes/pki/ca.crt which is in the master node, if you want to make things secure by switching off --kubelet-insecure-tls.

What I do not get is why metrics-server cannot use /var/lib/kubelet/pki/kubelet-client-[datetime].pem which is also signed by the same ca and it has the same cn as the kubelet-server.pem, except this one is used by kubelet to talk to kube-apiserver.

Baykonur · 2019-03-14T10:41:28Z

To answer my own question

What I do not get is why metrics-server cannot use /var/lib/kubelet/pki/kubelet-client-[datetime].pem which is also signed by the same ca and it has the same cn as the kubelet-server.pem, except this one is used by kubelet to talk to kube-apiserver.

The answer is here:

The client certificate provided by TLS bootstrapping is signed, by default, for client auth only, and thus cannot be used as serving certificates, or server auth.

However even though RotateKubeletServerCertificate is enabled by default, those CSRs will need to be approved (as described in this comment) once the existing server certificate expires --validity is 1 year.

See: kubernetes-sigs/metrics-server#146 Signed-off-by: Nicolas Lamirault <[email protected]>

fejta-bot · 2019-06-12T10:50:03Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2019-07-12T11:38:51Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2019-08-11T12:22:17Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2019-08-11T12:22:24Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

secret104278 · 2019-08-25T16:43:55Z

I found this issue is probably caused by kubernetes/kubeadm#1223

My solution is using api proxy by changing the lines in https://github.com/kubernetes-incubator/metrics-server/blob/afbc8a030383caaa0888ea1cdedf7c079f18c048/pkg/sources/summary/client.go#L93-L94 to

Host:   "kubernetes.default.svc:443",
Path:   fmt.Sprintf("/api/v1/nodes/%s/proxy/stats/summary/", host),

rebuild the metrics-server and change https://github.com/kubernetes-incubator/metrics-server/blob/afbc8a030383caaa0888ea1cdedf7c079f18c048/deploy/1.8%2B/resource-reader.yaml#L9-L12 to

  resources:
  - pods
  - nodes
  - nodes/proxy
  - nodes/stats

Hope this will help someones who struggling to use --kubelet-insecure-tls like me

paphillon · 2019-08-25T18:50:15Z

@secret104278 - This is how I have solved it. I have different CA for kubelet certs in my setup.

CA public key of kubelet CA authority. With this we don't need to set flag --kubelet-insecure-tls

kubectl create secret generic ca-kubelet-cert --from-file=./ca-kubelet.pem -n kube-system

Metrics server parameter

- --kubelet-certificate-authority=/etc/kubelet/ca/ca-kubelet.pem

Devying · 2020-06-24T11:39:34Z

@secret104278 - This is how I have solved it. I have different CA for kubelet certs in my setup.

CA public key of kubelet CA authority. With this we don't need to set flag --kubelet-insecure-tls
kubectl create secret generic ca-kubelet-cert --from-file=./ca-kubelet.pem -n kube-system 
Metrics server parameter
- --kubelet-certificate-authority=/etc/kubelet/ca/ca-kubelet.pem

I Got this

W0624 11:37:52.647355       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0624 11:37:52.647424       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x128 pc=0x16908a5]

goroutine 1 [running]:
sigs.k8s.io/metrics-server/cmd/metrics-server/app/options.Options.kubeletConfig(0xc0005361c8, 0xc000196d80, 0xc000075800, 0xc00030cf7c, 0x0, 0x0, 0x0, 0xdf8475800, 0x0, 0x280a, ...)
	/root/metrics-server/cmd/metrics-server/app/options/options.go:175 +0x105
sigs.k8s.io/metrics-server/cmd/metrics-server/app/options.Options.MetricsServerConfig(0xc0005361c8, 0xc000196d80, 0xc000075800, 0xc00030cf7c, 0x0, 0x0, 0x0, 0xdf8475800, 0x0, 0x280a, ...)
	/root/metrics-server/cmd/metrics-server/app/options/options.go:118 +0x11c
sigs.k8s.io/metrics-server/cmd/metrics-server/app.runCommand(0xc0000ce360, 0xc00007e1e0, 0x0, 0x0)
	/root/metrics-server/cmd/metrics-server/app/start.go:52 +0xa2
sigs.k8s.io/metrics-server/cmd/metrics-server/app.NewMetricsServerCommand.func1(0xc00050e780, 0xc0004b64b0, 0x0, 0x3, 0x0, 0x0)
	/root/metrics-server/cmd/metrics-server/app/start.go:37 +0x33
github.com/spf13/cobra.(*Command).execute(0xc00050e780, 0xc00003a0d0, 0x3, 0x3, 0xc00050e780, 0xc00003a0d0)
	/home/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0xc00050e780, 0xc000074180, 0x0, 0x0)
	/home/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	/home/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
	/root/metrics-server/cmd/metrics-server/metrics-server.go:38 +0xb6

mhermosi · 2022-12-16T16:55:45Z

if you are trying to access an external resource with an SSL generated by letsencrypt you should install the file isrgrootx1.pem.txt as ca-certificates.pem under a ConfigMap in Kubernetes.

The ConfigMap should be called for example ca-pemstore. Then on your deployment or pod definition you create a volume from that configMap and a volumeMount on the container with mountPath /etc/ssl/certs/ca-certificates.pem and subPath: ca-certificates.pem (I think the name must match the filename you have on the configMap) I recommend renaming the txt file to ca-certificates.pem and then run the command

kubectl -n create configmap ca-pemstore --from-file ca-certificates.pem

with that your applications inside the pod will be able to have the root certificate from letsencrypt

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Nov 7, 2018

metalmatze mentioned this issue Nov 15, 2018

Different CAs for kubeClient and promClient kubernetes-sigs/prometheus-adapter#130

Closed

itskingori mentioned this issue Dec 31, 2018

Fix metrics server addon kubernetes/kops#6201

Merged

Baykonur mentioned this issue Mar 20, 2019

[Kubeadm] "x509: certificate signed by unknown authority #67

Closed

nlamirault added a commit to zeiot-old/jarvis that referenced this issue Mar 24, 2019

Fix: x509: certificate signed by unknown authority

db406f0

See: kubernetes-sigs/metrics-server#146 Signed-off-by: Nicolas Lamirault <[email protected]>

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 12, 2019

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 12, 2019

k8s-ci-robot closed this as completed Aug 11, 2019

kurtostfeld mentioned this issue Feb 22, 2020

metrics-server with k8s 1.14.9 eks.8: Unable to authenticate the request due to an error: x509: certificate signed by unknown authority #443

Closed

BenTheElder mentioned this issue Apr 7, 2020

Add kubeadmConfigPatches and metrics-server references to docs kubernetes-sigs/kind#1456

Closed

JackTiber mentioned this issue May 6, 2020

Educative.io Course on Advanced Monitoring / Logging vfarcic/k8s-specs#17

Open

MatthewPattell mentioned this issue Oct 21, 2020

Document securing connection between Metrics Server <-> Kubelet #576

Open

Prabhaker24 mentioned this issue Nov 25, 2020

Requirements For HPA To Work For Pravega pravega/pravega-operator#480

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x509: certificate signed by unknown authority #146

x509: certificate signed by unknown authority #146

max201920 commented Oct 1, 2018

DirectXMan12 commented Oct 2, 2018

max201920 commented Oct 3, 2018

DirectXMan12 commented Oct 3, 2018

albrr commented Oct 9, 2018

k0nstantinv commented Oct 17, 2018

albrr commented Oct 17, 2018

towolf commented Oct 28, 2018

rayanebel commented Nov 5, 2018

DirectXMan12 commented Nov 7, 2018

rayanebel commented Nov 14, 2018

brancz commented Nov 15, 2018

metalmatze commented Nov 15, 2018

brancz commented Nov 15, 2018

Aiqbal1234 commented Nov 27, 2018 •

edited

Loading

DirectXMan12 commented Nov 28, 2018

desmatz commented Jan 31, 2019 •

edited

Loading

sarneaud commented Feb 3, 2019 •

edited

Loading

ghost commented Feb 14, 2019

ioanc commented Mar 4, 2019

paphillon commented Mar 8, 2019

charlesguoooo commented Mar 13, 2019 •

edited

Loading

Baykonur commented Mar 13, 2019

sarneaud commented Mar 14, 2019

Baykonur commented Mar 14, 2019 •

edited

Loading

Baykonur commented Mar 14, 2019

fejta-bot commented Jun 12, 2019

fejta-bot commented Jul 12, 2019

fejta-bot commented Aug 11, 2019

k8s-ci-robot commented Aug 11, 2019

secret104278 commented Aug 25, 2019

paphillon commented Aug 25, 2019

Devying commented Jun 24, 2020 •

edited

Loading

mhermosi commented Dec 16, 2022

x509: certificate signed by unknown authority #146

x509: certificate signed by unknown authority #146

Comments

max201920 commented Oct 1, 2018

DirectXMan12 commented Oct 2, 2018

max201920 commented Oct 3, 2018

DirectXMan12 commented Oct 3, 2018

albrr commented Oct 9, 2018

k0nstantinv commented Oct 17, 2018

albrr commented Oct 17, 2018

towolf commented Oct 28, 2018

rayanebel commented Nov 5, 2018

DirectXMan12 commented Nov 7, 2018

rayanebel commented Nov 14, 2018

brancz commented Nov 15, 2018

metalmatze commented Nov 15, 2018

brancz commented Nov 15, 2018

Aiqbal1234 commented Nov 27, 2018 • edited Loading

DirectXMan12 commented Nov 28, 2018

desmatz commented Jan 31, 2019 • edited Loading

sarneaud commented Feb 3, 2019 • edited Loading

ghost commented Feb 14, 2019

ioanc commented Mar 4, 2019

paphillon commented Mar 8, 2019

charlesguoooo commented Mar 13, 2019 • edited Loading

Baykonur commented Mar 13, 2019

sarneaud commented Mar 14, 2019

Baykonur commented Mar 14, 2019 • edited Loading

Baykonur commented Mar 14, 2019

fejta-bot commented Jun 12, 2019

fejta-bot commented Jul 12, 2019

fejta-bot commented Aug 11, 2019

k8s-ci-robot commented Aug 11, 2019

secret104278 commented Aug 25, 2019

paphillon commented Aug 25, 2019

Devying commented Jun 24, 2020 • edited Loading

mhermosi commented Dec 16, 2022

Aiqbal1234 commented Nov 27, 2018 •

edited

Loading

desmatz commented Jan 31, 2019 •

edited

Loading

sarneaud commented Feb 3, 2019 •

edited

Loading

charlesguoooo commented Mar 13, 2019 •

edited

Loading

Baykonur commented Mar 14, 2019 •

edited

Loading

Devying commented Jun 24, 2020 •

edited

Loading