Setup kuberenetes dashboard #1558

weihan1394 · 2021-04-27T03:51:54Z

I am trying to set up Kubernetes with bare machine and I have deployed my Nginx in the default namespace and I am trying to create a virtual server to route the dashboard. However, when I try to access the URL, it directed me to error 404 and when I view the logs from the pod it seems to be having a broken header. I came across kubernetes/ingress-nginx#3996 and they were giving suggestion to add the proxy protocol and if I were to add the below snippet to my config map, the other routes will fail.

proxy-protocol: "True"
real-ip-header: "proxy_protocol"
set-real-ip-from: "0.0.0.0/0"

2021/04/27 07:43:28 [error] 145#145: *155 broken header: "▬♥☺☻☺☺�♥♥�#�↓�|��-�B�d�s�HZ��]!��↨��☻ O|����%���o�/k��R       ��¶�u�♂����VI"��‼☺‼☻‼♥�+�/�,�0̨̩�‼�¶��/" while reading PROXY protocol, client: 192.168.254.9, server: 0.0.0.0:443

nginx-ingress-configmap

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-ingress-cm
  namespace: default
data:
  http2: "true"
  ssl-protocols: TLSv1.2 TLSv1.3
  ssl-prefer-server-ciphers: "true"
  ssl-ciphers: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
  set-real-ip-from: 0.0.0.0/0
  real-ip-header: X-Forwarded-For
  server-snippets: |
    proxy_hide_header Strict-Transport-Security;
    add_header Strict-Transport-Security "max-age=31536000" always;

Any hints what I have done wrongly? Thanks in advance.

The text was updated successfully, but these errors were encountered:

pleshakov · 2021-04-27T17:23:09Z

Hi @weihan1394

The proxy protocol is only needed if you front Ingress Controller pods with a load balancer like ELB, which uses proxy protocol to pass the client IP to NGINX:
clients -> ELB -> NGINX IC -> backends

Is that the case?

serhiiromaniuk · 2021-04-27T21:27:07Z

Hello there! So, what's the true way to reach Clients IPs @pleshakov ? #1561

weihan1394 · 2021-04-27T23:51:44Z

Hi @pleshakov, I am currently using this way: clients -> NGINX IC -> backends. Apparently, this configuration works for all my application but currently I am trying to setup the kubernetes dashboard with their default yaml

pleshakov · 2021-04-28T00:00:04Z

@weihan1394
Could you remove all the proxy protocol related configuration (proxy-protocol, real-ip-header and set-real-ip-from)?
Could you share the Ingress or VirtualServer resource you used for the dashboard?
Could you also try again to access the dashboard and share the output and the corresponding log lines?
Could you also share how you try to access the dashboard? For example, a curl command

weihan1394 · 2021-04-28T00:09:03Z

Hi @pleshakov Thanks for your kind help in advance. I have removed those proxy protocols. I am trying to access the dashboard from google chrome.

When I access the dashboard again with k8.moonshot.com it directed me to error 400 page.

virtual server for dashboard.yaml

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: kubernetes-dashboard-vs
  namespace: kubernetes-dashboard
spec:
  host: k8.moonshot.com
  tls:
    secret: nginx-tls-secret
    redirect:
      enable: true
      code: 301
  upstreams:
  - name: kubernetes-dashboard
    service: kubernetes-dashboard
    port: 443
  routes:
  - path: /
    action:
      pass: kubernetes-dashboard

dashboard service yaml

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard

Logs:

2021/04/28 00:05:05 [error] 162#162: *225 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.254.9, server: k8.moonshot.com, request: "GET / HTTP/2.0", upstream: "http://192.168.253.130:8443/", host: "k8.moonshot.com"       
192.168.254.9 - - [28/Apr/2021:00:05:05 +0000] "GET / HTTP/2.0" 400 48 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" "-"

-- edited
I was looking at the kubernetes dashboard logs I was thinking is it because the dashboard is using another secret token which is not the same as the one I am using for the virtual server.

 kubectl logs kubernetes-dashboard-78c79f97b4-6l2gm  -n kubernetes-dashboard
2021/04/27 01:36:45 Starting overwatch
2021/04/27 01:36:45 Using namespace: kubernetes-dashboard
2021/04/27 01:36:45 Using in-cluster config to connect to apiserver
2021/04/27 01:36:45 Using secret token for csrf signing
2021/04/27 01:36:45 Initializing csrf token from kubernetes-dashboard-csrf secret
2021/04/27 01:36:45 Empty token. Generating and storing in a secret kubernetes-dashboard-csrf
2021/04/27 01:36:45 Successful initial request to the apiserver, version: v1.21.0
2021/04/27 01:36:45 Generating JWE encryption key
2021/04/27 01:36:45 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting
2021/04/27 01:36:45 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard
2021/04/27 01:36:46 Initializing JWE encryption key from synchronized object
2021/04/27 01:36:46 Creating in-cluster Sidecar client
2021/04/27 01:36:46 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2021/04/27 01:36:46 Auto-generating certificates
2021/04/27 01:36:46 Successfully created certificates
2021/04/27 01:36:46 Serving securely on HTTPS port: 8443
2021/04/27 01:37:16 Successful request to sidecar

pleshakov · 2021-04-28T00:43:05Z

the 400 response code is because NGINX sent an HTTP request to the HTTPs port on the dashboard. To fix that, it is necessary to enable HTTPs between NGINX and the dashboard.

Could you update the VS to do that?

  - name: kubernetes-dashboard
    service: kubernetes-dashboard
    port: 443
    tls:
      enable: true

weihan1394 · 2021-04-28T00:50:10Z

@pleshakov oh wow! kudos, it works now. May I check if I understand my setup of the virtual server correctly?

I have deployed my NGINX in the default namespace, but I have also created 2 applications in 2 different namespaces. Let call it dev and prod. Am I right to say that I can just simply create the virtual server on the respective namespace? I will just need to create all the resources required in their own namespace.

dev:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: hellok8s-app-vs-2
  namespace: dev
spec:
  host: helloworld2.moonshot.com
  tls:
    secret: nginx-tls-secret
    redirect:
      enable: true
      code: 301
  upstreams:
  - name: hellok8s-2
    service: hellok8s-service-2
    port: 8080
  routes:
  - path: /
    action:
      pass: hellok8s-2

prod:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: hellok8s-app-vs-2
  namespace: prod
spec:
  host: helloworld2.moonshot.com
  tls:
    secret: nginx-tls-secret
    redirect:
      enable: true
      code: 301
  upstreams:
  - name: hellok8s-2
    service: hellok8s-service-2
    port: 8080
  routes:
  - path: /
    action:
      pass: hellok8s-2

However when I try to access the app from another namespace is it correct that I will just need to put the route as [namespace]/[name]

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: hellok8s-app-vs
spec:
  host: helloworld.moonshot.com
  tls:
    secret: nginx-tls-secret
    # basedOn: scheme
    redirect:
      enable: true
      code: 301
  upstreams:
  - name: hellok8s
    service: hellok8s-service
    port: 8080
  routes:
  - path: /one
    action:
      pass: hellok8s
  - path: /two
    route: dev/hellok8s-2

pleshakov · 2021-04-28T17:31:59Z

@weihan1394

Am I right to say that I can just simply create the virtual server on the respective namespace? I will just need to create all the resources required in their own namespace.

that is correct

However when I try to access the app from another namespace is it correct that I will just need to put the route as [namespace]/[name]

path: /two
route: dev/hellok8s-2

in this case, dev/hellok8s-2 needs to be a VirtualServerRoute, not VirtualServer. We have an example -- https://github.com/nginxinc/kubernetes-ingress/tree/master/examples-of-custom-resources/cross-namespace-configuration

weihan1394 · 2021-04-29T03:58:24Z

@pleshakov Thanks for your help, it helps a lot!

weihan1394 changed the title ~~Route kubernetes nginx-ingress to dashboard in another namespace~~ Setup kuberenetes dashboard Apr 27, 2021

weihan1394 closed this as completed Apr 28, 2021

weihan1394 reopened this Apr 28, 2021

weihan1394 closed this as completed Apr 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup kuberenetes dashboard #1558

Setup kuberenetes dashboard #1558

weihan1394 commented Apr 27, 2021 •

edited

Loading

pleshakov commented Apr 27, 2021

serhiiromaniuk commented Apr 27, 2021

weihan1394 commented Apr 27, 2021

pleshakov commented Apr 28, 2021

weihan1394 commented Apr 28, 2021 •

edited

Loading

pleshakov commented Apr 28, 2021

weihan1394 commented Apr 28, 2021 •

edited

Loading

pleshakov commented Apr 28, 2021

weihan1394 commented Apr 29, 2021

Setup kuberenetes dashboard #1558

Setup kuberenetes dashboard #1558

Comments

weihan1394 commented Apr 27, 2021 • edited Loading

pleshakov commented Apr 27, 2021

serhiiromaniuk commented Apr 27, 2021

weihan1394 commented Apr 27, 2021

pleshakov commented Apr 28, 2021

weihan1394 commented Apr 28, 2021 • edited Loading

pleshakov commented Apr 28, 2021

weihan1394 commented Apr 28, 2021 • edited Loading

pleshakov commented Apr 28, 2021

weihan1394 commented Apr 29, 2021

weihan1394 commented Apr 27, 2021 •

edited

Loading

weihan1394 commented Apr 28, 2021 •

edited

Loading

weihan1394 commented Apr 28, 2021 •

edited

Loading