Reaching running application in minikube at 'localhost' on Linux

[cheslijones]

I use --driver=docker and minikube tunnel in Windows (WSL2 and native) and macOS (Intel and M1) which makes the app running in the minikube cluster accessible in browser at localhost after applying ingress-nginx.

To make things consistent across Linux, macOS and Windows for a list of reasons that really aren't relevant to the question, I'd like to do the same in Linux (Ubuntu/Pop!_OS in particular).

I know Docker works differently with networking in Linux than it does with macOS and Windows, which is likely what is causing the issue.

For macOS and Windows, I'd normally just use the following which works regardless if using Docker Desktop or not:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/cloud/deploy.yaml

For Linux, I've tried three different things:

# the recommended method for minikube (which doesn't work in macOS nor WIndows if using --driver=docker)
minikube addons enable ingress

# Docker Desktop
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/cloud/deploy.yaml

# Bare Metal
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml

The results, whether using minikube tunnel or not in Linux, are the same: "localhost refused to connect."

So my question is, which may be a question for ingress-nginx, is there a way to get the combination of minikube and ingress-nginx to serve the application at localhost in Linux like it does for Windows and macOS?

If not, I guess I need to be revise my dev deployment scripts to make this exception, but would like to avoid that if possible.

[afbjorklund]

So the difference between those two deployments are if it uses a LoadBalancer or a NodePort

@@ -273,8 +273,7 @@
   name: ingress-nginx-controller
   namespace: ingress-nginx
 spec:
-  type: LoadBalancer
-  externalTrafficPolicy: Local
+  type: NodePort
   ports:
     - name: http
       port: 80
@@ -329,7 +328,6 @@
                   - /wait-shutdown
           args:
             - /nginx-ingress-controller
-            - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
             - --election-id=ingress-controller-leader
             - --ingress-class=nginx
             - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller

So if I understood your question, you want "minikube tunnel" to work with Docker Engine

https://minikube.sigs.k8s.io/docs/handbook/accessing/

I think this is currently governed by "NeedsPortForward":

// NeedsPortForward returns true if driver is unable provide direct IP connectivity
func NeedsPortForward(name string) bool {
        if !IsKIC(name) {
                return false
        }
        if oci.IsExternalDaemonHost(name) {
                return true
        }
        // Docker for Desktop
        return runtime.GOOS == "darwin" || runtime.GOOS == "windows" || detect.IsMicrosoftWSL()
}

Whereas Linux actually provides network access (docker0)

The logic above says that VMs and Docker Engine have IPs,
but that Remote Docker and Docker Desktop does not.

i.e. their VM has a "secret" IP which is not accessible directly

https://docs.docker.com/docker-for-windows/networking/#known-limitations-use-cases-and-workarounds
https://docs.docker.com/docker-for-mac/networking/#known-limitations-use-cases-and-workarounds

Basically, you want a tunnel from 127.0.0.1 to 172.17.0.0

It should be doable as an option (to tunnel), at least ?

[afbjorklund]

Now that docker driver creates private networks, it is normally 192.168.0.0 being used.

$ minikube tunnel
[sudo] password for anders: 
Status:	
	machine: minikube
	pid: 464139
	route: 10.96.0.0/12 -> 192.168.49.2
	minikube: Running
	services: [ingress-nginx-controller]
    errors: 
		minikube: no errors
		router: no errors
		loadbalancer emulator: no errors

But there is no need for a localhost ssh proxy, since the network is reachable directly...

$ kubectl get svc -A
NAMESPACE       NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
default         kubernetes                           ClusterIP      10.96.0.1       <none>          443/TCP                      16h
ingress-nginx   ingress-nginx-controller             LoadBalancer   10.103.153.91   10.103.153.91   80:31197/TCP,443:30922/TCP   9m57s
ingress-nginx   ingress-nginx-controller-admission   ClusterIP      10.101.113.43   <none>          443/TCP                      9m57s
kube-system     kube-dns                             ClusterIP      10.96.0.10      <none>          53/UDP,53/TCP,9153/TCP       16h

$ curl http://10.103.153.91:80
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

But we could make it "opt-in", also on Linux, and it would use the same ssh there:

❗ The service ingress-nginx-controller requires privileged ports to be exposed: [80 443]
🔑 sudo permission will be asked for it.
🏃 Starting tunnel for service ingress-nginx-controller.

sudo ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -N [email protected] -p 49167 -i /home/anders/.minikube/machines/minikube/id_rsa -L 80:10.103.153.91:80 -L 443:10.103.153.91:443

$ kubectl get svc -A
NAMESPACE       NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default         kubernetes                           ClusterIP      10.96.0.1       <none>        443/TCP                      16h
ingress-nginx   ingress-nginx-controller             LoadBalancer   10.103.153.91   127.0.0.1     80:31197/TCP,443:30922/TCP   14m
ingress-nginx   ingress-nginx-controller-admission   ClusterIP      10.101.113.43   <none>        443/TCP                      14m
kube-system     kube-dns                             ClusterIP      10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP       16h

$ curl http://127.0.0.1:80
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

(I did this by making NeedsPortForward return true also for regular Linux (not just WSL).

@@ -168,7 +168,8 @@ func NeedsPortForward(name string) bool {
                return true
        }
        // Docker for Desktop
-       return runtime.GOOS == "darwin" || runtime.GOOS == "windows" || detect.IsMicrosoftWSL()
+       //return runtime.GOOS == "darwin" || runtime.GOOS == "windows" || detect.IsMicrosoftWSL()
+       return true
 }
 
 // HasResourceLimits returns true if driver can set resource limits such as memory size or CPU count.

This was the option I was referring to, make it possible to select whether to use ssh or not.

But it's a bit of a hack either way, and you would be better off asking for the IP...

kubectl describe svc -n ingress-nginx ingress-nginx-controller

https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/

Instead of assuming the broken networking model of Docker Desktop being the norm ?

[medyagh]

@cheslijones does that answer your question ?
if you use tunnel you could use the IPs

[medyagh]

/triage needs-information
/kind support

[afbjorklund]

I think we need the option, at least for the ssh driver. There are no guarantees the other ports are opened...

But maybe it will just be another hardcoded selection.

        if IsSSH(name) {
                return true
        }

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten