Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ingress: timed out waiting for the condition - Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection #9615

Closed
gongzhengyang opened this issue Nov 6, 2020 · 10 comments
Closed

ingress: timed out waiting for the condition - Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection #9615

gongzhengyang opened this issue Nov 6, 2020 · 10 comments
Labels
cause/firewall-or-proxy When firewalls or proxies seem to be interfering kind/support Categorizes issue or PR as a support question. l/zh-CN Issues in or relating to Chinese triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@gongzhengyang
Copy link

重现问题所需的命令：
minikube addons enable ingress
失败的命令的完整输出：


🔎 Verifying ingress addon...

❌ Exiting due to MK_ENABLE: run callbacks: running callbacks: [verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition]

😿 If the above advice does not help, please let us know:
👉 https://github.com/kubernetes/minikube/issues/new/choose

minikube logs命令的输出：

Nov 06 02:59:17 minikube kubelet[4352]: E1106 02:59:17.047642 4352 kuberuntime_image.go:51] Pull image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1106 11:04:19.543294 107885 out.go:280] unable to parse "Nov 06 02:59:17 minikube kubelet[4352]: E1106 02:59:17.047979 4352 kuberuntime_manager.go:804] container &Container{Name:controller,Image:us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20,Command:[],Args:[/nginx-ingress-controller --configmap=$(POD_NAMESPACE)/nginx-load-balancer-conf --report-node-internal-ip-address --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services --udp-services-configmap=$(POD_NAMESPACE)/udp-services --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:http,HostPort:80,ContainerPort:80,Protocol:TCP,HostIP:,},ContainerPort{Name:https,HostPort:443,ContainerPort:443,Protocol:TCP,HostIP:,},ContainerPort{Name:webhook,HostPort:0,ContainerPort:8443,Protocol:TCP,HostIP:,},},Env:[]EnvVar{EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {} 100m DecimalSI},memory: {{94371840 0} {} 90Mi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:webhook-cert,ReadOnly:true,MountPath:/usr/local/certificates/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:ingress-nginx-token-527pw,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 10254 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:10,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 10254 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:10,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},Lifecycle:&Lifecycle{PostStart:nil,PreStop:&Handler{Exec:&ExecAction{Command:[/wait-shutdown],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[NET_BIND_SERVICE],Drop:[ALL],},Privileged:nil,SELinuxOptions:nil,RunAsUser:*101,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*true,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n": template: Nov 06 02:59:17 minikube kubelet[4352]: E1106 02:59:17.047979 4352 kuberuntime_manager.go:804] container &Container{Name:controller,Image:us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20,Command:[],Args:[/nginx-ingress-controller --configmap=$(POD_NAMESPACE)/nginx-load-balancer-conf --report-node-internal-ip-address --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services --udp-services-configmap=$(POD_NAMESPACE)/udp-services --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:http,HostPort:80,ContainerPort:80,Protocol:TCP,HostIP:,},ContainerPort{Name:https,HostPort:443,ContainerPort:443,Protocol:TCP,HostIP:,},ContainerPort{Name:webhook,HostPort:0,ContainerPort:8443,Protocol:TCP,HostIP:,},},Env:[]EnvVar{EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {} 100m DecimalSI},memory: {{94371840 0} {} 90Mi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:webhook-cert,ReadOnly:true,MountPath:/usr/local/certificates/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:ingress-nginx-token-527pw,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 10254 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:10,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 10254 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:10,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},Lifecycle:&Lifecycle{PostStart:nil,PreStop:&Handler{Exec:&ExecAction{Command:[/wait-shutdown],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[NET_BIND_SERVICE],Drop:[ALL],},Privileged:nil,SELinuxOptions:nil,RunAsUser:*101,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*true,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
:1: unexpected "}" in operand - returning raw string.
Nov 06 02:59:17 minikube kubelet[4352]: E1106 02:59:17.047979 4352 kuberuntime_manager.go:804] container &Container{Name:controller,Image:us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20,Command:[],Args:[/nginx-ingress-controller --configmap=$(POD_NAMESPACE)/nginx-load-balancer-conf --report-node-internal-ip-address --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services --udp-services-configmap=$(POD_NAMESPACE)/udp-services --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:http,HostPort:80,ContainerPort:80,Protocol:TCP,HostIP:,},ContainerPort{Name:https,HostPort:443,ContainerPort:443,Protocol:TCP,HostIP:,},ContainerPort{Name:webhook,HostPort:0,ContainerPort:8443,Protocol:TCP,HostIP:,},},Env:[]EnvVar{EnvVar{Name:POD_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},EnvVar{Name:POD_NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {} 100m DecimalSI},memory: {{94371840 0} {} 90Mi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:webhook-cert,ReadOnly:true,MountPath:/usr/local/certificates/,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:ingress-nginx-token-527pw,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 10254 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:10,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 10254 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:10,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},Lifecycle:&Lifecycle{PostStart:nil,PreStop:&Handler{Exec:&ExecAction{Command:[/wait-shutdown],},HTTPGet:nil,TCPSocket:nil,},},TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[NET_BIND_SERVICE],Drop:[ALL],},Privileged:nil,SELinuxOptions:nil,RunAsUser:*101,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*true,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5): ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Nov 06 02:59:17 minikube kubelet[4352]: E1106 02:59:17.048602 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 06 02:59:30 minikube kubelet[4352]: E1106 02:59:30.021995 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 02:59:43 minikube kubelet[4352]: E1106 02:59:43.021575 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 02:59:54 minikube kubelet[4352]: E1106 02:59:54.022075 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:00:06 minikube kubelet[4352]: E1106 03:00:06.024387 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:00:21 minikube kubelet[4352]: E1106 03:00:21.023546 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:00:32 minikube kubelet[4352]: E1106 03:00:32.021466 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:00:44 minikube kubelet[4352]: E1106 03:00:44.021498 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:00:56 minikube kubelet[4352]: E1106 03:00:56.021401 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:01:11 minikube kubelet[4352]: E1106 03:01:11.021817 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:01:22 minikube kubelet[4352]: E1106 03:01:22.023057 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:01:33 minikube kubelet[4352]: E1106 03:01:33.021570 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:01:46 minikube kubelet[4352]: E1106 03:01:46.026929 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:01:59 minikube kubelet[4352]: E1106 03:01:59.021786 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:02:12 minikube kubelet[4352]: E1106 03:02:12.021971 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:02:27 minikube kubelet[4352]: E1106 03:02:27.023599 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:02:38 minikube kubelet[4352]: E1106 03:02:38.021996 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:02:51 minikube kubelet[4352]: E1106 03:02:51.021894 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:03:06 minikube kubelet[4352]: E1106 03:03:06.023237 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:03:21 minikube kubelet[4352]: E1106 03:03:21.021346 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:03:36 minikube kubelet[4352]: E1106 03:03:36.022418 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:03:47 minikube kubelet[4352]: E1106 03:03:47.021173 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:03:58 minikube kubelet[4352]: E1106 03:03:58.021922 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""
Nov 06 03:04:11 minikube kubelet[4352]: E1106 03:04:11.021747 4352 pod_workers.go:191] Error syncing pod e9f15888-3c6e-4035-ac95-00a2cf41b9e5 ("ingress-nginx-controller-789d9c4dc-jhjrc_kube-system(e9f15888-3c6e-4035-ac95-00a2cf41b9e5)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""

==> storage-provisioner [76ab105c7e7c] <==
I1106 02:51:38.173403 1 leaderelection.go:242] attempting to acquire leader lease kube-system/k8s.io-minikube-hostpath...
I1106 02:51:38.178897 1 leaderelection.go:252] successfully acquired lease kube-system/k8s.io-minikube-hostpath
I1106 02:51:38.179233 1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"k8s.io-minikube-hostpath", UID:"9690a861-d1cb-447f-b4ab-690e29775718", APIVersion:"v1", ResourceVersion:"472", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' minikube_34ea5cb0-5769-408d-8f6d-fec8ab6dd0d6 became leader
I1106 02:51:38.179484 1 controller.go:799] Starting provisioner controller k8s.io/minikube-hostpath_minikube_34ea5cb0-5769-408d-8f6d-fec8ab6dd0d6!
I1106 02:51:38.279563 1 controller.go:848] Started provisioner controller k8s.io/minikube-hostpath_minikube_34ea5cb0-5769-408d-8f6d-fec8ab6dd0d6!

==> storage-provisioner [c7ee543aeef2] <==
F1106 02:51:37.441532 1 main.go:39] error getting server version: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout

使用的操作系统版本：
Ubuntu18.04
@gongzhengyang gongzhengyang added the l/zh-CN Issues in or relating to Chinese label Nov 6, 2020
@oldbirdgo
Copy link

me. too

@priyawadhwa
Copy link

Hey @2976560783 I noticed this log in your output:

failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20""

which seems really similar to this issue we had a few months ago: #8756 (comment)

Could you try what this user did in this comment (#8756 (comment)) and see if that resolves the issue?

@sharifelgamal sharifelgamal added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/needs-information Indicates an issue needs more information in order to work on it. labels Nov 17, 2020
@ahojcn
Copy link

ahojcn commented Dec 4, 2020

I have the same problem and my solution is as follows，I hope it will help you：

0、run Shadowsocks and connect a server
1、download the image us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.40.2
2、export image to temp.tar use command docker save us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.40.2 > temp.tar
3、upload temp.tar to minikube node
4、exec in minikube ssh，load it and exit docker load < temp.tar
5、run command minikube addons enable ingress again!

I didn't find some images of China, but it worked!
image

@ahojcn ahojcn mentioned this issue Dec 5, 2020
Closed
@harryttd
Copy link

harryttd commented Dec 9, 2020

I have an error that is a little different:

🔎  Verifying ingress addon...

❌  Exiting due to MK_ENABLE: run callbacks: running callbacks: [sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.19.2/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml: Process exited with status 1
stdout:
configmap/nginx-load-balancer-conf unchanged
configmap/tcp-services unchanged
configmap/udp-services unchanged
serviceaccount/ingress-nginx unchanged
serviceaccount/ingress-nginx-admission unchanged
clusterrole.rbac.authorization.k8s.io/system::ingress-nginx unchanged
role.rbac.authorization.k8s.io/system::ingress-nginx unchanged
rolebinding.rbac.authorization.k8s.io/system::ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/system::ingress-nginx unchanged
role.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
rolebinding.rbac.authorization.k8s.io/system::ingress-nginx-admission unchanged
deployment.apps/ingress-nginx-controller unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
job.batch/ingress-nginx-admission-create unchanged
service/ingress-nginx-controller-admission unchanged

stderr:
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
The Job "ingress-nginx-admission-patch" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"ingress-nginx-admission-patch", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.kubernetes.io/component":"admission-webhook", "app.kubernetes.io/instance":"ingress-nginx", "app.kubernetes.io/name":"ingress-nginx", "controller-uid":"0b2e66d0-8422-43e9-a88d-4b85c6d28e9d", "job-name":"ingress-nginx-admission-patch"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"patch", Image:"jettech/kube-webhook-certgen:v1.3.0", Command:[]string(nil), Args:[]string{"patch", "--webhook-name=ingress-nginx-admission", "--namespace=kube-system", "--patch-mutating=false", "--secret-name=ingress-nginx-admission", "--patch-failure-policy=Fail"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc00e80a1b0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"ingress-nginx-admission", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00c3fea00), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable
]

@tstromberg tstromberg changed the title minikube addons enable ingress　verifying ingress addon pods : timed out waiting for the condition ingress: timed out waiting for the condition - Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection Dec 10, 2020
@tstromberg
Copy link
Contributor

这是因为中国无法访问托管入口插件的资源库（us.gcr.io）。

您可以在使用过--image-mirror-country = cn的群集还是手动设置--image-repository的群集吗？ 我相信那应该可以。

This is because China cannot access the repository (us.gcr.io) which hosts the ingress addon.

Can you try a cluster where you've used --image-mirror-country=cn or manually set --image-repository? I believe it should work then.

@tstromberg tstromberg added cause/firewall-or-proxy When firewalls or proxies seem to be interfering triage/needs-information Indicates an issue needs more information in order to work on it. kind/support Categorizes issue or PR as a support question. and removed priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/needs-information Indicates an issue needs more information in order to work on it. labels Dec 14, 2020
@priyawadhwa
Copy link

Hey @249043822 were you able to try the above suggestions?

@medyagh
Copy link
Member

medyagh commented Feb 10, 2021

Hi @2976560783 , I haven't heard back from you, I wonder if you still have this issue?
Regrettably, there isn't enough information in this issue to make it actionable, and a long enough duration has passed, so this issue is likely difficult to replicate.

I will close this issue for now but please feel free to reopen whenever you feel ready to provide more information.

@medyagh medyagh closed this as completed Feb 10, 2021
@Bytesu
Copy link

Bytesu commented Aug 22, 2021 •
edited
Loading

I have the same problem and my solution is as follows，I hope it will help you：

0、run Shadowsocks and connect a server
1、download the image us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.40.2
2、export image to temp.tar use command docker save us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.40.2 > temp.tar
3、upload temp.tar to minikube node
4、exec in minikube ssh，load it and exit docker load < temp.tar
5、run command minikube addons enable ingress again!

I didn't find some images of China, but it worked!
image

I couldn't understand. why does not it works for me ?

logs is same as below :

I0822 14:48:11.939518 416739 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0822 14:48:11.939551 416739 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: []

@shyandsy
Copy link

I'm trying to use image from Tsinghua university

vim /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=kubernetes
baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/yum/repos/kubernetes-el7-$basearch
enabled=1

why minikube still try to load image from k8s.gcr.io/ingress-nginx

@gongzhengyang
Copy link
Author

gongzhengyang commented Apr 26, 2023 •
edited
Loading

  1. Sorry for replying after many years, I just saw this issue last night, I was a student when I opened this issue,
    and now I am a programming developer. Time is so fast.
  2. I am very grateful to the open source developers for the enthusiastic answers and contributions.
  3. There is a big Wall between China and the world, in China, Github is sometimes inaccessible, others like Google, Facebook, and Netflix are absolutely inaccessible, a VPN or v2ray technology is needed for many websites and services.
  4. This Issue is just because of the Wall, hopefully one day the Wall will come down just like the Berlin Wall.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cause/firewall-or-proxy When firewalls or proxies seem to be interfering kind/support Categorizes issue or PR as a support question. l/zh-CN Issues in or relating to Chinese triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@tstromberg @medyagh @shyandsy @oldbirdgo @Bytesu @priyawadhwa @harryttd @sharifelgamal @gongzhengyang @ahojcn