Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition #8756

Closed
staticdev opened this issue Jul 17, 2020 · 35 comments Β· Fixed by #8839
Labels
addon/ingress kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@staticdev
Copy link
Contributor

Steps to reproduce the issue:

  1. minikube start --vm-driver=docker --memory 3072
  2. minikube addons enable ingress

Full output of failed command: tried 3 times.
πŸ˜„ minikube v1.12.1 on Ubuntu 20.04
✨ Using the docker driver based on user configuration
πŸ‘ Starting control plane node minikube in cluster minikube
πŸ”₯ Creating docker container (CPUs=2, Memory=3072MB) ...
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.2 ...
πŸ”Ž Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
πŸ„ Done! kubectl is now configured to use "minikube"
πŸ”Ž Verifying ingress addon...

πŸ’£ enable failed: run callbacks: running callbacks: [verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition]

😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
πŸ‘‰ https://github.com/kubernetes/minikube/issues/new/choose

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

@tstromberg
Copy link
Contributor

Can you please share the output of:

  • kubectl descr po -A

The only thing new in v.12.1 is an updated version of ingress: #8705

I was not able to replicate this issue on Docker/Linux or Hyperkit/macOS. My suspicion is that perhaps the download failed? Regardless, we can do a better job of showing an actionable error message.

@tstromberg tstromberg added addon/ingress triage/needs-information Indicates an issue needs more information in order to work on it. kind/support Categorizes issue or PR as a support question. labels Jul 17, 2020
@tstromberg tstromberg changed the title Error on ingress addon after upgrading from v1.12.0 to v1.12.1 verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition Jul 17, 2020
@staticdev
Copy link
Contributor Author

staticdev commented Jul 18, 2020

I saw on #8705 that nginx-ingress-controller or kube-webhook-certgen, I could try downgrading separately each of these components to try to isolate the error. But I am not sure how is the best way for me to do this test.

kubectl descr po -A

kube-system   coredns-66bff467f8-pkgjv                    1/1     Running     0          4m34s
kube-system   etcd-minikube                               1/1     Running     0          4m31s
kube-system   ingress-nginx-admission-create-f4kgh        0/1     Completed   0          4m17s
kube-system   ingress-nginx-admission-patch-v7wth         0/1     Completed   0          4m17s
kube-system   ingress-nginx-controller-69ccf5d9d8-8pmdm   1/1     Running     0          4m17s
kube-system   kube-apiserver-minikube                     1/1     Running     0          4m31s
kube-system   kube-controller-manager-minikube            1/1     Running     0          4m31s
kube-system   kube-proxy-7m7tl                            1/1     Running     0          4m34s
kube-system   kube-scheduler-minikube                     1/1     Running     0          4m31s
kube-system   metrics-server-5f956b6d5f-9zfvx             1/1     Running     0          4m22s
kube-system   storage-provisioner                         1/1     Running     0          4m35s

@shyamkhakhar9
Copy link

I also face the same issue while trying to addons and I have ingress nginx yaml where when I apply kubectl it gives me following error.. how to resolve that

Error from server (InternalError): error when creating "ingress-service.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.kube-system.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.97.204.103:443: connect: connection refused

@staticdev
Copy link
Contributor Author

@tstromberg update: after some more tests, I see that most of the times it gives the error, but sometimes not. As you said, the max waiting time may be on the edge of the average time it takes. I would like to isolate which version update is causing this increased time so that we can upstream this issue to the component with degraded performance.

How can I do that? Do I need to recompile minikube from source or there is an easier way to do it?

@chitranshi21
Copy link

chitranshi21 commented Aug 3, 2020

Hi,
I am getting the same error.
enable failed: run callbacks: running callbacks: [verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition]
minikube version - V1.12.1
Any updates ?

@anotherandi
Copy link

I tried down-dating to 1.12.0 but had the same issue on windows 10.

@chukmunnlee
Copy link

chukmunnlee commented Aug 14, 2020

I manage to get enable the ingress add on by enabling cni and install Cilium. Have not tried with other providers

I'm on 1.12.3

@jaurestchoffo
Copy link

jaurestchoffo commented Aug 21, 2020

Still no solution for this problem ??? I've been struggling for a while for my project and I have to support soon
thank you in advance for any solution
that is what i have

πŸ”Ž Verifying ingress addon...

πŸ’£ enable failed: run callbacks: running callbacks: [verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition]

😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
πŸ‘‰ https://github.com/kubernetes/minikube/issues/new/choose

@vvlookman
Copy link

Because the image is not pulled, try workaround:

$ kubectl get pod -A

List all the system pod names

$ kubectl describe pod/ingress-nginx-controller-xxxxxx --namespace=kube-system

Find out the image name which has not been pulled successfully, in my case is quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0

$ minikube ssh
$ docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0

Pull the image in minikube vm manully

@djvasquezvm
Copy link

Same problem. I followed your steps but my problem was:
Warning FailedMount 12m (x7 over 12m) kubelet, minikube MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found

So I realized it was SELinux. If you have SELinux enforcing, try using permissive mode.

Worked for me!

Because the image is not pulled, try workaround:

$ kubectl get pod -A

List all the system pod names

$ kubectl describe pod/ingress-nginx-controller-xxxxxx --namespace=kube-system

Find out the image name which has not been pulled successfully, in my case is quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0

$ minikube ssh
$ docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0

Pull the image in minikube vm manully

@ELTEGANI
Copy link

ELTEGANI commented Sep 4, 2020

I faced the same I found the problem from my poor internet connection

@839928622
Copy link

I faced the same I found the problem from my poor internet connection
did u get it fix later on ?

@ELTEGANI
Copy link

I faced the same I found the problem from my poor internet connection
did u get it fix later on ?

Yeb fixed it

@renich
Copy link

renich commented Sep 30, 2020

In my case, I live in MΓ©xico and I have a 100 Mbps connection. Everything installed correctly but not the nginx ingress:

0s          Normal    BackOff             pod/ingress-nginx-controller-789d9c4dc-7wpkv    Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ImagePullBackOff
0s          Normal    BackOff             pod/ingress-nginx-controller-789d9c4dc-7wpkv    Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ImagePullBackOff
0s          Normal    Pulling             pod/ingress-nginx-controller-789d9c4dc-7wpkv    Pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ErrImagePull
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Failed to pull image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20": rpc error: code = Unknown desc = Invalid image name "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20", unknown transport "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller"
0s          Normal    BackOff             pod/ingress-nginx-controller-789d9c4dc-7wpkv    Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ImagePullBackOff
0s          Warning   Failed              pod/ingress-nginx-controller-789d9c4dc-7wpkv    Error: ImagePullBackOff

@renich
Copy link

renich commented Sep 30, 2020

It seems that the URL being used isn't correct. If you go here: https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/US/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20 it works, but the URL logged is: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

If you put them side by side, Γ¬t's easy to notice the difference:

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20
us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

@renich
Copy link

renich commented Sep 30, 2020

So, for me, edititng the deployment with the correct image URL seems to have worked.

image

To do this, just: kubectl -n kube-system edit deployment ingress-nginx-controller and look for "image:" and change it to:

image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

image

@pimenvibritania
Copy link

So, for me, edititng the deployment with the correct image URL seems to have worked.

image

To do this, just: kubectl -n kube-system edit deployment ingress-nginx-controller and look for "image:" and change it to:

image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

image

WOOWWW Thank You!! its work!

@WillNilges
Copy link

I'm still having trouble. I edited the image url, but I'm still not able to pull the image

[wnilges@thingsboard-minikube ~]$ kubectl get pods -A
NAMESPACE     NAME                                        READY   STATUS              RESTARTS   AGE
kube-system   coredns-f9fd979d6-lm5j8                     1/1     Running             0          26m
kube-system   etcd-minikube                               1/1     Running             0          26m
kube-system   ingress-nginx-admission-create-gxwkc        0/1     ImagePullBackOff    0          9m19s
kube-system   ingress-nginx-admission-patch-mz5bx         0/1     ImagePullBackOff    0          9m8s
kube-system   ingress-nginx-controller-7977f85d85-xbnwr   0/1     ContainerCreating   0          9m45s
kube-system   kube-apiserver-minikube                     1/1     Running             0          26m
kube-system   kube-controller-manager-minikube            1/1     Running             0          26m
kube-system   kube-proxy-2p9wr                            1/1     Running             0          26m
kube-system   kube-scheduler-minikube                     1/1     Running             0          26m
kube-system   storage-provisioner                         1/1     Running             1          26m

@WillNilges
Copy link

For reference, here's my full minikube start attempt:

[wnilges@thingsboard-minikube root]$  minikube start --addons=ingress --alsologtostderr -v=5

(It's really big)

@WillNilges
Copy link

Weather update: Works on my local Ubuntu 20 machine, not so much on my Fedora Server 32 VM. Highly wack.

@hongbo-miao
Copy link

hongbo-miao commented Dec 1, 2020

Thanks @renich ! Met same issue. After kubectl -n kube-system edit deployment ingress-nginx-controller, change

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

to

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

(You can try verify by opening first URL in browser, which gets 404 error, the second one which is also version 0.34.1 which can be opened)

Basically just need remove :v0.34.1.

After editing, sine it is Vim, just type :wq or :x and hit Enter.

And then do minikube addons enable ingress again.

@danielbeeke
Copy link

In my case I had something running on port 80, that can also happen.

@sahilgrover306
Copy link

IT Works after changing the image URL. Thanks for the help

@EHJ-52n
Copy link

EHJ-52n commented Apr 12, 2021

Removing the version tag :v0.44.0 following @hongbo-miao instructions above solved this issue for me.

Hence, the issues seems to be not solved in

$ minikube version
minikube version: v1.19.0
commit: 15cede53bdc5fe242228853e737333b09d4336b5
Pods before "fix" application
$ kubectl get pod -A
NAMESPACE              NAME                                        READY   STATUS      RESTARTS   AGE
demo-sensorweb         helgoland-cf7ccb684-2w8ds                   1/1     Running     19         32d
ingress-nginx          ingress-nginx-admission-create-n5wpw        0/1     Completed   0          119m
ingress-nginx          ingress-nginx-admission-patch-knxnr         0/1     Completed   1          119m
ingress-nginx          ingress-nginx-controller-5d88495688-xdtsd   0/1     Pending     0          119m
kube-system            coredns-74ff55c5b-j956h                     1/1     Running     67         104d
kube-system            etcd-minikube                               1/1     Running     88         104d
kube-system            ingress-nginx-admission-create-pkqsj        0/1     Completed   0          40d
kube-system            ingress-nginx-admission-patch-zzpjx         0/1     Completed   0          40d
kube-system            ingress-nginx-controller-65cf89dc4f-jhhqt   1/1     Running     34         40d
kube-system            kube-apiserver-minikube                     1/1     Running     73         40d
kube-system            kube-controller-manager-minikube            1/1     Running     73         40d
kube-system            kube-proxy-thhs2                            1/1     Running     48         52d
kube-system            kube-scheduler-minikube                     1/1     Running     73         40d
kube-system            registry-pj8fc                              1/1     Running     21         32d
kube-system            registry-proxy-72gcx                        1/1     Running     20         32d
kube-system            storage-provisioner                         1/1     Running     88         104d
kubernetes-dashboard   dashboard-metrics-scraper-f6647bd8c-wfx2l   1/1     Running     37         40d
kubernetes-dashboard   kubernetes-dashboard-968bcb79-mdc6n         1/1     Running     52         40d
Pods after "fix" application
$ kubectl get pod -A
NAMESPACE              NAME                                        READY   STATUS      RESTARTS   AGE
demo-sensorweb         helgoland-cf7ccb684-2w8ds                   1/1     Running     21         32d
ingress-nginx          ingress-nginx-admission-create-n5wpw        0/1     Completed   0          134m
ingress-nginx          ingress-nginx-admission-patch-knxnr         0/1     Completed   1          134m
ingress-nginx          ingress-nginx-controller-5d88495688-xdtsd   1/1     Running     2          134m
kube-system            coredns-74ff55c5b-j956h                     1/1     Running     69         104d
kube-system            etcd-minikube                               1/1     Running     90         104d
kube-system            ingress-nginx-admission-create-pkqsj        0/1     Completed   0          40d
kube-system            ingress-nginx-admission-patch-zzpjx         0/1     Completed   0          40d
kube-system            ingress-nginx-controller-6cb6b94bc9-79mh6   0/1     Pending     0          12m
kube-system            kube-apiserver-minikube                     1/1     Running     75         40d
kube-system            kube-controller-manager-minikube            1/1     Running     75         40d
kube-system            kube-proxy-thhs2                            1/1     Running     50         52d
kube-system            kube-scheduler-minikube                     1/1     Running     75         40d
kube-system            registry-pj8fc                              1/1     Running     23         32d
kube-system            registry-proxy-72gcx                        1/1     Running     22         32d
kube-system            storage-provisioner                         1/1     Running     91         104d
kubernetes-dashboard   dashboard-metrics-scraper-f6647bd8c-wfx2l   1/1     Running     39         40d
kubernetes-dashboard   kubernetes-dashboard-968bcb79-mdc6n         1/1     Running     55         40d
Minikube start logs
$ minikube start
πŸ˜„  minikube v1.19.0 on Ubuntu 20.04
✨  Using the docker driver based on existing profile
πŸ‘  Starting control plane node minikube in cluster minikube
πŸ”„  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.0 ...
πŸ”Ž  Verifying Kubernetes components...
    β–ͺ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    β–ͺ Using image kubernetesui/dashboard:v2.1.0
    β–ͺ Using image kubernetesui/metrics-scraper:v1.0.4
    β–ͺ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    β–ͺ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    β–ͺ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    β–ͺ Using image registry:2.7.1
    β–ͺ Using image gcr.io/google_containers/kube-registry-proxy:0.4
πŸ”Ž  Verifying registry addon...
πŸ”Ž  Verifying ingress addon...
🌟  Enabled addons: default-storageclass, registry, storage-provisioner, dashboard, ingress
πŸ„  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

I can provide additional information, if required.

Kind regards and have a nice day!

@vagdevik
Copy link

vagdevik commented May 13, 2021

I had the same issue.

The error disappeared when I have run the same command

minikube addons enable ingress

again after a minute.
Then, the ingress was set-up after some time. So, the issue might also be due to some network delays, I guess.

@hongbo-miao
Copy link

hongbo-miao commented May 13, 2021

Thanks @renich ! Met same issue. After kubectl -n kube-system edit deployment ingress-nginx-controller, change

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

to

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

(You can try verify by opening first URL in browser, which gets 404 error, the second one which is also version 0.34.1 which can be opened)

Basically just need remove :v0.34.1.

After editing, sine it is Vim, just type :wq or :x and hit Enter.

And then do minikube addons enable ingress again.

Just add another update. In later installation, this bug is still there, however, it seems not affecting anything. Even I have a new fresh installation from ground. What I did is just

minikube start
minikube addons enable ingress

and let minikube retry at the background, eventually it will succeed. The longest time takes like 3 hours after over 10 times retry I think.

@TimothyLoyer
Copy link

Does anyone know why the URL is wrong or causing issues, and when it will be fixed? Having the ingress-controller break every time I (or someone on my team) restarts Minikube is becoming quite a problem.

@TimothyLoyer
Copy link

TimothyLoyer commented Jun 16, 2021

I did some searching and found this- looks like the registry used for the IngressController does not support using a tag and image digest in the same URL the way that docker.io does. google/go-containerregistry#351

Maybe I'm mistaken and GCR doesn't work exactly like this (the issue I linked was closed with a merge request quite some time ago) but it does seem to be behaving similarly. This page offers two distinct solutions for pulling from GCR: https://cloud.google.com/container-registry/docs/pushing-and-pulling#pulling_images_from_a_registry

@williamhanseneric
Copy link

Having something so fundamentally broken like ingress is fairly disheartening...

@dogewzy
Copy link

dogewzy commented Nov 5, 2021

for user in china,you can specified http_proxy_port and https_proxy_port in minikube

@mnahinkhan
Copy link

What worked for me:

minikube start --driver=virtualbox

@aliaksander-samuseu
Copy link

aliaksander-samuseu commented Jul 12, 2022

It's still a problem, I'm facing this issue in Ubuntu 20 - I'm running # minikube addons enable ingress, it hangs there for a while, then exits with this error. How can it not be fixed 2 years after it's reported?
PS.
Somehow it works now, I'll observe it for a bit longer..

@miladr0
Copy link

miladr0 commented Jul 15, 2022

I ran to this issue on minikube version v1.26.0 too.
steps that I've done to solve it:

  • type kubectl get pods -n ingress-nginx and you should see something like the following:
NAME                                        READY   STATUS      RESTARTS        AGE
ingress-nginx-admission-create-zhp94        0/1     Completed   0               3h59m
ingress-nginx-admission-patch-ggrd8         0/1     Completed   0               3h59m
ingress-nginx-controller-755dfbfc65-xtrs5   0/1     ImagePullBackOff     1 (3h11m ago)   3h59m
  • and then if you do kubectl describe pod -n ingress-nginx <pod-name> you should be able to see all of the logs within the pod.
    I checked the logs of ingress-nginx-controller-755dfbfc65-xtrs5 and saw its cant download the image.

  • I logged in to minikube container using minikube ssh

  • then manually download ingress-nginx-controller image using below command:

time docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8

Note: I needed to download v1.2.1 of ingress-nginx but if you need to download a newer or older version you can check the google containers registry to find correct sha256:xxxx

@Anyc66666666
Copy link

minikube addons enable ingress --images="KubeWebhookCertgenCreate=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1,KubeWebhookCertgenPatch=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1,IngressController=google_containers/nginx-ingress-controller:v1.2.1" --registries="IngressController=registry.cn-hangzhou.aliyuncs.com"

@jxiang93
Copy link

start your minikube with command below and it should work

minikube start --insecure-registry="registry.k8s.io"

then

minikube addons enable ingress

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
addon/ingress kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Development

Successfully merging a pull request may close this issue.