FailedCreatePodSandBox with DNS pod #507

smileisak · 2017-10-24T10:55:48Z

Is this a BUG REPORT

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", 
  GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-
  17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", 
  GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-
  11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
Bare metal

OS (e.g. from /etc/os-release):

  NAME="Red Hat Enterprise Linux Server"
  VERSION="7.2 (Maipo)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="7.2"
  PRETTY_NAME="Red Hat Enterprise Linux Server 7.2 (Maipo)"

Kernel (e.g. uname -a):

  Linux s00vl9974124 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

Others:

What happened?

Trying to install Kubernetes Cluster version 1.8.1 using kubeadm. But After installing network add-on ( weave, calico and flannel ) i have a weird issue with dns pod.

kubectl get pods --all-namespaces -o wide

    kube-system   calico-etcd-fxd9k                          1/1       Running             0          8m        10.244.175.11   s00vl9974124
    kube-system   calico-kube-controllers-6ff88bf6d4-nrvh2   1/1       Running             0          8m        10.244.175.12   s00vl9974125
    kube-system   calico-node-qdlch                          2/2       Running             0          8m        10.244.175.11   s00vl9974124
    kube-system   calico-node-scrm4                          2/2       Running             0          8m        10.244.175.12   s00vl9974125
    kube-system   etcd-s00vl9974124                          1/1       Running             0          16m       10.244.175.11   s00vl9974124
    kube-system   kube-apiserver-s00vl9974124                1/1       Running             0          15m       10.244.175.11   s00vl9974124
    kube-system   kube-controller-manager-s00vl9974124       1/1       Running             1          16m       10.244.175.11   s00vl9974124
    kube-system   kube-dns-545bc4bfd4-j2rwn                  0/3       ContainerCreating   0          15m       <none>          s00vl9974125
    kube-system   kube-proxy-ck75k                           1/1       Running             0          11m       10.244.175.11   s00vl9974124
    kube-system   kube-proxy-sj5cx                           1/1       Running             0          12m       10.244.175.12   s00vl9974125
    kube-system   kube-scheduler-s00vl9974124                1/1       Running             0          15m       10.244.175.11   s00vl9974124

k describe pod kube-dns-545bc4bfd4-j2rwn -n kube-system

Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                    -------------   --------        ------                  -------
  17m           14m             14      default-scheduler                       Warning         FailedScheduling        No nodes are available that match all of the predicates: NodeNotReady (1), NodeUnderDiskPressure (1).
  14m           13m             3       default-scheduler                       Warning         FailedScheduling        No nodes are available that match all of the predicates: NodeNotReady (2), NodeUnderDiskPressure (1).
  12m           10m             9       default-scheduler                       Warning         FailedScheduling        No nodes are available that match all of the predicates: NodeNotReady (2).
  8m            8m              1       kubelet, s00vl9974125                   Normal          SuccessfulMountVolume   MountVolume.SetUp succeeded for volume "kube-dns-config"
  8m            8m              1       kubelet, s00vl9974125                   Normal          SuccessfulMountVolume   MountVolume.SetUp succeeded for volume "kube-dns-token-r5lh5"
  8m            8m              1       kubelet, s00vl9974125                   Warning         DNSSearchForming        Search Line limits were exceeded, some dns names have been omitted, the applied search line is: service.ns.svc.cluster.local fra.net.intra fr.net.intra net.intra fr.xcd.net.intra insurance.corp
  8m            8m              1       kubelet, s00vl9974125                   Warning         FailedCreatePodSandBox  Failed create pod sandbox.
  8m            2m              11      kubelet, s00vl9974125                   Warning         FailedSync              Error syncing pod
  7m            2m              11      kubelet, s00vl9974125                   Normal          SandboxChanged          Pod sandbox changed, it will be killed and re-created.

What you expected to happen?

DNS pod runs and get and IP address.

How to reproduce it (as minimally and precisely as possible)?

Install Kubernetes using kubeadm behind a corporate porxy using this command:

kubeadm init --apiserver-advertise-address=10.244.175.11

Anything else we need to know?

I'm behind a corporate proxy.
I've added these ENV vars:

printf -v pool '%s,' 192.168.0.{1..253}
printf -v service '%s,' 10.96.0.{1..253}
export no_proxy=$no_proxy,${service%,}${pool%,}
export NO_PROXY=$no_proxy

journalctl -u kubelet | grep sandbox

Oct 24 11:07:25 s00vl9974124 kubelet[23323]: E1024 11:07:25.489030   23323 
kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-545bc4bfd4-pz5m2_kube-
system(7aed8012-b899-11e7-9876-005056008ce4)" failed: rpc error: code = DeadlineExceeded desc = 
context deadline exceeded

The text was updated successfully, but these errors were encountered:

dims · 2017-10-24T20:41:20Z

@smileisak Can you please check the kubelet logs to see if you can find a clue on why FailedCreatePodSandBox occurs?

jamiehannaford · 2017-11-01T15:03:28Z

@smileisak Were you able to pin down what happened in the kubelet logs?

arunpjohny · 2017-11-02T10:05:11Z

@jamiehannaford @dims

We are getting the same error in multiple deployments, one example is the echoheaders deployment that we are using to test ingress

Deployment

apiVersion: v1
kind: ReplicationController
metadata:
  name: echoheaders
  namespace: nginx-ingress
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: echoheaders
    spec:
      containers:
      - name: echoheaders
        image: gcr.io/google_containers/echoserver:1.4
        ports:
        - containerPort: 8080

kubectl -n nginx-ingress describe pods echoheaders-7pbth

Name:           echoheaders-7pbth
Namespace:      nginx-ingress
Node:           ip-x-x-x-x.ap-south-1.compute.internal/xxx.xxx.xxx.xxx
Start Time:     Thu, 02 Nov 2017 14:36:06 +0530
Labels:         app=echoheaders
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"nginx-ingress","name":"echoheaders","uid":"0ad6cbdf-bbdb-11e7-97ea-020...
Status:         Pending
IP:
Created By:     ReplicationController/echoheaders
Controlled By:  ReplicationController/echoheaders
Containers:
  echoheaders:
    Container ID:
    Image:              gcr.io/google_containers/echoserver:1.4
    Image ID:
    Port:               8080/TCP
    State:              Waiting
      Reason:           ContainerCreating
    Ready:              False
    Restart Count:      0
    Environment:        <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-fz9kh (ro)
Conditions:
  Type          Status
  Initialized   True
  Ready         False
  PodScheduled  True
Volumes:
  default-token-fz9kh:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-fz9kh
    Optional:   false
QoS Class:      BestEffort
Node-Selectors: <none>
Tolerations:    node.alpha.kubernetes.io/notReady:NoExecute for 300s
                node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  FirstSeen     LastSeen        Count   From                                               SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                                               -------------   --------        ------                  -------
  48m           48m             1       default-scheduler                                                  Normal          Scheduled               Successfully assigned echoheaders-7pbth to ip-x-x-x-x.ap-south-1.compute.internal
  48m           48m             1       kubelet, ip-x-x-x-x.ap-south-1.compute.internal                    Normal          SuccessfulMountVolume   MountVolume.SetUp succeeded for volume "default-token-fz9kh"
  44m           44m             1       kubelet, ip-x-x-x-x.ap-south-1.compute.internal                    Warning         FailedCreatePodSandBox  Failed create pod sandbox.
  44m           18m             14      kubelet, ip-x-x-x-x.ap-south-1.compute.internal                    Normal          SandboxChanged          Pod sandbox changed, it will be killed and re-created.
  44m           2m              22      kubelet, ip-x-x-x-x.ap-south-1.compute.internal                    Warning         FailedSync              Error syncing pod

journalctl -u kubelet | less | grep echoheaders

Nov 02 09:06:06 ip-172-18-64-25 kubelet[5422]: I1102 09:06:06.665903    5422 kubelet.go:1837] SyncLoop (ADD, "api"): "echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)"
Nov 02 09:06:06 ip-172-18-64-25 kubelet[5422]: I1102 09:06:06.707652    5422 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-fz9kh" (UniqueName: "kubernetes.io/secret/0f6268a9-bfad-11e7-9385-027a5c8a1c86-default-token-fz9kh") pod "echoheaders-7pbth" (UID: "0f6268a9-bfad-11e7-9385-027a5c8a1c86")
Nov 02 09:06:06 ip-172-18-64-25 kubelet[5422]: I1102 09:06:06.807905    5422 reconciler.go:257] operationExecutor.MountVolume started for volume "default-token-fz9kh" (UniqueName: "kubernetes.io/secret/0f6268a9-bfad-11e7-9385-027a5c8a1c86-default-token-fz9kh") pod "echoheaders-7pbth" (UID: "0f6268a9-bfad-11e7-9385-027a5c8a1c86")
Nov 02 09:06:06 ip-172-18-64-25 kubelet[5422]: I1102 09:06:06.814501    5422 operation_generator.go:484] MountVolume.SetUp succeeded for volume "default-token-fz9kh" (UniqueName: "kubernetes.io/secret/0f6268a9-bfad-11e7-9385-027a5c8a1c86-default-token-fz9kh") pod "echoheaders-7pbth" (UID: "0f6268a9-bfad-11e7-9385-027a5c8a1c86")
Nov 02 09:06:06 ip-172-18-64-25 kubelet[5422]: I1102 09:06:06.971330    5422 kuberuntime_manager.go:370] No sandbox for pod "echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)" can be found. Need to start a new one
Nov 02 09:06:07 ip-172-18-64-25 kubelet[5422]: I1102 09:06:07.320564    5422 kubelet.go:1871] SyncLoop (PLEG): "echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)", event: &pleg.PodLifecycleEvent{ID:"0f6268a9-bfad-11e7-9385-027a5c8a1c86", Type:"ContainerStarted", Data:"4f486f52b1de4f0482e3deec9f5b6e1fcfbe7fe1552abcdeea16fd38dfb36e05"}
Nov 02 09:10:06 ip-172-18-64-25 kubelet[5422]: E1102 09:10:06.971757    5422 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 02 09:10:06 ip-172-18-64-25 kubelet[5422]: E1102 09:10:06.971776    5422 kuberuntime_manager.go:632] createPodSandbox for pod "echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 02 09:10:06 ip-172-18-64-25 kubelet[5422]: E1102 09:10:06.971877    5422 pod_workers.go:182] Error syncing pod 0f6268a9-bfad-11e7-9385-027a5c8a1c86 ("echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)"), skipping: failed to "CreatePodSandbox" for "echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)" with CreatePodSandboxError: "CreatePodSandbox for pod \"echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)\" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Nov 02 09:10:07 ip-172-18-64-25 kubelet[5422]: I1102 09:10:07.533116    5422 kuberuntime_manager.go:401] Sandbox for pod "echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)" has no IP address.  Need to start a new one
Nov 02 09:12:07 ip-172-18-64-25 kubelet[5422]: E1102 09:12:07.533485    5422 pod_workers.go:182] Error syncing pod 0f6268a9-bfad-11e7-9385-027a5c8a1c86 ("echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)"), skipping: failed to "KillPodSandbox" for "0f6268a9-bfad-11e7-9385-027a5c8a1c86" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Nov 02 09:12:07 ip-172-18-64-25 kubelet[5422]: I1102 09:12:07.988436    5422 kuberuntime_manager.go:401] Sandbox for pod "echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)" has no IP address.  Need to start a new one
Nov 02 09:14:07 ip-172-18-64-25 kubelet[5422]: E1102 09:14:07.988835    5422 pod_workers.go:182] Error syncing pod 0f6268a9-bfad-11e7-9385-027a5c8a1c86 ("echoheaders-7pbth_nginx-ingress(0f6268a9-bfad-11e7-9385-027a5c8a1c86)"), skipping: failed to "KillPodSandbox" for "0f6268a9-bfad-11e7-9385-027a5c8a1c86" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"

Environment

kops version: Version 1.7.1 (git-c69b811)
kubectl version:

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:38:10Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

smileisak · 2017-11-06T09:01:09Z

@dims && @jamiehannaford this is kubelet logs:

Nov 06 09:58:48 s00vl9974123 kubelet[558]: E1106 09:58:48.855639     558 
kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-545bc4bfd4-mjn49_kube-
system(2ffdece1-c2d0-11e7-98cb-005056008ce3)" failed: rpc error: code = Unknown desc = failed to 
start sandbox container for pod "kube-dns-545bc4bfd4-mjn49": Error response from daemon: oci 
runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: 
container init caused \"process_linux.go:351: running prestart hook 0 caused \\\"fork/exec 
/usr/bin/dockerd.#prelink#.XSURZo (deleted): no such file or directory\\\"\""

Nov 06 09:58:48 s00vl9974123 kubelet[558]: E1106 09:58:48.855652     558 
kuberuntime_manager.go:632] createPodSandbox for pod "kube-dns-545bc4bfd4-mjn49_kube-
system(2ffdece1-c2d0-11e7-98cb-005056008ce3)" failed: rpc error: code = Unknown desc = failed to 
start sandbox container for pod "kube-dns-545bc4bfd4-mjn49": Error response from daemon: oci 
runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: 
container init caused \"process_linux.go:351: running prestart hook 0 caused \\\"fork/exec 
/usr/bin/dockerd.#prelink#.XSURZo (deleted): no such file or directory\\\"\""

Nov 06 09:58:48 s00vl9974123 kubelet[558]: E1106 09:58:48.855714     558 pod_workers.go:182] Error 
syncing pod 2ffdece1-c2d0-11e7-98cb-005056008ce3 ("kube-dns-545bc4bfd4-mjn49_kube-
system(2ffdece1-c2d0-11e7-98cb-005056008ce3)"), skipping: failed to "CreatePodSandbox" for "kube-
dns-545bc4bfd4-mjn49_kube-system(2ffdece1-c2d0-11e7-98cb-005056008ce3)" with 
CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-545bc4bfd4-mjn49_kube-
system(2ffdece1-c2d0-11e7-98cb-005056008ce3)\" failed: rpc error: code = Unknown desc = failed to 
start sandbox container for pod \"kube-dns-545bc4bfd4-mjn49\": Error response from daemon: oci 
runtime error: container_linux.go:265: starting container process caused \"process_linux.go:368: 
container init caused \\\"process_linux.go:351: running prestart hook 0 caused \\\\\\\"fork/exec 
/usr/bin/dockerd.#prelink#.XSURZo (deleted): no such file or directory\\\\\\\"\\\"\""

 Nov 06 09:58:48 s00vl9974123 kubelet[558]: W1106 09:58:48.855949     558 container.go:354] Failed 
 to create summary reader for "/kubepods/burstable/pod2ffdece1-c2d0-11e7-98cb-
 005056008ce3/07bda4b2ec90b54b282cbd296b90f7b5d94658886481301148e8586a397b850f": none 
 of the resources are being tracked.

jamiehannaford · 2017-11-06T11:51:19Z

@smileisak Hmm, that's strange. Would you mind restarting docker with systemctl restart docker and seeing if that changes anything? xref: moby/moby#29640

smileisak · 2017-11-06T12:06:37Z

@jamiehannaford by restarting docker everything is working now ! but it's weird as an error

jamiehannaford · 2017-11-06T12:58:31Z

@smileisak Yeah... I think this is more of a Docker issue than kubeadm. Are you okay with me closing this?

smileisak · 2017-11-06T13:00:36Z

@jamiehannaford yes you can close this issue 😄 thank you for your help ! 😃

shibd · 2017-12-01T13:14:50Z

Hello, have you solved it

jamiehannaford · 2017-12-01T13:40:04Z

@shibd Try restarting your docker, that could be the issue. If not, can you open a new issue with your kubelet logs.

srossross · 2017-12-02T06:16:22Z

I am having this issue and docker restart did not solve it.

jamiehannaford · 2017-12-04T18:30:09Z

@srossross Please open a new issue with full DNS and kubelet logs.

lkjsavolainen · 2017-12-29T15:46:41Z

In my case this was caused by cni-plugins-amd64-v0.6.0.tgz not downloading completely and therefore the loopback plugin was missing which prevented the pods from being created on one of the worker nodes. Found this out by sshing to the worker node and looking at the logs with journalctl

kwaazaar · 2018-02-09T09:24:32Z

I got this error when Docker no longer worked after installing ubuntu 16.04 upgrades. Upgrading Docker to the latest version (17.12-ce in my case), everything immediately worked fine again. So this errors seems to be caused by Docker errors mostly.

luxas added the kind/support Categorizes issue or PR as a support question. label Oct 27, 2017

jamiehannaford closed this as completed Nov 6, 2017

floreks mentioned this issue Nov 17, 2017

dashboard pod can't run on current kubeadm kubernetes/dashboard#2547

Closed

srossross mentioned this issue Dec 4, 2017

kube-dns fails to start with - FailedCreatePodSandBox error #587

Closed

kacole2 mentioned this issue Apr 30, 2019

Prometheus Deployment Not Working on PKS CNA-Tech/Apps-on-PKS#12

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FailedCreatePodSandBox with DNS pod #507

FailedCreatePodSandBox with DNS pod #507

smileisak commented Oct 24, 2017 •

edited

Loading

dims commented Oct 24, 2017

jamiehannaford commented Nov 1, 2017

arunpjohny commented Nov 2, 2017 •

edited

Loading

smileisak commented Nov 6, 2017

jamiehannaford commented Nov 6, 2017

smileisak commented Nov 6, 2017

jamiehannaford commented Nov 6, 2017

smileisak commented Nov 6, 2017

shibd commented Dec 1, 2017

jamiehannaford commented Dec 1, 2017

srossross commented Dec 2, 2017

jamiehannaford commented Dec 4, 2017

lkjsavolainen commented Dec 29, 2017

kwaazaar commented Feb 9, 2018

FailedCreatePodSandBox with DNS pod #507

FailedCreatePodSandBox with DNS pod #507

Comments

smileisak commented Oct 24, 2017 • edited Loading

Is this a BUG REPORT

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

dims commented Oct 24, 2017

jamiehannaford commented Nov 1, 2017

arunpjohny commented Nov 2, 2017 • edited Loading

Environment

smileisak commented Nov 6, 2017

jamiehannaford commented Nov 6, 2017

smileisak commented Nov 6, 2017

jamiehannaford commented Nov 6, 2017

smileisak commented Nov 6, 2017

shibd commented Dec 1, 2017

jamiehannaford commented Dec 1, 2017

srossross commented Dec 2, 2017

jamiehannaford commented Dec 4, 2017

lkjsavolainen commented Dec 29, 2017

kwaazaar commented Feb 9, 2018

smileisak commented Oct 24, 2017 •

edited

Loading

arunpjohny commented Nov 2, 2017 •

edited

Loading