rhel7 k3s start not working with containerd - failed to find snapshotter overlayfs #495

rako1980 · 2019-05-23T15:05:14Z

I am trying to start k3s in rhel7. The service starts fine but no pods get deployed. Looks like some issues in containerd as shown in the log.
I suspect this warning in containerd is causing the issue:
failed to create CRI service: failed to find snapshotter "overlayfs"

System: rhel7.6
Trying to install k3s in air gap mode.
File contents:

./
- k3s
- k3s-airgap-images-amd64.tar

#./k3s server 
...
INFO[2019-05-23T07:52:35.466217824-07:00] Run: k3s kubectl
INFO[2019-05-23T07:52:35.466227353-07:00] k3s is up and running
INFO[2019-05-23T07:52:35.513838186-07:00] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log
INFO[2019-05-23T07:52:35.513952421-07:00] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
INFO[2019-05-23T07:52:35.514177382-07:00] Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory"
INFO[2019-05-23T07:52:36.515150846-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T07:52:37.515878061-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
E0523 07:52:37.557634   32117 resource_quota_controller.go:171] initial monitor sync has error: [couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=helmcharts": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=helmcharts", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=addons": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=addons", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=listenerconfigs": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=listenerconfigs"]
INFO[2019-05-23T07:52:38.516591198-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T07:52:39.517415316-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
...

The last line repeats indefinitely.

containerd.log:

time="2019-05-23T07:51:52.904216373-07:00" level=info msg="Connect containerd service"
time="2019-05-23T07:51:52.904375778-07:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to find snapshotter \"overlayfs\""
time="2019-05-23T07:51:52.904402647-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2019-05-23T07:51:52.904679823-07:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock
time="2019-05-23T07:51:52.904699156-07:00" level=info msg="containerd successfully booted in 0.003223s"

Note:

firewalld disabled
host IP resolved to server name in /etc/hosts

The text was updated successfully, but these errors were encountered:

erikwilson · 2019-05-23T15:55:04Z

Is the overlay kernel module available? (output for lsmod | grep overlay)
Anything useful in the logs leading up to k3s is up and running?

rako1980 · 2019-05-23T17:09:13Z

@erikwilson

# lsmod | grep over
overlay                71964  0

Yes k3s service is running. Here is the status

[root@xxxxxx ~]# systemctl status k3s
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-05-23 09:52:07 PDT; 10min ago
     Docs: https://k3s.io
  Process: 40305 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 40303 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
 Main PID: 40308 (k3s-server)
    Tasks: 30
   Memory: 102.8M
   CGroup: /system.slice/k3s.service
           ├─40308 /usr/local/bin/k3s server
           └─40331 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var...

May 23 10:02:41 xxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:41.038711533-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:42 xxxxxxxxx k3s[40308]: time="2019-05-23T10:02:42.039755725-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:43 xxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:43.041077281-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:44 xxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:44.042223950-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:45 xxxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:45.043232193-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:46 xxxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:46.044413974-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:47 xxxxxxxxx k3s[40308]: time="2019-05-23T10:02:47.045353133-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:48 xxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:48.046353999-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:49 xxxxxxxxxxx k3s[40308]: time="2019-05-23T10:02:49.047118545-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
May 23 10:02:50 xxxxxxxxx k3s[40308]: time="2019-05-23T10:02:50.048046848-07:00" level=info msg="Waiting for containerd startup: rpc error: code = Un...meService"
Hint: Some lines were ellipsized, use -l to show in full.

# k3s kubectl get all --all-namespaces
NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP                  16m
kube-system   service/kube-dns     ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP,9153/TCP   16m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/1     0            0           16m

NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   0/1                      16m

rako1980 · 2019-05-23T17:42:51Z

Here is the detail of the log when I start as #./k3s server

# ./k3s server
INFO[2019-05-23T10:41:15.365566723-07:00] Starting k3s v0.5.0 (8c0116dd)
INFO[2019-05-23T10:41:15.366560494-07:00] Running kube-apiserver --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-username-headers=X-Remote-User --allow-privileged=true --advertise-address=127.0.0.1 --tls-cert-file=/var/lib/rancher/k3s/server/tls/localhost.crt --insecure-port=0 --bind-address=127.0.0.1 --requestheader-extra-headers-prefix=X-Remote-Extra- --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --advertise-port=6445 --service-account-issuer=k3s --api-audiences=unknown --kubelet-client-key=/var/lib/rancher/k3s/server/tls/token-node.key --requestheader-allowed-names=kubernetes-proxy --requestheader-group-headers=X-Remote-Group --watch-cache=false --authorization-mode=Node,RBAC --tls-private-key-file=/var/lib/rancher/k3s/server/tls/localhost.key --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --secure-port=6444 --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/token-node-1.crt
E0523 10:41:15.370664   47043 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.370946   47043 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.370990   47043 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.371025   47043 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.371056   47043 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.371083   47043 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
W0523 10:41:15.419277   47043 genericapiserver.go:315] Skipping API batch/v2alpha1 because it has no resources.
W0523 10:41:15.426893   47043 genericapiserver.go:315] Skipping API node.k8s.io/v1alpha1 because it has no resources.
E0523 10:41:15.447519   47043 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447559   47043 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447605   47043 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447680   47043 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447715   47043 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0523 10:41:15.447743   47043 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
INFO[2019-05-23T10:41:15.451764805-07:00] Running kube-scheduler --port=10251 --bind-address=127.0.0.1 --secure-port=0 --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --leader-elect=false
INFO[2019-05-23T10:41:15.452138189-07:00] Running kube-controller-manager --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --cluster-cidr=10.42.0.0/16 --root-ca-file=/var/lib/rancher/k3s/server/tls/token-ca.crt --leader-elect=false --allocate-node-cidrs=true --port=10252 --bind-address=127.0.0.1 --secure-port=0
W0523 10:41:15.464384   47043 authorization.go:47] Authorization is disabled
W0523 10:41:15.464410   47043 authentication.go:55] Authentication is disabled
INFO[2019-05-23T10:41:15.531532252-07:00] Listening on :6443
E0523 10:41:15.538233   47043 controller.go:148] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
INFO[2019-05-23T10:41:15.632791038-07:00] Node token is available at /var/lib/rancher/k3s/server/node-token
INFO[2019-05-23T10:41:15.632821655-07:00] To join node to cluster: k3s agent -s https://10.103.102.48:6443 -t ${NODE_TOKEN}
INFO[2019-05-23T10:41:15.633755733-07:00] Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.64.0.tgz
INFO[2019-05-23T10:41:15.634040227-07:00] Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml
INFO[2019-05-23T10:41:15.634192136-07:00] Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml
INFO[2019-05-23T10:41:15.681774803-07:00] Wrote kubeconfig /etc/rancher/k3s/k3s.yaml
INFO[2019-05-23T10:41:15.681799980-07:00] Run: k3s kubectl
INFO[2019-05-23T10:41:15.681807644-07:00] k3s is up and running
INFO[2019-05-23T10:41:15.734059148-07:00] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log
INFO[2019-05-23T10:41:15.734193744-07:00] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
INFO[2019-05-23T10:41:15.734413260-07:00] Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory"
INFO[2019-05-23T10:41:16.736592830-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
W0523 10:41:17.678890   47043 controllermanager.go:445] Skipping "csrsigning"
INFO[2019-05-23T10:41:17.737374016-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:18.738180671-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:19.739156698-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:20.740059356-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:21.740946864-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:22.741856234-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-23T10:41:23.742698558-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
^CINFO[2019-05-23T10:41:23.998087128-07:00] Received SIGTERM, cancelling
FATA[2019-05-23T10:41:23.998308827-07:00] context canceled

warmchang · 2019-05-24T00:58:27Z

Because of SELinux?

rako1980 · 2019-05-24T15:30:19Z

No SELinux:

# getenforce
Disabled

rako1980 · 2019-05-24T15:40:19Z

I was able to make some progress with docker though.

#./k3s server --docker

But, needed to start the docker native.cgroupdriver with cgroupfs instead of systemd:

# cat /etc/systemd/system/multi-user.target.wants/docker.service | grep native.cgroupdriver
          --exec-opt native.cgroupdriver=cgroupfs \

However, my intention is to use the containerd instead of docker. I still assume there is some issue with the spawned contained running from k3s.

Here is the details k3s stdout and the corresponding containerd.log:

]# ./k3s server
INFO[2019-05-24T08:36:49.648912110-07:00] Starting k3s v0.5.0 (8c0116dd)
INFO[2019-05-24T08:36:49.650162620-07:00] Running kube-apiserver --advertise-port=6445 --api-audiences=unknown --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/token-node-1.crt --requestheader-group-headers=X-Remote-Group --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --service-cluster-ip-range=10.43.0.0/16 --bind-address=127.0.0.1 --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-allowed-names=kubernetes-proxy --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --authorization-mode=Node,RBAC --advertise-address=127.0.0.1 --insecure-port=0 --tls-cert-file=/var/lib/rancher/k3s/server/tls/localhost.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/localhost.key --service-account-issuer=k3s --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --requestheader-username-headers=X-Remote-User --watch-cache=false --allow-privileged=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --secure-port=6444 --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --kubelet-client-key=/var/lib/rancher/k3s/server/tls/token-node.key
E0524 08:36:49.654488   79239 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657296   79239 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657381   79239 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657421   79239 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657454   79239 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.657484   79239 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
W0524 08:36:49.702595   79239 genericapiserver.go:315] Skipping API batch/v2alpha1 because it has no resources.
W0524 08:36:49.724691   79239 genericapiserver.go:315] Skipping API node.k8s.io/v1alpha1 because it has no resources.
E0524 08:36:49.756678   79239 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756724   79239 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756779   79239 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756833   79239 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756869   79239 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0524 08:36:49.756899   79239 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
INFO[2019-05-24T08:36:49.761231703-07:00] Running kube-scheduler --leader-elect=false --port=10251 --bind-address=127.0.0.1 --secure-port=0 --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml
INFO[2019-05-24T08:36:49.761597686-07:00] Running kube-controller-manager --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --cluster-cidr=10.42.0.0/16 --root-ca-file=/var/lib/rancher/k3s/server/tls/token-ca.crt --leader-elect=false --allocate-node-cidrs=true --port=10252 --bind-address=127.0.0.1 --secure-port=0
W0524 08:36:49.776390   79239 authorization.go:47] Authorization is disabled
W0524 08:36:49.776420   79239 authentication.go:55] Authentication is disabled
INFO[2019-05-24T08:36:49.837533400-07:00] Listening on :6443
E0524 08:36:49.838660   79239 controller.go:148] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
INFO[2019-05-24T08:36:49.939031633-07:00] Node token is available at /var/lib/rancher/k3s/server/node-token
INFO[2019-05-24T08:36:49.939066279-07:00] To join node to cluster: k3s agent -s https://10.103.102.48:6443 -t ${NODE_TOKEN}
INFO[2019-05-24T08:36:49.943090685-07:00] Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.64.0.tgz
INFO[2019-05-24T08:36:49.943421808-07:00] Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml
INFO[2019-05-24T08:36:49.944462948-07:00] Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml
INFO[2019-05-24T08:36:50.004464688-07:00] Wrote kubeconfig /etc/rancher/k3s/k3s.yaml
INFO[2019-05-24T08:36:50.004506548-07:00] Run: k3s kubectl
INFO[2019-05-24T08:36:50.004523123-07:00] k3s is up and running
INFO[2019-05-24T08:36:50.064578703-07:00] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log
INFO[2019-05-24T08:36:50.064952262-07:00] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
INFO[2019-05-24T08:36:50.065173135-07:00] Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory"
INFO[2019-05-24T08:36:51.066282525-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:52.067114633-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:53.068189094-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:54.069177923-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:55.070198945-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:56.071389818-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:57.072397121-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:58.075790324-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:36:59.077008601-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:00.077919013-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:01.079128839-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:02.080065210-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
W0524 08:37:02.280312   79239 controllermanager.go:445] Skipping "root-ca-cert-publisher"
W0524 08:37:02.282988   79239 controllermanager.go:445] Skipping "csrsigning"
W0524 08:37:02.434796   79239 shared_informer.go:312] resyncPeriod 49515231569317 is smaller than resyncCheckPeriod 75729967585759 and the informer has already started. Changing it to 75729967585759
W0524 08:37:02.435431   79239 shared_informer.go:312] resyncPeriod 55270902515792 is smaller than resyncCheckPeriod 75729967585759 and the informer has already started. Changing it to 75729967585759
E0524 08:37:02.435703   79239 resource_quota_controller.go:171] initial monitor sync has error: [couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=listenerconfigs": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=listenerconfigs", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=addons": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=addons", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=helmcharts": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=helmcharts"]
INFO[2019-05-24T08:37:03.080919525-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:04.081925505-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:05.083064795-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:06.084115511-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
INFO[2019-05-24T08:37:07.085146046-07:00] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
^CINFO[2019-05-24T08:37:07.425870802-07:00] Received SIGTERM, cancelling
FATA[2019-05-24T08:37:07.426526098-07:00] context canceled

/var/lib/rancher/k3s/agent/containerd/containerd.log

time="2019-05-24T08:36:50.131372621-07:00" level=info msg="starting containerd" revision= version=1.2.5+unknown
time="2019-05-24T08:36:50.132169525-07:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2019-05-24T08:36:50.132221714-07:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2019-05-24T08:36:50.132267099-07:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2019-05-24T08:36:50.132487035-07:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.overlayfs" error="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
time="2019-05-24T08:36:50.132524711-07:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2019-05-24T08:36:50.132548844-07:00" level=warning msg="could not use snapshotter overlayfs in metadata plugin" error="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
time="2019-05-24T08:36:50.132842127-07:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2019-05-24T08:36:50.132953494-07:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2019-05-24T08:36:50.133036536-07:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133082568-07:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133120453-07:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133153366-07:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133172447-07:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133193487-07:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133217195-07:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133238861-07:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2019-05-24T08:36:50.133300940-07:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2019-05-24T08:36:50.133361641-07:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2019-05-24T08:36:50.133891262-07:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2019-05-24T08:36:50.133946642-07:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2019-05-24T08:36:50.134023974-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134082887-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134114423-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134151760-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134175442-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134197101-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134216812-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134233085-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134252815-07:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2019-05-24T08:36:50.134336176-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134363697-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134380869-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134399399-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.134528786-07:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: Root: Options:<nil>} UntrustedWorkloadRuntime:{Type: Engine: Root: Options:<nil>} Runtimes:map[] NoPivot:false} CniConfig:{NetworkPluginBinDir:/var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d/bin NetworkPluginConfDir:/var/lib/rancher/k3s/agent/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Auths:map[]} StreamServerAddress:d53icpdev001t0x StreamServerPort:10010 EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false} ContainerdRootDir:/var/lib/rancher/k3s/agent/containerd ContainerdEndpoint:/run/k3s/containerd/containerd.sock RootDir:/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri StateDir:/run/k3s/containerd/io.containerd.grpc.v1.cri}"
time="2019-05-24T08:36:50.134599727-07:00" level=info msg="Connect containerd service"
time="2019-05-24T08:36:50.134834052-07:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to find snapshotter \"overlayfs\""
time="2019-05-24T08:36:50.134863257-07:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2019-05-24T08:36:50.135193569-07:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock
time="2019-05-24T08:36:50.135223048-07:00" level=info msg="containerd successfully booted in 0.004611s"

aucampia · 2019-05-27T20:55:36Z

AFAICT this is caused by


WARN[2019-05-27T22:53:37.502861403+02:00] could not use snapshotter overlayfs in metadata plugin  error="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"

aucampia · 2019-05-28T05:05:00Z

I had this issue and I solved it by using a xfs filesystem created with:

mkfs -t xfs -n ftype=1 -L LABEL -f DEVICE

Some details here: https://docs.docker.com/storage/storagedriver/overlayfs-driver/

rako1980 · 2019-05-28T14:36:39Z

Thanks a bunch @aucampia . That is the the issue. Later versions of RHEL7 mkfs the new filesystem with default fstype=1. However, I was using the old rhel7 version from template where /var was already formatted. Had to create a new xfs filesystem /var/lib/rancher where the new ftype setting is automatically formatted with fstype 1, and there is no longer overlayfs issue with containerd. My coredns and traefik pods and now running :)

jeroavf · 2019-08-15T13:44:00Z

I had this issue and I solved it by using a xfs filesystem created with:
mkfs -t xfs -n ftype=1 -L LABEL -f DEVICE
Some details here: https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Thanks , it solved my problem also. I'm running K3S on a Oracle Linux Server release 7.5 VM and now it is functioning with containerd.

xarses · 2019-11-25T08:36:11Z

I was also eaten by this trying to run on a much older file-system

brandond · 2020-12-05T00:50:41Z

Closing due to age

rako1980 changed the title ~~rhel7 k3s does not start - failed to find snapshotter overlayfs~~ rhel7 k3s start not working - failed to find snapshotter overlayfs May 24, 2019

rako1980 changed the title ~~rhel7 k3s start not working - failed to find snapshotter overlayfs~~ rhel7 k3s start not working with containerd - failed to find snapshotter overlayfs May 24, 2019

xarses mentioned this issue Nov 25, 2019

cannot run k3s server #60

Closed

davidnuzik added the os/rhel label Nov 26, 2019

cfontes mentioned this issue Jan 2, 2020

[Feature] Create/Restore Cluster Snapshots k3d-io/k3d#160

Open

ushell mentioned this issue Feb 23, 2020

k3s agent start report ＂unknown service runtime.v1alpha2.RuntimeService＂ #1443

Closed

This was referenced Apr 29, 2020

Centos 7 Containerd mismatch with k3s #1508

Closed

Cannot remount block device PersistentVolumes on k3s v1.18 #1682

Closed

ktsakalozos mentioned this issue Sep 16, 2020

Facing microk8s is not running issue.No resources found in default namespace showing for microk8s.kubectl get nodes canonical/microk8s#1568

Closed

brandond closed this as completed Dec 5, 2020

jiayihu mentioned this issue Dec 27, 2020

Mounting containerd filesystem on k3s Alpine jiayihu/fedra-thesis#37

Closed

cho-chem mentioned this issue May 21, 2024

使用secretflow-allinone进行p2p模式部署发生问题 secretflow/secretflow#1287

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rhel7 k3s start not working with containerd - failed to find snapshotter overlayfs #495

rhel7 k3s start not working with containerd - failed to find snapshotter overlayfs #495

rako1980 commented May 23, 2019 •

edited

Loading

erikwilson commented May 23, 2019

rako1980 commented May 23, 2019

rako1980 commented May 23, 2019

warmchang commented May 24, 2019

rako1980 commented May 24, 2019 •

edited

Loading

rako1980 commented May 24, 2019 •

edited

Loading

aucampia commented May 27, 2019 •

edited

Loading

aucampia commented May 28, 2019

rako1980 commented May 28, 2019

jeroavf commented Aug 15, 2019

xarses commented Nov 25, 2019

brandond commented Dec 5, 2020

rhel7 k3s start not working with containerd - failed to find snapshotter overlayfs #495

rhel7 k3s start not working with containerd - failed to find snapshotter overlayfs #495

Comments

rako1980 commented May 23, 2019 • edited Loading

erikwilson commented May 23, 2019

rako1980 commented May 23, 2019

rako1980 commented May 23, 2019

warmchang commented May 24, 2019

rako1980 commented May 24, 2019 • edited Loading

rako1980 commented May 24, 2019 • edited Loading

aucampia commented May 27, 2019 • edited Loading

aucampia commented May 28, 2019

rako1980 commented May 28, 2019

jeroavf commented Aug 15, 2019

xarses commented Nov 25, 2019

brandond commented Dec 5, 2020

rako1980 commented May 23, 2019 •

edited

Loading

rako1980 commented May 24, 2019 •

edited

Loading

rako1980 commented May 24, 2019 •

edited

Loading

aucampia commented May 27, 2019 •

edited

Loading