Centos 7 Containerd mismatch with k3s

[MikeCongdon1]

Version:
k3s version v1.17.3+k3s1 (5b17a17)
No flags used to install or run.

Centos 7
lsb_release -d
Description: CentOS Linux release 7.7.1908 (Core)
docker version:
Docker version 19.03.7, build 7141c199a2
containerd -v
containerd containerd.io 1.2.13 7ad184331fa3e55e52b890ea95e65ba581ae3429

Describe the bug
I cannot for the life of me figure out a workaround on this:
I ran the default install script and it appeared to install correctly, but I never got any nodes in my namespace.
So, systemctl stop k3s
k3s server
It won't get past here: (I'm on AWS here)

I0307 21:06:59.747315    5642 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
W0307 21:06:59.747377    5642 handler_proxy.go:97] no RequestInfo found in the context
E0307 21:06:59.747420    5642 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0307 21:06:59.747433    5642 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
INFO[2020-03-07T21:08:14.187827884Z] Waiting for master node ip-MYIP.compute.internal startup: nodes "ip-MYIP.compute.internal" not found 
E0307 21:08:14.851628    5642 scheduler.go:639] error selecting node for pod: no nodes available to schedule pods
E0307 21:08:14.851740    5642 scheduler.go:639] error selecting node for pod: no nodes available to schedule pods
E0307 21:08:14.851912    5642 scheduler.go:639] error selecting node for pod: no nodes available to schedule pods
E0307 21:08:14.852031    5642 scheduler.go:639] error selecting node for pod: no nodes available to schedule pods
INFO[2020-03-07T21:08:15.117920302Z] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService 
INFO[2020-03-07T21:08:13.185056136Z] Waiting for master node ip-MYIP1.compute.internal startup: nodes "ip-MYIP.compute.internal" not found 
INFO[2020-03-07T21:08:14.116878755Z] Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService 

(that last part repeats when starting up k3s server.

This might be a containerd issue ultimately though. I try to run containerd and get:

WARN[2020-03-07T21:13:53.687831694Z] could not use snapshotter overlayfs in metadata plugin  error="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"

Is there a way to not have to reformat?

To Reproduce
install k3s on centos 7 with
docker version:
Docker version 19.03.7, build 7141c199a2
containerd -v
containerd containerd.io 1.2.13 7ad184331fa3e55e52b890ea95e65ba581ae3429

run:
k3s server
and you'll get the above rpc error after about 20s of seemingly beneficial startup.

Expected behavior

I expect containerd & k3s to speak the same language, or error handling to reroute the command safely

Actual behavior

k3s will appear to load, but never load Master Node.

Additional context

Centos 7 is a pain to deal with for this stuff, but it's required by my sysadmins & security folks...
Also, this particular issue is a legacy deployment, so I don't think reformatting with fs_type=1 is an option for me.

Are there any known workarounds here, or am I stuck?
Thanks!

[brandond]

If the underlying fs is indeed the problem, you could always attach another ebs volume, format it correctly, and use that to host the containerd overlayfs mount.

[erikwilson]

Related: #495 (comment) ?

[chrisbecke]

In the case that /var is a derived from an image as a legacy filesystem, where would a d_type=1 filesystem be mounted for containerd to use? /var/lib/rancher?

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.