This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Plans for ZFS support? #66
Comments
|
We specifically removed the ZFS snapshotter. The reason being that we don't intend to include the ZFS user space as I believe that is not portable across kernel versions. So we can include the ZFS snapshotter and you would be required to first install the ZFS tools, which is common place already. |
|
Thanks for the quick feedback! The ZFS userspace tools are generally present when ZFS is used because otherwise ZFS cannot be used. So that’s really a non-issue as far as I can see. Except perhaps when running k3s in a Docker container on top of ZFS, but I’m not sure whether that would work anyway. |
|
Is there a timeline for including the ZFS snapshotter? |
|
ZFS is sadly quite painful with Docker in Docker and similar scenarios. It might be best to avoid the problem by creating a volume in your ZFS pool, formatting that volume to ext4, and having docker use "overlay2" on top of that, instead of "zfs". The I just finished setting this up and it nicely solves my problems with k3s and also kind. I use these for testing and development, and there the volume should be just fine. |
|
I was able to point k3s at the host's containerd (which had the zfs snapshotter and was configured to use it by default) and successfully run pods. $ k3s server --container-runtime-endpoint /run/containerd/containerd.sock
$ sudo k3s kubectl -n kube-system get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
local-path-provisioner-58fb86bdfd-tx9lt 1/1 Running 0 4m52s 10.88.0.137 rincon <none> <none>
coredns-57d8bbb86-7t6v6 1/1 Running 0 4m52s 10.88.0.136 rincon <none> <none>
$ zfs list -r vault/storage/containerd
NAME USED AVAIL REFER MOUNTPOINT
vault/storage/containerd 37.2M 466G 26K /var/lib/containerd/io.containerd.snapshotter.v1.zfs
vault/storage/containerd/1 450K 466G 450K legacy
vault/storage/containerd/10 25K 466G 20.0M legacy
vault/storage/containerd/3 15K 466G 451K legacy
vault/storage/containerd/4 15K 466G 451K legacy
vault/storage/containerd/5 238K 466G 238K legacy
vault/storage/containerd/6 3.67M 466G 3.67M legacy
vault/storage/containerd/7 19.8M 466G 20.0M legacy
vault/storage/containerd/8 13.0M 466G 16.6M legacy
vault/storage/containerd/9 35K 466G 16.6M legacy |
|
@terinjokes Hi, could you please share more details about containerd with zfs support. How can I install containerd with zfs support? |
|
Another possible workaround, again it involves creating ext4 filesystem, but this time using Docker itself through Docker loopback volume plugin, not by asking zfs to do it. Described in rancher/k3d docs https://github.com/rancher/k3d/blob/dc4c29361fd433d3592b62d3d8d23256a5ed5728/docs/examples.md#running-on-filesystems-k3s-doesnt-like-btrfs-tmpfs- |
|
Well you can apparently leverage the host's docker for k3s, which can spawn containers on a ZFS-based graph. This however completely defeats the purpose of packaging CRI in k3s. |
This is not necessary the case now for Proxmox-based distribution with version >= 5, and also now in Ubuntu 20.04 (albeit deemed "experimental"), both of which cherry-picks and integrates OpenZFS Linux module codebase into their kernel recipe. OpenZFS ABI is very stable but is denied entering kernel mainline due to some stupid license politics problem. And even if it is unstable, you can always use DKMS to hopefully keep it up with the new kernel. Despite this, I think k3s team still won't consider adding ZFS support anyway, but what about re-adding AUFS support? AUFS works generally fine atop of ZFS despite there are some reported edges cases related to file mounts not in sync, and AUFS is more general than ZFS and easier to merge in k3s I think. EDIT: aaaaaaaaaaaand btw the current setback is to create a ext4 volume as containerd backend like @jirkadanek did, but on a different location # echo "[+] making sure everything in containerd is empty to let it mount"
# rm -rf /var/lib/rancher/k3s/agent/containerd/*
# echo "[+] create the k3s hierarchy"
# zfs create rpool/k3s
# zfs create -s -V 128GB rpool/k3s/containerd
# echo "[+] wait for the newly created volume to format"
# mkfs.ext4 /dev/rpool/k3s/containerd
# echo "[+] adding the mount to fstab"
# echo "/dev/rpool/k3s/containerd /var/lib/rancher/k3s/agent/containerd ext4 auto 0 0" > /etc/fstab
# echo "[+] manually mounting the volume to the specific mount point"
# mount /dev/rpool/k3s/containerd /var/lib/rancher/k3s/agent/containerd
# echo "[!] if you saw this message without any error it means you're good to go"Keep in mind that this is still not perfect -- so you basically recreated snapshot on ZFS, and so the atomic COW goodies is probably gone, you have no fine control over the data size whatsoever, I think in the future maybe we could get to install k3s plugins to accommodate situations like these. ZFS is a very popular filesystem among Proxmox users after all, and it's a shame such a beautiful piece of work is being insulted like this. |
|
Again,again,again - when you return ZFS snapshotter to containerd (v1.19.5+k3s2) ? Now I must use external containerd from debian testing (Bullseye) with ZFS snapshotter (and calico CNI): |
|
I don't believe we've changed any of our plans around ZFS support, for the reasons @ibuildthecloud described in #66 (comment) |
|
For anyone else that's found this, https://blog.nobugware.com/post/2019/k3s-containterd-zfs/ gives some indication about using an external containerd with k3s. Personally, I plan on just setting up a zfs volume for |
👆 I use this as well for my single node k3s cluster and find it works great. |
|
@AstraLuma @mhumeSF Can you guys exolain how you did exposed the zpool to the docker container? I deployed k3d but when tried to deploy the zfs-localpv I was getting errors indicating that the docker containers serving as nodes for the k3d cannot work with zfs, cannot see the zpool and etc. I willbe very grateful if someone of you describe how you exposed the zpool to the k3d so that it can be used by the zfs-localpv! |
|
@lord-kyron I don't run k3d and I think k3s in docker with zfs support would be difficult see #66 (comment). I run k3s on a root volume that is ext4. |
I'm not at all. I'm running k3s directly on the host. |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Deployment on ZFS is currently not possible, because OverlayFS does not work with ZFS:
From the containerd logfile, it appears the ZFS snapshotter is not included:
ZFS support would be awesome! Are there any plans to include that?
The text was updated successfully, but these errors were encountered: