Large containerd Snapshot Directories and Cleanup Assistance

[modigithub]

I have been encountering an issue where the directories io.containerd.snapshotter.v1.overlayfs, io.containerd.snapshotter.v1.native and io.containerd.content.v1.content in /var/lib/containerd are occupying a large amount of space:

io.containerd.snapshotter.v1.overlayfs: 6.2GB
io.containerd.snapshotter.v1.native: 3.5GB
io.containerd.content.v1.content: 2.5GB
I have attempted to use crictl rmi --prune to clean up unneeded images, and I have tried creating a script to remove non-running containers, but the directories still remain large. I have also reviewed the containerd and Kubernetes documentation, but I'm still unclear about why these directories are so large and how to properly clean them. i also created an sh script to cleanup:

containers=$(crictl ps -a -q)

for container in $containers
do
container_state=$(crictl inspect $container | grep ""state"" | awk -F" '{print $4}')

if [[ "$container_state" != "CONTAINER_RUNNING" ]]; then
    echo "Removing container $container with state $container_state..."
    crictl rm $container
else
    echo "Container $container is running, skipping..."
fi

done

Questions:

1. Why are these directories (io.containerd.snapshotter.v1.overlayfs, io.containerd.snapshotter.v1.native and io.containerd.content.v1.content) so large?
2. How is the space in these directories allocated and used by containerd and Kubernetes?
3. Is it safe to manually delete contents in these directories? If not, why?
4. What is the recommended way to clean up these directories and free up space?
5. How does OverlayFS work in context of containerd and Kubernetes and why is it occupying so much space?
6. Is there any correlation between these large directories and Persistent Volumes (PVs) or Persistent Volume Claims (PVCs)?
   Is there any known issue in the specific version of containerd or Kubernetes that might be causing this?
   Environment:

• Kubernetes version:
  Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.7
  Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:29:58Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: Hetzner cloud server
• OS:
  Debian GNU/Linux 11 (bullseye)
  Kernel (Linux Server 5.10.0-17-amd64 Calico and host reboot issue #1 SMP Debian 5.10.136-1 (2022-08-13) x86_64 GNU/Linux):
• Install tools (e.g., kubespray):
• Version of Ansible: ansible [core 2.12.5]
• Version of Python: Python 3.9.2
• Kubespray version (commit) (git rev-parse --short HEAD): 31d7e64

[Vieufoux]

@modigithub Do you have an answer now? I had the same problem and had the same doubt

[modigithub]

Hi @Vieufoux,

No, I haven't found a solution yet. Moreover, I'm not even sure if kubespray is the appropriate place for this issue, as it seems to me that containerd is actually the root of the problem. Perhaps someone will have the time to answer my questions. I would really appreciate any insights or assistance from the community on this issue.

Kind regards,
@modigithub

[maksymsoc]

I actually have the same problem and I am looking for a solution. I found that you can change
/etc/kubernetes/kubelet/kubelet-config.json
Add or modify the following lines:
"image-gc-high-threshold": "75", "image-gc-low-threshold": "65"

Apply Changes: After saving the configuration, restart the kubelet service:
sudo systemctl restart kubelet

[VannTen]

io.containerd.snapshotter.v1.overlayfs: 6.2GB
io.containerd.snapshotter.v1.native: 3.5GB
io.containerd.content.v1.content: 2.5GB

This isn't that large ^.
Anyway, this is more of a containerd questions. See some links which could be helpful:
containerd/containerd#6295
https://stackoverflow.com/a/71908863
/close

[k8s-ci-robot]

@VannTen: Closing this issue.

In response to this:

io.containerd.snapshotter.v1.overlayfs: 6.2GB
io.containerd.snapshotter.v1.native: 3.5GB
io.containerd.content.v1.content: 2.5GB

This isn't that large ^.
Anyway, this is more of a containerd questions. See some links which could be helpful:
containerd/containerd#6295
https://stackoverflow.com/a/71908863
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.