Dockerhub creds for image pulls

[Christop1964]

What happened?

We always reach the docker rate limit (100 / 6) when we deploy Kubernetes with kubespray.
Not sure which default id get used when pull images from docker.
We are looking for a possibility to use a private dockerhub account to pull Images from docker.

We are using the kubspray ansible playbook

The solution from issue #6883 is not working anymore because containered_config is depricated.

What did you expect to happen?

That the deployment of kubernetes with a private dockerhub account does not fail when images get pulled from docker.

How can we reproduce it (as minimally and precisely as possible)?

We looking for a possibility to use a private dockerhub account to pull Images from docker

OS

printf "$(uname -srm)\n$(cat /etc/os-release)"

Linux 4.18.0-552.el8_10.x86_64 x86_64

NAME="Red Hat Enterprise Linux"
VERSION="8.10 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.10 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.10
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.10"[root@kubetest-ocp-host ~]#

Version of Ansible

ansible --version

ansible [core 2.16.7]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.12/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.12.1 (main, Feb 21 2024, 10:25:11) [GCC 8.5.0 20210514 (Red Hat 8.5.0-21)] (/usr/bin/python3)
jinja version = 3.1.4
libyaml = True

Version of Python

python3 --version Python 3.12.1

Version of Kubespray (commit)

README.md: - kubernetes v1.29.5

Network plugin used

calico

Full inventory with variables

cat inventory/mycluster/inventory.yaml

all:
hosts:
kube-32:
ansible_host: 172.16.3.92
ip: 172.16.3.92
access_ip: 172.16.3.92
kube-33:
ansible_host: 172.16.3.93
ip: 172.16.3.93
access_ip: 172.16.3.93
kube-34:
ansible_host: 172.16.3.94
ip: 172.16.3.94
access_ip: 172.16.3.94
kube-35:
ansible_host: 172.16.3.95
ip: 172.16.3.95
access_ip: 172.16.3.95
kube-36:
ansible_host: 172.16.3.96
ip: 172.16.3.96
access_ip: 172.16.3.96
children:
kube_control_plane:
hosts:
kube-32:
kube-33:
kube_node:
hosts:
kube-32:
kube-33:
kube-34:
kube-35:
kube-36:
etcd:
hosts:
kube-32:
kube-33:
kube-34:
k8s_cluster:
children:
kube_control_plane:
kube_node:
calico_rr:
hosts: {}

Command used to invoke ansible

ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml

Output of ansible run

2024-06-26 14:08:17,320: [I] TASK [download : debug] ********************************************************
2024-06-26 14:08:17,320: [I] ok: [kube-32] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:17,320: [I] ok: [kube-33] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:17,320: [I] ok: [kube-34] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:17,320: [I] ok: [kube-35] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:17,320: [I] ok: [kube-36] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:52,456: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (4 retries left).
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (4 retries left).
2024-06-26 14:08:52,457: [I]
2024-06-26 14:08:52,457: [I] TASK [download : Download_container | Download image if required] **************
2024-06-26 14:08:52,457: [I] changed: [kube-33]
2024-06-26 14:08:52,457: [I] changed: [kube-32]
2024-06-26 14:08:52,457: [I] changed: [kube-36]
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (3 retries left).
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (2 retries left).
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (3 retries left).
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (1 retries left).
2024-06-26 14:08:52,457: [I] fatal: [kube-35]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "--hosts-dir", "/etc/containerd/certs.d", "docker.io/library/nginx:1.25.2-alpine"], "delta": "0:00:00.803006", "end": "2024-06-26 14:08:36.020879", "msg": "non-zero return code", "rc": 1, "start": "2024-06-26 14:08:35.217873", "stderr": "time="2024-06-26T14:08:35+02:00" level=warning msg="DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead."\nctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit", "stderr_lines": ["time="2024-06-26T14:08:35+02:00" level=warning msg="DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead."", "ctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"], "stdout": "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.1 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.2 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.3 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.4 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.5 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.6 s total: 0.0 B (0.0 B/s) ", "stdout_lines": ["docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.1 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.2 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.3 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.4 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.5 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.6 s total: 0.0 B (0.0 B/s) "]}
2024-06-26 14:08:52,458: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (2 retries left).
2024-06-26 14:08:52,458: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (1 retries left).
2024-06-26 14:08:52,459: [I] fatal: [kube-34]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "--hosts-dir", "/etc/containerd/certs.d", "docker.io/library/nginx:1.25.2-alpine"], "delta": "0:00:00.867968", "end": "2024-06-26 14:08:52.421105", "msg": "non-zero return code", "rc": 1, "start": "2024-06-26 14:08:51.553137", "stderr": "time="2024-06-26T14:08:51+02:00" level=warning msg="DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead."\nctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit", "stderr_lines": ["time="2024-06-26T14:08:51+02:00" level=warning msg="DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead."", "ctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"], "stdout": "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.1 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.2 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.3 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.4 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.5 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.6 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.7 s total: 0.0 B (0.0 B/s) ", "stdout_lines": ["docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.1 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.2 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.3 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.4 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.5 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.6 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.7 s total: 0.0 B (0.0 B/s) "]}
2024-06-26 14:08:52,460: [I]
2024-06-26 14:08:52,460: [I] NO MORE HOSTS LEFT *************************************************************
2024-06-26 14:08:52,460: [I]
2024-06-26 14:08:52,460: [I] PLAY RECAP *********************************************************************
2024-06-26 14:08:52,460: [I] kube-32 : ok=354 changed=58 unreachable=0 failed=0 skipped=417 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I] kube-33 : ok=324 changed=58 unreachable=0 failed=0 skipped=361 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I] kube-34 : ok=308 changed=55 unreachable=0 failed=1 skipped=358 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I] kube-35 : ok=297 changed=53 unreachable=0 failed=1 skipped=358 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I] kube-36 : ok=298 changed=54 unreachable=0 failed=0 skipped=358 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I]
2024-06-26 14:08:53,599: [E] Command finished with rc = 2

Anything else we need to know

No response

[tico88612]

Maybe you need to set containerd_registry_auth in inventory/<YOUR_INVENTORY>/group_vars/all/containerd.yml?

kubespray/inventory/sample/group_vars/all/containerd.yml

Lines 43 to 46 in 9a69221

	# containerd_registry_auth:
	# - registry: 10.0.0.2:5000
	# username: user
	# password: pass

[kosalaat]

Faced the same issue. Problem seem to be there is no way to pass --user with the ctr command nor ctr does not seem to utilize the config.toml from containerd.

After configuring containerd to utilise Docker creds ctr still fails:

# /usr/local/bin/ctr -n k8s.io images pull --hosts-dir /etc/containerd/certs.d docker.io/library/alpine:latest
docker.io/library/alpine:latest:                                               resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d: downloading    |--------------------------------------|    0.0 B/1.8 KiB
elapsed: 7.0 s                                                                 total:   0.0 B (0.0 B/s)
ctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/alpine/manifests/sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

but crictl seem to works fine:

# /usr/local/bin/crictl pull docker.io/library/alpine:latest
Image is up to date for sha256:91ef0af61f39ece4d6710e465df5ed6ca12112358344fd51ae6a3b886634148b

AFA I can see, ctr comes from {{nerdctl_image_pull_command}} which sets to ctr pull command. Is there any particular reason why this is set this way?

Why not use crictl?

[FilipeNas]

same problem here

[tico88612]

#10907
#11001

[kosalaat]

Both above pull requests seem to have not gone through. But this worked for me. I set my docker.io creds with containerd_registry_auth:

containerd_registry_auth:
  - registry: registry-1.docker.io
    username: Username
    password: P@ssw0rd

But stil ctr wil not work with the containerd registry creds, but crictl works. Under group_vars/all/ I added nerdctl_image_pull_command: "/usr/local/bin/crictl images pull" which overrides ctr to crictl.

[FilipeNas]

Thanks for the insight @kosalaat !

Why dont we just set the image pull command to use crictl?
edit: the following var replacement worked.

image_command_tool: "crictl"

Then we can even delete nerdctl_image_pull_command.

Is there any requirement of using ctr / nerdctl for downloading the images?
I feel it would be better to just use one tool to manage the images in ansible.

We can also provide the other container tools like (ctr or nerdctl) for the user to manage and tshoot inside the machines, but for ansible why not go with just one tool?

Thanks

[kosalaat]

image_command_tool is CRI dependant, and crictl is not an universal option.

#10670 current defaults were set through this issue.

This was before DockerHub introduced rate limitations and I think it's time we switch to crictl from ctr.

[FilipeNas]

from another thread, it seems ctr is used to dev and debug containerd:

@dee-kryvenko with ctr you need to pass in the credentials, it will not use any existing configuration that resulted from docker login or configuring the cri plugin. nerdctl is the preferred tool for just using containerd as an end user, ctr is intended for developing/debugging containerd.

Originally posted by @dmcgowan in #2628

I am having difficulties using ctr with JFrog Artifactory. (With error: failed to authorize: failed to fetch anonymous token: unexpected status from GET request)
My hosts.toml file (directly typed the header configuration and restarted containerd, just to test):

What do you mean by ´crictl´ is not an universal option? Isn't ´crictl´ designed specifically for CRI-compatible container runtimes?

And since ctr is used for debug and dev, shouldn't we use an agnostic approach like ´crictl´.

Thanks :))

[kosalaat]

crictl works well for containerd and cri-o, but not sure about dockerd.

[VannTen]

What do you mean by ´crictl´ is not an universal option? Isn't ´crictl´ designed specifically for CRI-compatible container runtimes?

The CRI does not support save/load images , which is used by kubespray for out-of-band image provisionning (when using download_once / download_delegate)