Cri-o is unable to pull Arm64 Calico node:v3.16.4 image.

[anthr76]

Environment:

• Cloud provider or hardware configuration: Arm64 Raspberry Pis Bare Metal/Mixed Architecture

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): 20.04.1 LTS (Focal Fossa) 5.4.0-1015-raspi aarch64

• Version of Ansible (ansible --version): 2.9.15

• Version of Python (python --version): 3.9.0

Kubespray version (commit) (git rev-parse --short HEAD): 0f7341b

Network plugin used: Calico

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"): https://gist.github.com/anthr76/840fb231e60b494d6a599d8131049210

Command used to invoke ansible: ansible-playbook -i inventory/nwk1/inventory.ini --become --become-user=root cluster.yml

Output of ansible run: https://gist.github.com/anthr76/1ce5698dcfc97055fc2ddd43f260ad10

Anything else do we need to know:

Not sure what exactly is going on here as it appears it should be pulling this image. Which show's arm64 support.

[anthr76]

FWIW:

It looks like pulling the manifest from quay is successful.

root@master-01:~# crictl pull docker.io/calico/node:v3.16.4
FATA[0000] pulling image: rpc error: code = Unknown desc = Error choosing image instance: no image found in manifest list for architecture arm64, variant v8, OS linux 
root@master-01:~# crictl pull calico/node:v3.16.4
Image is up to date for quay.io/calico/node@sha256:23072129e5e6926ba90388cfd4f2a3e61447d8383c962be9ca3c1869eb5210d7

[anthr76]

var calico_node_image_repo: "{{ quay_image_repo }}/calico/node" Resolves this issue then fails on docker.io/calico/cni this seems like a cri-o or dockerhub issue.

[anthr76]

Nevertheless would it be desirable to swap the calico images to quay instead of dockerhub considering the newly imposed rate limits by dockerhub?

[anthr76]

I resolved this for taking the latter and running off the master branch instead of a versioned tag. My original environment on was on 75d648c manually setting K8s version 1.19.3 on this version cri-o version was defaulted to 1.17 causing me to run into this bug cri-o/cri-o#3233