Trying to download gcr.io/google-containers/pause:3.3 which does not exist

[Kaylleur]

Hi,

I have a kubernetes cluster on 3 servers that are in version 1.18.0. I'm trying to upgrade to 1.19.7 them with kubespray in version 2.17.0.
I already upgraded from 1.15.11 to 1.16, 1.16 to 1.17 and 1.17 to 1.18 and it works perfectly.

When I upgrade them it try to download : gcr.io/google-containers/pause:3.3
This tag of this container doesn't exist in repo
and doesn't exist on my server.

I check the command executed by kubespray and I can found this container on tag 3.1 or 3.2 but not on 3.3 (it seems normal).

Environment:

• Cloud provider or hardware configuration:
  3 Virtual servers 2 CPU, 2 Go ram, 80 Go storage

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  All servers are under

Linux 5.4.0-88-generic x86_64
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"

• Version of Ansible (ansible --version):

ansible 2.10.11
  config file = /home/thomas/Projects/kubespray/ansible.cfg
  configured module search path = ['/home/thomas/Projects/kubespray/library']
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0]

• Version of Python (python --version):
  Python 3.8.10

Kubespray version (commit) (git rev-parse --short HEAD):
tag : v2.17.0 or b83e8b02

Network plugin used:
Connection direct to the server with ssh

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

All host are configured in my ssh config

[all]
vizima-1
vizima-2
vizima-3

[kube-master]
vizima-1
vizima-2
vizima-3

[etcd]
vizima-1
vizima-2
vizima-3

[kube-node]
vizima-1
vizima-2
vizima-3

[calico-rr]

[k8s-cluster:children]
kube-master
kube-node
calico-rr

Command used to invoke ansible:
ansible-playbook -i ../infra/inventory/vizima/inventoy.ini cluster.yml -b -v -e kube_version=v1.19.7 -e upgrade_cluster_setup=true
or
ansible-playbook -i ../infra/inventory/vizima/inventoy.ini upgrade-cluster.yml -b -v -e kube_version=v1.19.7

Output of ansible run:

fatal: [vizima-3 -> vizima-3]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/bin/docker", "pull", "gcr.io/google-containers/pause:3.3"], "delta": "0:00:00.825759", "end": "2021-10-18 17:51:43.927477", "msg": "non-zero return code", "rc": 1, "start": "2021-10-18 17:51:43.101718", "stderr": "Error response from daemon: manifest for gcr.io/google-containers/pause:3.3 not found: manifest unknown: Failed to fetch \"3.3\" from request \"/v2/google-containers/pause/manifests/3.3\".", "stderr_lines": ["Error response from daemon: manifest for gcr.io/google-containers/pause:3.3 not found: manifest unknown: Failed to fetch \"3.3\" from request \"/v2/google-containers/pause/manifests/3.3\"."], "stdout": "", "stdout_lines": []}

[oomichi]

The pause version has been updated to current 3.3 with #6660 which pull request is 1 year ago.
In addition, the image should be pulled from k8s.gcr.io, not gcr.io.
We should be able to pull the image like

$ sudo docker pull k8s.gcr.io/pause:3.3
3.3: Pulling from pause
aeab776c4837: Pull complete 
Digest: sha256:a319ac2280eb7e3a59e252e54b76327cb4a33cf8389053b0d78277f22bbca2fa
Status: Downloaded newer image for k8s.gcr.io/pause:3.3
k8s.gcr.io/pause:3.3

[Kaylleur]

I pulled manualy k8s.gcr.io/pause:3.3 but still block on the old name of the image. So i tagged it into gcr.io/google-containers/pause:3.3. It works for this image but now it attempt to download gcr.io/google-containers/coredns/coredns:v1.8.0...
The image is available on k8s.gcr.io/coredns/coredns:v1.8.0... Looking for this issue is should be able on gcr.io/google-containers...
But I still don't know why Kubespray is trying to upgrade these versions, whose are not on my cluster...

[floryut]

I pulled manualy k8s.gcr.io/pause:3.3 but still block on the old name of the image. So i tagged it into gcr.io/google-containers/pause:3.3. It works for this image but now it attempt to download gcr.io/google-containers/coredns/coredns:v1.8.0... The image is available on k8s.gcr.io/coredns/coredns:v1.8.0... Looking for this issue is should be able on gcr.io/google-containers... But I still don't know why Kubespray is trying to upgrade these versions, whose are not on my cluster...

Looks like your kube_image_repo is set with the wrong path; if all images are pulled from gcr.io instead of k8s.gcr.io

[Kaylleur]

My cluster was in version v1.15.11 before I started to upgrade, my version was too old ?
Do you know where I can fix this path ?

[floryut]

My cluster was in version v1.15.11 before I started to upgrade, my version was too old ? Do you know where I can fix this path ?

The path is here

kubespray/roles/download/defaults/main.yml

Line 61 in ea8e2fc

kube_image_repo: "k8s.gcr.io"

You may have change it in your variables though

[Kaylleur]

Here is my yaml :

# gcr and kubernetes image repo define
gcr_image_repo: "gcr.io"
kube_image_repo: "k8s.gcr.io"

I do not modify the yaml of kubespray...

[Kaylleur]

On the previous job in kubespray it list image used on my cluster and I got previous version image like gcr.io/google-containers/pause:3.2. Like it is training to maintain theses images

[Kaylleur]

The problem was about my group-vars was set with the 1.15.11 version and not set in 1.19. I changed in group_vars/k8s-cluster/k8s-cluster.yml line 235 kube_image_repo: "gcr.io/google-container" to kube_image_repo: "k8s.gcr.io".
Thanks to @Sryther who help me, and to all.