cri-o 1.17.0 breaks ARM clusters

[everflux]

After updating crio to 1.17.0 my arm cluster got broken, the essential pause image could no longer be pulled, pods do not start after the upgrade since the images are cross-build on amd64 for ARM64 resulting in an image that is in fact usable on arm64 but cri-o refuses to try.

Trying to pull manually:

$ crictl pull k8s.gcr.io/pause-arm64:3.1
FATA[0001] pulling image failed: rpc error: code = Unknown desc = Image architecture mismatch: image uses "amd64", expecting "arm64" 

From the logfiles

Feb 08 17:10:16 n2-master0 kubelet[6717]: E0208 17:10:16.050473    6717 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "kube-apiserver-n2-master0_kube-system(c0554e8450d1ed33e8b37ef2bd7d6b60)" failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_kube-apiserver-n2-master0_kube-system_c0554e8450d1ed33e8b37ef2bd7d6b60_0": Image architecture mismatch: image uses "amd64", expecting "arm64"

The general problem was reported in this issue containers/podman#4849 a workaround for podman was implemented.

As long as crossplatform builds result in images tagged with the wrong architecture I see it as a bug in cri-o to be unable to use the images that worked in the past. afaik there is no easy fix for the bad tagging, especially with images build on dockerhub with limited options to influence the build process.

Current workaround: Downgrade to cri-o 1.16.1

[vrothberg]

We should update containers/image to v5.2.0 which relaxes the the arch checks that were added with v5.1.0.

[everflux]

Still the case with cri-o 1.17.3

Error choosing an image from manifest list docker://k8s.gcr.io/pause:3.1: no image found in manifest list for architecture arm64, variant v8, OS linux"

I am really disappointed about this happening, don't you have any tests in place for common use cases?

[vrothberg]

I am really disappointed about this happening, don't you have any tests in place for common use cases?

It's hard to test against broken images and without a doubt, the pause:3.1 is broken. In the meantime, it has been fixed and released with pause:3.2.

Can you set pause_image="k8s.gcr.io/pause:3.2" in the crio.conf?

[everflux]

I understand the argument that pause is broken - but if that is deemed broken there are many many images that are built that way and worked fine with cri-o 1.16.1 and stopped working with cri-o 1.17.0.

From my perspective as a user it feels like crio is a little ... arrogant to force the world to change the behavior just to please its model of a perfect world not taking into account the current state of affairs.
There should be at least a workaround or switch "use the image as stated, I know what I am doing" for the time until all images in the world are built properly.

Don't get me wrong, it was a conscious decision for me to use cri-o instead of another container runtime on the ARM cluster exactly to be able to provide feedback and help testing it out. But if it turns out that I am stuck with an old version of cri-o I won't be able to do that.

[vrothberg]

Note that this regression is unintentional.

From my perspective as a user it feels like crio is a little ... arrogant to force the world to change the behavior just to please its model of a perfect world not taking into account the current state of affairs.

We are between a rock and hard place since especially ARM users needed support to reliably select the correct image/manifest from a manifest list (or OCI index) with an extended support to select by variant. The error here is new.

I believe we can account for cases where the list/index claims a different arch/os than the image's manifest (and use these as a fallback when selecting the manifests). @mtrmac @nalind WDYT?

There should be at least a workaround or switch "use the image as stated, I know what I am doing" for the time until all images in the world are built properly.

You can do that by pulling by digest.

Can you set pause_image="k8s.gcr.io/pause:3.2" in the crio.conf?

Did you try that?

[everflux]

Note that this regression is unintentional.
Never suspected anything else :)

I believe we can account for cases where the list/index claims a different arch/os than the image's manifest (and use these as a fallback when selecting the manifests). @mtrmac @nalind WDYT?

That would be lovely!

Resorting to an image hash would be indeed a solution, thanks for pointing that out, though it is quite inconvenient.

Can you set pause_image="k8s.gcr.io/pause:3.2" in the crio.conf?

Did you try that?

Yes, that worked, for the pause image.
Weave Overlay network is failing with the same root cause, and a couple of other cross-built images, as suspected.

After about 10 minutes trying to (re)start pods crio crashes, but this could be totally unrelated and caused by the underlying hardware, but in case it is of any help:

       Stack trace of thread 4335:
      #0  0x000000000046a938 n/a (crio + 0x6a938)
      #1  0x0000000000451048 _start (crio + 0x51048)
      #2  0x000000000045164c _start (crio + 0x5164c)
      #3  0x0000000000450028 _start (crio + 0x50028)
     #4  0x000000000046aca8 n/a (crio + 0x6aca8)
     #5  0x0000ffff8e69856c n/a (linux-vdso.so.1 + 0x56c)

I will attach the stack trace from the go process.

crio-3968.log

I really do look forward to some sort of fallback solution when choosing the right container image like you drafted. Sounds like the best of both worlds. Thanks for taking care!

[nalind]

I believe we can account for cases where the list/index claims a different arch/os than the image's manifest (and use these as a fallback when selecting the manifests). @mtrmac @nalind WDYT?

In the cases where I've seen them disagree, the mainfest list was correct while the image's manifest was not. Looking at pause:3.1's manifest, it contains an "arm64" entry, but that entry doesn't include a "variant" value, so I think the matching logic didn't accept the entry as a match for what it wanted.

[vrothberg]

@nalind, the "arm64" entry points to a manifest which claims to be for "amd64". The variant selection code will also match if the variant is empty.

[nalind]

The error message in #3233 (comment) looks like it's coming from github.com/containers/image/manifest.Schema2List.ChooseInstance().
If I'm reading MatchesPlatform() right, if the passed-in wanted object includes a variant value, and the candidate entry from the manifest list doesn't include a variant value, it won't match.
If I'm reading WantedPlatforms() right, the default on an arm or arm64 system is to attempt to include a "variant" value in the criteria that it returns.

[mtrmac]

From my perspective as a user it feels like crio is a little ... arrogant to force the world to change the behavior just to please its model of a perfect world not taking into account the current state of affairs.

There was never any conscious decision to break images, even if for some bigger benefit. In the first instance, we were running into cases of users using mismatching architectures when they didn’t mean to and we weren’t aware of images that the users wanted to run that were malformed.

This instance is a bug in a supposed improvement to ARM support (by differentiating the various ARM variants). I’m sorry that we didn’t notice this case — the code looked good, had tests, it is just a situation that was missed, apparently by ARM users who wanted this improvement as well.

I think the primary thing that could help for all such similar cases is having ARM clusters in CI, if someone were interested in making that work and there were infrastructure available to run the tests there (and if it didn’t slow down the testing significantly). Or maybe just a sanity check to run on ARM before a release. Either way, I’m not aware of the current CI architecture having ARM hosts — but that may well just be my ignorance.

---

The error message in #3233 (comment) looks like it's coming from github.com/containers/image/manifest.Schema2List.ChooseInstance().
If I'm reading MatchesPlatform() right, if the passed-in wanted object includes a variant value, and the candidate entry from the manifest list doesn't include a variant value, it won't match.

Yup. To reproduce on non-ARM,

$ skopeo --override-os linux --override-arch arm --override-variant v7 copy docker://k8s.gcr.io/pause:3.1 dir:t

[everflux]

I didn't want to sound rude in any way, sorry if that was the case. Thanks for explaining the situation!
Having ARM as part of the CI infrastructure sounds good.

While I don't have personal experience with it, the Amazon EC2 Graviton instances might be worth a try since they are massively faster than raspberry or other SBCs. Amazon even offers credits for opensource projects: https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/

[vrothberg]

Yup. To reproduce on non-ARM,

True. I misread the code while skimming.

[mtrmac]

#3626 was filed for the “new” variant of this issue, do we want to keep two issues for two variants of the problem (and close this one as fixed previously), or to keep a single issue (and close one of the two as a duplicate)?

For the record, this is tracked in containers/image#898 .

[pjoomen]

#3626 was filed for the “new” variant of this issue, do we want to keep two issues for two variants of the problem (and close this one as fixed previously), or to keep a single issue (and close one of the two as a duplicate)?

I agree that these two issues are related. The only thing setting them apart, and therefore might be a reason to keep them seperate, is that #3626 was introduced in 1.17.3.

[matthiaslein]

It looks to me like this issue was just backported to 1.16.6

Two days ago, an update came through the opensuse repo I'm using for cri-o (cri-o-1.16/unknown,now 1.16.6~1 arm64 [installed]) and since then, I'm getting "Error choosing an image from manifest list docker://k8s.gcr.io/pause:3.1: no image found in manifest list for architecture arm64, variant v8, OS linux" for every image on my Raspberry Pi K8s cluster.

Can someone confirm?

[steled]

Hi,

I'm getting a similar error:

root@node02:~# kubeadm config images pull -v3
I0508 19:56:54.769266    3578 initconfiguration.go:103] detected and using CRI socket: /var/run/crio/crio.sock
I0508 19:56:54.793960    3578 version.go:183] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
W0508 19:56:55.840864    3578 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
failed to pull image "k8s.gcr.io/kube-apiserver:v1.18.2": output: time="2020-05-08T19:57:00+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error choosing image instance: no image found in manifest list for architecture arm, variant v7, OS linux"
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

See below my configuration:

root@node02:~# crio -v
crio version 1.17.4

root@node02:~# apt list cri-o-1.17
Listing... Done
cri-o-1.17/unknown,now 1.17.4~1 armhf [installed]

root@node02:~# apt list kube*
Listing... Done
kubeadm/kubernetes-xenial,now 1.18.2-00 armhf [installed]
kubectl/kubernetes-xenial,now 1.18.2-00 armhf [installed]
kubectx/stable 0.6.2-1 all
kubelet/kubernetes-xenial,now 1.18.2-00 armhf [installed]
kubernetes-cni/kubernetes-xenial,now 0.7.5-00 armhf [installed,automatic]
kubetail/stable 1.6.5-2 all

root@node02:~# uname -a
Linux node02 4.19.97-v7l+ #1294 SMP Thu Jan 30 13:21:14 GMT 2020 armv7l GNU/Linux

I'm working on a Raspberry Pi 4.

If you need more informations just let me know.

[therevoman]

I ran into this error while deploying kubespray on a Pine64 cluster running Fedora 32. I'd like to continue with kube 1.18, but the calico images will not pull with crictl because of this issue.

Nodes are Pine64 Sopine modules, which report arm64 v8.

fatal: [controller-1 -> 192.168.88.22]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/crictl", "pull", "docker.io/calico/kube-controllers:v3.14.0"], "delta": "0:00:00.999507", "end": "2020-05-31 00:35:03.776004", "msg": "non-zero return code", "rc": 1, "start": "2020-05-31 00:35:02.776497", "stderr": "time=\"2020-05-31T00:35:03-06:00\" level=fatal msg=\"pulling image failed: rpc error: code = Unknown desc = Error choosing image instance: no image found in manifest list for architecture arm64, variant v8, OS linux\"", "stderr_lines": ["time=\"2020-05-31T00:35:03-06:00\" level=fatal msg=\"pulling image failed: rpc error: code = Unknown desc = Error choosing image instance: no image found in manifest list for architecture arm64, variant v8, OS linux\""], "stdout": "", "stdout_lines": []}

[haircommander]

@therevoman AFAICT, this will be fixed in #3797. I will work on backporting the required PRs to 1.17 and 1.16. thanks for the patience folks 👍

[steled]

Hi,

is there any update? It isn't still working for now.

[haircommander]

what version @steled ?

[steled]

Hi @haircommander,

see below a list of some information.

pi@raspberrypi:~ $ sudo kubeadm config images pull --v=3
I0827 17:58:05.604065    2016 initconfiguration.go:103] detected and using CRI socket: /var/run/crio/crio.sock
I0827 17:58:05.605822    2016 version.go:182] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
I0827 17:58:06.262903    2016 version.go:251] remote version is much newer: v1.19.0; falling back to: stable-1.17
I0827 17:58:06.263044    2016 version.go:182] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.17.txt
W0827 17:58:06.650910    2016 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0827 17:58:06.651540    2016 validation.go:28] Cannot validate kubelet config - no validator is available
failed to pull image "k8s.gcr.io/kube-apiserver:v1.17.11": output: time="2020-08-27T17:58:07+01:00" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error choosing image instance: no image found in manifest list for architecture arm, variant v7, OS linux"
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

pi@raspberrypi:~ $ crio -v
crio version 1.17.4

pi@raspberrypi:~ $ kubelet --version
Kubernetes v1.17.11

pi@raspberrypi:~ $ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.11", GitCommit:"ea5f00d93211b7c80247bf607cfa422ad6fb5347", GitTreeState:"clean", BuildDate:"2020-08-13T15:17:52Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/arm"}

pi@raspberrypi:~ $ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.11", GitCommit:"ea5f00d93211b7c80247bf607cfa422ad6fb5347", GitTreeState:"clean", BuildDate:"2020-08-13T15:20:25Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/arm"}

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux

Hope this helps.

[haircommander]

sorry for the delay @steled this is fixed in cri-o 1.17.5, can you try that version?

[steled]

Hi @haircommander,

no problem.
I can't find that version:

pi@raspberrypi:~ $ sudo apt list --upgradable
Listing... Done
bind9-host/stable 1:9.11.5.P4+dfsg-5.1+deb10u2 armhf [upgradable from: 1:9.11.5.P4+dfsg-5.1+deb10u1]
containers-common/unknown 1.1.1~2 all [upgradable from: 1.1.1~1]
kubeadm/kubernetes-xenial 1.19.0-00 armhf [upgradable from: 1.17.11-00]
kubectl/kubernetes-xenial 1.19.0-00 armhf [upgradable from: 1.17.11-00]
kubelet/kubernetes-xenial 1.19.0-00 armhf [upgradable from: 1.17.11-00]
libbind9-161/stable 1:9.11.5.P4+dfsg-5.1+deb10u2 armhf [upgradable from: 1:9.11.5.P4+dfsg-5.1+deb10u1]
libdns-export1104/stable 1:9.11.5.P4+dfsg-5.1+deb10u2 armhf [upgradable from: 1:9.11.5.P4+dfsg-5.1+deb10u1]
libdns1104/stable 1:9.11.5.P4+dfsg-5.1+deb10u2 armhf [upgradable from: 1:9.11.5.P4+dfsg-5.1+deb10u1]
libisc-export1100/stable 1:9.11.5.P4+dfsg-5.1+deb10u2 armhf [upgradable from: 1:9.11.5.P4+dfsg-5.1+deb10u1]
libisc1100/stable 1:9.11.5.P4+dfsg-5.1+deb10u2 armhf [upgradable from: 1:9.11.5.P4+dfsg-5.1+deb10u1]
libisccc161/stable 1:9.11.5.P4+dfsg-5.1+deb10u2 armhf [upgradable from: 1:9.11.5.P4+dfsg-5.1+deb10u1]
libisccfg163/stable 1:9.11.5.P4+dfsg-5.1+deb10u2 armhf [upgradable from: 1:9.11.5.P4+dfsg-5.1+deb10u1]
liblwres161/stable 1:9.11.5.P4+dfsg-5.1+deb10u2 armhf [upgradable from: 1:9.11.5.P4+dfsg-5.1+deb10u1]

pi@raspberrypi:~ $ sudo apt search cri-o
Sorting... Done
Full Text Search... Done
cri-o-1.15/unknown 1.15.4-1~dev~ubuntu19.04~ppa4 armhf
  OCI-based implementation of Kubernetes Container Runtime Interface.

cri-o-1.16/unknown 1.16.6~1 armhf
  OCI-based implementation of Kubernetes Container Runtime Interface.

cri-o-1.16-dbgsym/unknown 1.16.6~1 armhf
  debug symbols for cri-o-1.16

cri-o-1.17/unknown,now 1.17.4~1 armhf [installed]
  OCI-based implementation of Kubernetes Container Runtime Interface.

cri-o-runc/unknown 1.0.0-6 armhf
  Open Container Project - runtime

cri-o-runc-dbgsym/unknown 1.0.0-6 armhf
  debug symbols for cri-o-runc

pi@raspberrypi:~ $ sudo apt-cache policy cri-o-1.17
cri-o-1.17:
  Installed: 1.17.4~1
  Candidate: 1.17.4~1
  Version table:
 *** 1.17.4~1 500
        500 http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Raspbian_10  Packages
        100 /var/lib/dpkg/status

And can't find such version also at: http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Raspbian_10/
or: https://github.com/cri-o/cri-o/tags

May I miss some source?

[haircommander]

oops my mistake, we never actually released 1.17.5. I am working with the other maintainers on getting it out the door!

[steled]

Sounds great. Will wait for the new release.

[haircommander]

1.17.5 is in the OBS repo now:
https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable:cri-o:1.17/cri-o
I have yet to make a pinned package yet, as there's a centos 7 error I need to chase down

Also, please refer to install.md for newer installation instructions. I see you're downloading using the old strategy (from the main libcontainers:stable repo) rather than the new strategy (creating specific cri-o releases in a libcontainers:stable:cri-o repo) I've attempted to add rasbian 10 and 9 to meet your needs :smiley:

[steled]

Hi @haircommander,

thanks for your support and the tip with the new strategy.
So I changed the strategy as described at APT based operating systems
For $OS I used Raspbian_10 and for $Version I used 1.17
Download and installatin works but there is no binary for crio :-D
Maybe I missed something?

EDIT:
I also tested it with Debian_Testing and cri-o version 1.17 but also didn't work.

EDIT 2:
Debian_Testing and cri-o versoin 1.18 seems to work.

[haircommander]

oh goodness, yes I did seem to break something. looking into it @steled

[haircommander]

okay @steled the issue should be fixed on Debian. there's something up with the Raspbian installation that I don't understand, that will have to wait for tomorrow, unless Debian installation meets your needs

[steled]

Good morning @haircommander,

I'm using Debian as test environment and Raspbian as prod so Debian is fine for now.
I hope it is not such a big problem with Raspbian.

Thanks and best regards

[steled]

Hi @haircommander,

just to let you know.
I tested Debian_Testing and cri-o version 1.17 and it works now.

Best regards and a nice weekend

[steled]

Hi @haircommander,

I tested a clean install of cri-o version 1.17 with cri-o-runc but this doesn't seem to work.
See the following informations:

steled@master01:~$ sudo apt search cri-o
cri-o/unknown,now 1.17.5~1 amd64 [installed]
  OCI-based implementation of Kubernetes Container Runtime Interface.

steled@master01:~$ sudo apt search cri-o-runc
cri-o-runc/unknown 1.0.0-6 armhf
  Open Container Project - runtime

steled@master01:~$ sudo apt-get install cri-o-runc
cri-o-runc is already the newest version (1.0.0-6).


steled@master01:~$ crio -v
crio version 1.17.5
commit: "251a47ba0930c28e83dea8d409b79f568dd711aa-dirty"

sudo journalctl -xe
Sep 08 19:36:51 master01 systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
-- Subject: A start job for unit crio.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit crio.service has begun execution.
--
-- The job identifier is 2334.
Sep 08 19:36:51 master01 crio[13077]: time="2020-09-08 19:36:51.556067963+02:00" level=fatal msg="runtime config: runtime validation: \"runc\" not found in $PATH: exec: \"runc\": executable file not found in $PATH"
Sep 08 19:36:51 master01 systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit crio.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Sep 08 19:36:51 master01 systemd[1]: crio.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit crio.service has entered the 'failed' state with result 'exit-code'.
Sep 08 19:36:51 master01 systemd[1]: Failed to start Container Runtime Interface for OCI (CRI-O).
-- Subject: A start job for unit crio.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit crio.service has finished with a failure.
--
-- The job identifier is 2334 and the job result is failed.

steled@master01:~$ sudo less /etc/crio/crio.conf
[crio.runtime.runtimes.runc]
runtime_path = ""
runtime_type = "oci"
runtime_root = "/run/runc"

I can't find a runc nor a cri-o-runc command.

[haircommander]

interesting, runtime_path = "" should be populated with the location of cri-o-runc. can you open a new issue, I believe it to be unrelated to this one (which I also think should be closed)

[steled]

ok I opened ticket #4177 ... I think that the cri-o-runc problem is not related to this problem...
I hope that this problem isn't that hard to solve...

[steled]

it seems that the part cgroup_manager at file /etc/crio/crio.conf changed:

cgroup_manager = "cgroupfs"

I added at the file /var/lib/kubelet/config.yaml the configuration cgroupDriver: systemd
so I also had to change the value at /etc/crio/crio.conf from cgroupfs to systemd

without this I get the following error at pod creation:

Failed to create pod sandbox: rpc error: code = Unknown desc = cri-o configured with cgroupfs cgroup manager, but received systemd slice as parent

[haircommander]

that seems to be from the same cause as #4177, for some reason, the debian builds aren't picking up overridden defaults

[haircommander]

I am closing this, as 1.17.5 has been released. please reopen if you disagree

[steled]

sorry but is it right that builds for Raspbian failed, see: https://build.opensuse.org/package/live_build_log/devel:kubic:libcontainers:stable:cri-o:1.17/cri-o/Raspbian_10/armv7l

it also seems that there isn't a new build at: https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.17/Raspbian_10/

btw. it seems that I can't reopen the issue