Failed to run cri-o 1.20 on Ubuntu 18.04

[Drakkai]

Description

I tried to install kubernetes 1.20.2 using kubespray based on cri-o and Ubuntu 18.04. Unfortunately, the installation ended with an error starting crio.service. So I tried to install cri-o on a fresh system, but without success - I am not able to run cri-o version 1.20 on Ubuntu 18.04. I installed it according to the instructions on the website https://cri-o.io/. I get the same error for version 1.19.

root@test:~# systemctl start crio
Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.
root@test:~# journalctl -u crio
-- Logs begin at Sun 2021-02-14 15:00:46 CET, end at Sun 2021-02-14 15:02:52 CET. --
Feb 14 15:02:16 test systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
Feb 14 15:02:16 test crio[2605]: time="2021-02-14T15:02:16+01:00" level=info msg="Starting CRI-O, version: 1.20.0, git: d388528dbed26b93c5bc1c89623607a1e597aa57(clean)"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.134115020+01:00" level=info msg="Node configuration value for hugetlb cgroup is true"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.134200522+01:00" level=info msg="Node configuration value for pid cgroup is true"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.134305024+01:00" level=error msg="Node configuration validation for memoryswap cgroup failed: node not configured with memory swap"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.134459526+01:00" level=info msg="Node configuration value for memoryswap cgroup is false"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.135773751+01:00" level=info msg="Node configuration value for systemd CollectMode is true"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.187879808+01:00" level=info msg="Overlay test mount did not indicate whether or not metacopy is being used: failed to mount overlay for metacopy check with \"nodev,metacopy=on\" options: invalid argument"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.188078612+01:00" level=fatal msg="Validating root config: failed to get store to set defaults: failed to mount overlay for metacopy check with \"nodev,metacopy=on\" options: invalid argument"
Feb 14 15:02:16 test systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE
Feb 14 15:02:16 test systemd[1]: crio.service: Failed with result 'exit-code'.
Feb 14 15:02:16 test systemd[1]: Failed to start Container Runtime Interface for OCI (CRI-O).

Steps to reproduce the issue:

1. Install Ubuntu 18.04
2. Add repositories and keys
3. Install cri-o and cri-o-runc
4. Reload systemd
5. Failed to start crio.service

Describe the results you received:
Validating root config: failed to get store to set defaults: failed to mount overlay for metacopy check with \"nodev,metacopy=on\" options: invalid argument

Describe the results you expected:
Run crio.service without problem.

Additional information you deem important (e.g. issue happens only occasionally):

Output of crio --version:

INFO[0000] Starting CRI-O, version: 1.20.0, git: d388528dbed26b93c5bc1c89623607a1e597aa57(clean)
crio version 1.20.0
Version:       1.20.0
GitCommit:     d388528dbed26b93c5bc1c89623607a1e597aa57
GitTreeState:  clean
BuildDate:     2020-12-16T11:12:35Z
GoVersion:     go1.15.2
Compiler:      gc
Platform:      linux/amd64
Linkmode:      dynamic

Additional environment details (AWS, VirtualBox, physical, etc.):
Ubuntu 18.04 on Hyper-V

root@test:~# uname -a
Linux test 4.15.0-135-generic #139-Ubuntu SMP Mon Jan 18 17:38:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@test:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

[haircommander]

can you try to run
sed -i 's/,metacopy=on//g' /etc/containers/storage.conf and see if that works? It seems ubuntu 18.04 doesn't support the option

[Drakkai]

Thanks, it works. Maybe it should be mentioned in the documentation?

root@test:~# sed -i 's/,metacopy=on//g' /etc/containers/storage.conf
root@test:~# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: https://github.com/cri-o/cri-o
root@test:~# systemctl start crio
root@test:~# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-02-15 18:19:21 CET; 2s ago
     Docs: https://github.com/cri-o/cri-o
 Main PID: 2705 (crio)
    Tasks: 8
   CGroup: /system.slice/crio.service
           └─2705 /usr/bin/crio

[haircommander]

we should really fix this in the package itself, it isn't too hard to auto-detect, and not worth end-user time/energy :)

[haircommander]

thanks for the report @Drakkai

[champtar]

An easy rule would be kernel >= 4.19 or kernel = 4.18.0 (EL 8.x)

[champtar]

One of the painful part is that ubuntu (18.04) has different kernel versions in the cloud than in normal images.
Cleanest way to fix this is to use today failing check and turn that into an autodetection rule, so this also handle custom kernel

[avoidik]

we should really fix this in the package itself, it isn't too hard to auto-detect, and not worth end-user time/energy :)

on the other hand, you can of course keep it for some nameless poor soul to learn about how to troubleshoot low level storage issues between CRI and Kubernetes API :)

[haircommander]

it's all a learning experience 😄

[danilutkin]

thank you for this experience )

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[haircommander]

I think autodetecting ended up being more tricky than expected

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

Closing this issue since it had no activity in the past 90 days.