New flake: invalid path etc/hosts: OCI runtime error

[edsantiago]

This is another hard-to-index bug because it appears in multiple tests. It seems to be pod-related:

# podman [options] run --pod testpod -d quay.io/libpod/alpine:latest top
time="2022-01-03T09:50:13-06:00" level=error msg="Starting some container dependencies"
time="2022-01-03T09:50:13-06:00" level=error msg="\"invalid path `etc/hosts`: OCI runtime error\""
Error: error starting some containers: internal libpod error

Note the missing root-level slash in etc/hosts. I don't know if that's a clue or not. I can't reproduce on my laptop.

The first appearance in my logs dates to early September. Below is (I think) a list of some logs that show the bug. I haven't confirmed all the links, so some of those may be misleading. But somewhere in there are real failures, and once it happens, it seems to happen often.

Podman pod rm [It] podman pod start/remove multiple pods via --pod-id-file

• fedora-35 : int podman fedora-35 root host
  • PR compat: image normalization: handle sha256 prefix podman#12663
    • 12-21 05:10
    • 12-21 05:10
    • 12-21 05:10
    • 12-21 03:34
    • 12-21 03:34
    • 12-21 03:34
    • 12-20 12:18
  • PR [CI:DOCS] Small checkpoint/restore man page fixes podman#12662
    • 12-21 03:49
    • 12-21 03:49
    • 12-21 02:32
    • 12-21 02:32
    • 12-21 02:32
    • 12-20 12:05
    • 12-20 12:05
    • 12-20 12:05
    • 12-20 11:25
    • 12-20 11:25
    • 12-20 11:25
  • PR [CI:DOCS] Explicitly mention that checkpointing systemd containers might fail podman#12659
    • 12-20 12:05
    • 12-20 12:05
    • 12-20 12:05
    • 12-20 11:23
    • 12-20 11:23
    • 12-20 11:23
  • PR fix(generate): fix up podman generate kube missing env field bug podman#12648
    • 12-18 06:09
    • 12-18 06:09
    • 12-18 06:09
    • 12-17 21:10
    • 12-17 21:10
    • 12-17 21:10
  • PR legacy events: also set Action="die" podman#12643
    • 01-03 11:13
    • 01-03 11:13
    • 01-03 11:13
    • 01-03 08:49

Podman pod stop [It] podman pod start/stop multiple pods via --pod-id-file

• fedora-35 : int podman fedora-35 root host
  • PR compat: image normalization: handle sha256 prefix podman#12663
    • 12-21 05:10
    • 12-21 05:10
    • 12-20 12:18
    • 12-20 12:18
    • 12-20 12:18
  • PR [CI:DOCS] Small checkpoint/restore man page fixes podman#12662
    • 12-21 03:49
    • 12-21 03:49
    • 12-21 03:49
  • PR [CI:DOCS] Explicitly mention that checkpointing systemd containers might fail podman#12659
    • 12-20 12:05
    • 12-20 12:05
  • PR fix(generate): fix up podman generate kube missing env field bug podman#12648
    • 12-18 06:09
    • 12-17 21:10
    • 12-17 21:10
  • PR legacy events: also set Action="die" podman#12643
    • 01-03 11:13
    • 01-03 08:49

[rhatdan]

@giuseppe PTAL
I think we should fix Podman to list the name of the OCI-Runtime
time="2022-01-03T09:50:13-06:00" level=error msg="Starting some container dependencies"
time="2022-01-03T09:50:13-06:00" level=error msg=""invalid path etc/hosts: OCI runtime crun error""
Error: error starting some containers: internal libpod error

[giuseppe]

there was an issue in crun that could be related: #819

Since the PR containers/podman#12643 wasn't rebased to include the yum upgrade -y crun fix we currently have, I'd say to wait until we have updated images and see if it happens again