Error: OCI runtime error: error creating systemd unit libpod-sha.scope: got failed

[edsantiago]

New flake:

# $ podman run --name ignore-me --name my-container quay.io/libpod/testimage:20210427 true
# Error: OCI runtime error: error creating systemd unit `libpod-1b6bc93400c0c619d576f5efbcf411c9a4b09be70424e2081c8726b76424cfe6.scope`: got `failed`
# [ rc=126 (** EXPECTED 0 **) ]

Happens on the first podman run in system tests. Once it happens, all future tests will fail (i.e. this destroys the system). Two instances in the last week, both ubuntu-2104 rootless:

• ubuntu-2104 : sys podman ubuntu-2104 rootless host
  • PR Handle hard links in remote builds #10421
    • 05-22 06:28
  • PR Clear the storage-options from the graphdriver if users specifies --root #10400
    • 05-19 16:35

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago what is the latest on this flake?

[edsantiago]

Seen June 23 in int podman ubuntu-2104 rootless host. (Prior two failures were in sys tests, this one is int; all so far are rootless ubuntu 2104). As with the others, once it triggers, all subsequent tests fail.

[edsantiago]

Also seen June 8 in sys podman ubuntu-2104 rootless host.

[edsantiago]

Seen just now, again in sys podman ubuntu-2104 rootless

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

August 10, 23, 23 and 24. All of them ubuntu-2104

[edsantiago]

And, speak of the devil, another one right now

[vrothberg]

That looks like a runc issue to me (see https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/systemd/common.go#L354).

@kolyshkin , do you have a suspicion what's going on?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@AkihiroSuda @kolyshkin Thoughts?

[kolyshkin]

This means that runc asked systemd to create a unit, and systemd was unable to do that. Alas it only returns a single word ("failed"), I guess systemd logs might have some more details.

Now, I have never seen anything like this myself, and have no idea what could be the reason. If there is a way to repro, or get systemd logs after such failure, I'll be happy to take a look.

If I remember correctly, older runc versions used to ignore the failure from systemd, which resulted in errors later (upon changing unit properties or unit removal). This was fixed so that runc would error out earlier, which in theory is easier to debug.

Sorry for not being very helpful -- someone who's more familiar with system might have more ideas.

[edsantiago]

Since my last comment here the flake has appeared twice: Sept 9 and Sept 10, both on ubuntu-2104 rootless. No other instances since then (at least that my flake logger sees).

I realize this means that someone will close this issue. I'm semi-ok with that, since there's really nothing useful to go on; I just want the closing to be done intentionally and with awareness by someone other than me.

[rhatdan]

Ok, I will do the honors.

[edsantiago]

Reported by @baude today in planning meeting: ubuntu 2110 rootless

[kolyshkin]

@edsantiago is there a way to see system journal from that system? It might shed some light on what happened.

Again, this is systemd being unable to create a unit; all it returns back to runc is failed. Unless I have logs or a repro I can't do anything about it.

[edsantiago]

@kolyshin I think so: this?

More generally: from any log page, scroll to the top, click the Task link (not the Build one), and that gives you all the Cirrus details. The link I included here is to the journal section.

[kolyshkin]

OK this is actually crun, not runc. Otherwise all I said above is still true.

Error: OCI runtime error: crun: error creating systemd unit libpod-394f64f70cd61df9eeab89d8a24a7c5e8384536c3394c5d3efa08f82da7f3778.scope: got failed