podman-remote run --rm: Error removing container / error looking up container / mounts: layer not known

[edsantiago]

This looks kind of serious, mostly because it's so easily and often reproduced:

[ assumes podman system service running elsewhere]
$ while :;do ./bin/podman-remote run --rm alpine date;echo $?;done
Thu May 20 21:39:28 UTC 2021
0
...
Thu May 20 21:39:33 UTC 2021
ERRO[0000] Error removing container b3286c58744903d1e5b7d679ca9168ca16c2d5d1a6084b3aa3919c41cbe28f8e: error looking up container "b3286c58744903d1e5b7d679ca9168ca16c2d5d1a6084b3aa3919c41cbe28f8e" mounts: layer not known
...
Thu May 20 21:39:35 UTC 2021
WARN[0000] Container 6b4d7060b9466bd7c0e806758b3c2366227ff415784541882ea324c0bfb1b9e2 does not exist: container 6b4d7060b9466bd7c0e806758b3c2366227ff415784541882ea324c0bfb1b9e2 does not exist in database: no such container
0
...
Thu May 20 21:39:37 UTC 2021
ERRO[0000] Error removing container 20c17035174b364f7e3d983ee0af899eb7af635c5a8d2d09b48b44932d606086: error looking up container "20c17035174b364f7e3d983ee0af899eb7af635c5a8d2d09b48b44932d606086" mounts: layer not known

Sometimes it's ERRO, sometimes WARN. In all cases, exit status is 0 (no error).

master @ e48aa8c, root and rootless

[rhatdan]

This is the fix in Buildah to show log messages on the client side, these messages used to be just on the server side. I think what is happening, is a race condition between the container being removed in the API and the container being removed via conmon.

In either case we should drop the remove error on container does not exist to debug it should not be warning or error.

[edsantiago]

Ah, thanks for the analysis. I would recommend making this a top priority; otherwise, podman-remote tests in CI are going to start flaking uncontrollably.

[rhatdan]

I ran your test with the patch for a minute and saw no errors.

[edsantiago]

#10427 and #10421 have merged, I believe this is fixed.