remote: flake: podman rm (more than 1 container): no such container

[edsantiago]

Seen this one at least twice in CI. Here's a reproducer, albeit not a great one:

# cat testme
#!/bin/bash

set -ex

while :;do
    podman-remote run    -v /myvol1 --name c1 alpine true
    podman-remote run -d -v /myvol2 --name c2 alpine true
    podman-remote wait c2
    podman-remote rm c1 c2
done

# T0=$SECONDS;./testme;T1=$SECONDS;echo;echo $((T1-T0))
...
+ podman-remote run -v /myvol1 --name c1 alpine true
+ podman-remote run -d -v /myvol2 --name c2 alpine true
c1e9e53e217d2c081044f63de0fe4f41a9537072680d6b1b14cbb344e9590711
+ podman-remote rm c1 c2
c1e9e53e217d2c081044f63de0fe4f41a9537072680d6b1b14cbb344e9590711
Error: no container with name or ID c1e9e53e217d2c081044f63de0fe4f41a9537072680d6b1b14cbb344e9590711 found: no such container

Will usually fail within 5 minutes, but once it took 20. The wait is not actually necessary, but for some reason it makes the script fail more quickly. I'm pretty sure the -v is necessary, I couldn't get failures without it (but maybe I just didn't wait long enough).

CI failures: today, and Aug 28

[vrothberg]

Thanks a ton for the reproducer, @edsantiago!

I'll take a look. This one has been quite annoying.

[vrothberg]

I can reproduce and after an audit of the code I suspected that the issue is not on the server-side removal but somehow corrupted data on the client. So I started logging which containers the client wants to remove:

ERRO[0000] *** removing "c1c9a6f8cf12ce6252064aba9371c72ba9bb044b5b9e997cbfc87f4904f71875"
ERRO[0000] *** removing "c1c9a6f8cf12ce6252064aba9371c72ba9bb044b5b9e997cbfc87f4904f71875"

... and indeed the client tries to remove the same container twice although it should be two different IDs (for the two containers).

Digging further.

[vrothberg]

Found it ...