race: 'podman stop' does not always remove all podman mounts

[edsantiago]

One of the e2e tests, podman list running container in test/e2e/mount_test.go, occasionally flakes in CI. Basically, podman mount is finding an active mountpoint even after the container is podman stopped. Rerunning podman mount a second later finds no mounts, so it's pretty likely a race: maybe there's some cleanup that isn't happening in time.

If this race condition is OK, that is, if it doesn't matter whether podman mount shows a dead mountpoint from a stopped container, then the e2e test must be fixed.

If this race condition is not OK, that is, if podman stop should guarantee that there are no mount points when it exits, then podman stop must be fixed.

Reproducer:

#!/bin/bash

set -e

T0=$SECONDS

while :;do
    cid=$(podman run -dt docker.io/library/alpine:latest top)
    podman mount --notruncate | grep -q $cid
    podman stop $cid > /dev/null
    m2=$(podman mount --notruncate)
    if [[ "$m2" =~ $cid ]]; then
        echo "FOO! Still mounted!"
        echo "$m2"
        echo "time = $(( $SECONDS - $T0 )) seconds"
        sleep 1
        echo
        echo "after sleep 1:"
        podman mount
        exit 1
    fi
    podman rm $cid >/dev/null
done

Sample run:

# /tmp/mtest
FOO! Still mounted!
2b473b127c369cd11da3c775780ea2e693e17511534c1e296980a35443d14a70 /var/lib/containers/storage/overlay/5798db64ca882ed888a403a5398d1bb5d020c8c0bab719ba67a71b13673b4773/merged
time = 96 seconds

after sleep 1:

CI failure is f30; I can reproduce with podman-1.8.0-4.fc30.

Problem still present in rawhide: podman-1.8.3-0.75.dev.gitf7dffed.fc33

[rhatdan]

Well if the container is running podman stop will stop the container, then conmon realizes that the container has stopped, and then execs podman container cleanup to cleanup the container. This is where the race happens.
We could make podman stop wait for the container to enter the stop state or disappear, but it could wait for a very long time,

[mheon]

Hmmmm. I think that, right now, podman stop does not provide a guarantee that a container has been cleaned up immediately after exit (we instead wait for the cleanup process to do it). I'm not sure if this is desirable, though. If we want to add such a guarantee, it would be trivial to make podman stop call ctr.Cleanup() immediately before exiting.

[rhatdan]

Wouldn't that give you a race as well, in that the container would still be running? Or at least podman container cleanup would fail.

[mheon]

I think we guarantee that the container is stopped after the Stop() API call - it's just that we don't actually verify that the cleanup has completed when podman stop exits.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mheon]

I'll self-assign this. It's fairly low priority, but hopefully I can get to it sometime in the next few weeks.