APIv2: DELETE libpod/containers/foo?force=true: unlinkat: directory not empty

[edsantiago]

Probable flake; filing so we can track it. In API tests:

ok 256 [25-containersMore] POST libpod/containers/foo/mount [] : status=200
ok 257 [25-containersMore] Check container mount ('/tmp/test-apiv2.tmp.QENVh1/overlay/7a4337c7d7df8e5d442d365fb12fc4feba80a722bf7775d1416acc001cfe92d2/merged') ~ .*merged
ok 258 [25-containersMore] POST libpod/containers/foo/unmount [] : status=204
not ok 259 [25-containersMore] DELETE libpod/containers/foo?force=true : status
#  expected: 204
#    actual: 500
  expected: 204
not ok 260 [25-containersMore] DELETE libpod/containers/foo?force=true: 204 status returns no output
#  expected: ''
#    actual: {"cause":"1 error occurred:
	* unlinkat /tmp/test-apiv2.tmp.QENVh1/overlay/7a4337c7d7df8e5d442d365fb12fc4feba80a722bf7775d1416acc001cfe92d2: directory not empty
         
","message":"error removing container 41d043ac3e04c3988f10f94be3b457fe2f79f058505ea7c0ad88ecaa0b6123d8 root filesystem: 1 error occurred:
	* unlinkat /tmp/test-apiv2.tmp.QENVh1/overlay/7a4337c7d7df8e5d442d365fb12fc4feba80a722bf7775d1416acc001cfe92d2: directory not empty
         
","response":500}
  expected: ''

Source: apiv2_test in #7534

Am assuming it's a flake, since I didn't touch any of this code. Will restart failing test.

[vrothberg]

Also seen here https://cirrus-ci.com/task/6597748277379072 in #6811

[vrothberg]

I've been poking around in the code and couldn't find anything suspicious in the code that would explain the flake.

However, local podman will evict containers in case of force which the remote one does not. Cc @baude @jwhonce.

Could it be that the unmount (right before the delete in the test) hasn't finished? @giuseppe WDYT?

[mheon]

My first suspicion would be that the unmount has not completely finished. If we add a 5-second sleep between them, that would help confirm the race.

[edsantiago]

This one is quickly becoming the number-one flake

259 [25-containersMore] DELETE libpod/containers/foo?force=true : status

• fedora : special_testing_rootless
  • PR run_networking e2e test: add cleanup to some tests #7593
    • 09-10 18:13
• fedora-31 : test fedora-31
  • PR Migrate away from docker.io #7534
    • 09-02 14:54
• fedora-32 : special_testing_rootless
  • PR Bump github.com/onsi/gomega from 1.10.1 to 1.10.2 #7552
    • 09-08 06:43
• fedora-c5809900649447424 : special_testing_rootless
  • PR Show c/storage (Buildah/CRI-O) containers in ps #7290
    • 09-09 06:48
• fedora-c5809900649447424 : test fedora
  • PR Show c/storage (Buildah/CRI-O) containers in ps #7290
    • 09-09 06:59
• prior-fedora : test prior-fedora
  • PR run_networking e2e test: add cleanup to some tests #7593
    • 09-10 18:27
• ubuntu-c5809900649447424 : test ubuntu
  • PR generate systemd: catch --name=foo #7571
    • 09-09 09:01

260 [25-containersMore] DELETE libpod/containers/foo?force=true: 204 status returns no output

• fedora : special_testing_rootless
  • PR run_networking e2e test: add cleanup to some tests #7593
    • 09-10 18:13
• fedora-31 : test fedora-31
  • PR Migrate away from docker.io #7534
    • 09-02 14:54
• fedora-32 : special_testing_rootless
  • PR Bump github.com/onsi/gomega from 1.10.1 to 1.10.2 #7552
    • 09-08 06:43
• fedora-c5809900649447424 : special_testing_rootless
  • PR Show c/storage (Buildah/CRI-O) containers in ps #7290
    • 09-09 06:48
• fedora-c5809900649447424 : test fedora
  • PR Show c/storage (Buildah/CRI-O) containers in ps #7290
    • 09-09 06:59
• prior-fedora : test prior-fedora
  • PR run_networking e2e test: add cleanup to some tests #7593
    • 09-10 18:27
• ubuntu-c5809900649447424 : test ubuntu
  • PR generate systemd: catch --name=foo #7571
    • 09-09 09:01