create --init-ctr once: flake: occasionally seems to actually (and incorrectly) run

[edsantiago]

This is a weird one. Only two occurrences (four, if you count retries within the same PR):

           podman make sure once container is removed
           /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_initcontainers_test.go:101
         
         [BeforeEach] Podman init containers
           /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_initcontainers_test.go:22
         [It] podman make sure once container is removed
           /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_initcontainers_test.go:101
         Running: podman [options] create --init-ctr once --pod new:foobar quay.io/libpod/alpine:latest bin/sh -c echo RvmaTMZpEbGBRJfg > /dev/shm/stbkBXTERreu
         21d775a67a44bb9deee0d5895f7bcfaa8bfd553847d68f7c87895c9bea2861b6
         Running: podman [options] create --pod foobar -t quay.io/libpod/alpine:latest top
         c0b94e4017e4766be71b2f22ee9a11ab2d8e7a82503c6d8d4f43b39ce1430386
         Running: podman [options] pod start foobar
         1c69bbfba8a4ad47516cdad9162751c3f5b57672be2a835821c0a7892315ef97
         Running: podman [options] container exists 21d775a67a44bb9deee0d5895f7bcfaa8bfd553847d68f7c87895c9bea2861b6
         Running: podman [options] pod stop foobar
         1c69bbfba8a4ad47516cdad9162751c3f5b57672be2a835821c0a7892315ef97
         Running: podman [options] pod start foobar
         1c69bbfba8a4ad47516cdad9162751c3f5b57672be2a835821c0a7892315ef97
         Running: podman [options] exec -it c0b94e4017e4766be71b2f22ee9a11ab2d8e7a82503c6d8d4f43b39ce1430386 cat /dev/shm/stbkBXTERreu
         RvmaTMZpEbGBRJfg      <------- expected: ENOENT, also expected exit status 1 (actual is 0)

This init functionality is new to me, and I haven't poked deeply enough to grok the test, but from my limited understanding this failure mode just shouldn't be possible. Filing as a placeholder.

Podman init containers [It] podman make sure once container is removed

• fedora-33 : int podman fedora-33 root container
  • PR Remove references to kube being development #11662
    • 09-20 14:34
  • PR cgroup-info: check if user.slice is valid before accessing value #11463
    • 09-06 16:16
    • 09-06 16:16
    • 09-06 16:16

Only fedora 33, and only since September. But just in case it's related, Ubuntu also had a failure, albeit a different error message:

Running: podman [options] exec -it b474b315832e6bf6cfdd1d5110b710302ef380ac97035a0fa9ca3034f88ab110 cat /dev/shm/sKRCHOZzSNUg
Error: container create failed (no logs from conmon): EOF    <--- yielding exit status 255, instead of the expected 1

• ubuntu-2104 : int podman ubuntu-2104 root host
  • PR network create: add warning for deprecated macvlan flag #11402
    • 09-07 05:10

[rhatdan]

@baude PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

Podman init containers [It] podman make sure once container is removed

• fedora-33 : int podman fedora-33 root container
  • PR Slirp4netns with ipv6 set net.ipv6.conf.default.accept_dad=0 #12098
    • 10-26 13:18
  • PR Fix a few problems in 'podman logs --tail' with journald driver #12067
    • 10-26 13:09
    • 10-26 13:09
    • 10-26 13:09
  • PR System tests: confirm that -a and -l clash #12049
    • 10-20 13:58
  • PR system tests: CONTAINER_* and --help: cleanup #12041
    • 10-19 17:17
    • 10-19 17:17
    • 10-19 17:17
  • PR codespell code #11948
    • 10-13 07:27
    • 10-13 07:27
  • PR introduce --replace flag for play kube #11671
    • 10-07 23:03
    • 10-07 23:03
    • 10-07 23:03

Some of these are triple-failures, meaning a user has to manually press re-run.

[edsantiago]

• fedora-33 : int podman fedora-33 root container
  • PR Bump Catatonit up to v0.1.7 #12223
    • 11-08 12:24
    • 11-08 12:24
    • 11-08 12:24
  • PR Slirp4netns with ipv6 set net.ipv6.conf.default.accept_dad=0 #12098
    • 10-26 13:18
  • PR Record the image stream along with the path #12090
    • 10-26 17:33
  • PR Fix a few problems in 'podman logs --tail' with journald driver #12067
    • 10-26 13:09
    • 10-26 13:09
    • 10-26 13:09
  • PR System tests: confirm that -a and -l clash #12049
    • 10-20 13:58
• ubuntu-2104 : int podman ubuntu-2104 rootless host
  • PR Set DOCKER_HOST in the VM #12066
    • 10-22 22:11
    • 10-22 22:11
    • 10-22 22:11

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

Last 30 days:

Podman init containers [It] podman make sure once container is removed

• fedora-33 : int podman fedora-33 root container
  • PR Backport CVE-2021-4024 fix to RHEL branch #12520
    • 12-06 10:24
    • 12-06 10:24
    • 12-06 10:24
• ubuntu-2110 : int remote ubuntu-2110 root host [remote]
  • PR filters: add basic pattern matching for label keys i.e --filter label=<pattern> #12295
    • 11-15 08:17 IGNORE THIS ONE

[vrothberg]

[...] exec -it 10742392702cc3a10235c7165007d81f6761502374d4fc7646bd2a896770c5f6 cat /dev/shm/MWQLTJoqqNsU
[...] Error: container create failed (no logs from conmon): EOF

@edsantiago is this the same as #10927?

[edsantiago]

Oops. Yes, sorry. My script just reports the failing test, and I blindly copied. Sorry about that. I've edited my comment, and confirmed that the other failures (in 12520) are the once flake.

[vrothberg]

OK, I think I found something.

[vrothberg]

I see two (separate) issues in the linked failures:

1. podman [options] exec -it c733c9d4b1c641454b402d3fd54630656e45e725f8f34df66765c1075457f866 cat /dev/shm/ajFADnTdMnDi should exit 1 but sometimes exits 0. I see this only on F33 container! Since we don't exercise that anymore, I consider it "fixed". The file shouldn't be in /dev/shm anymore once the Pod got restarted and it exceeds my knowledge what may would go wrong there.

2. The error below which I have a fix for:

         Running: podman [options] pod start foobar
         time="2021-10-23T01:13:43Z" level=error msg="Starting some container dependencies"
         time="2021-10-23T01:13:43Z" level=error msg="\"error creating systemd unit `libpod-c4240de22f92a54349e709c9131c9f7c7044936e1d6b5d0989736cbae3121d7e.scope`: got `failed`: OCI runtime error\""
         Error: error starting some containers: internal libpod error

[vrothberg]

#12593 for 2)

[vrothberg]

@edsantiago can you confirm my theory on 1)?

[edsantiago]

I can't confirm it, but since you have a possible fix, and since f33 is gone, it's perfectly fine with me if we sweep (1) under the rug.

[vrothberg]

Thanks!

[vrothberg]

With #12593 merged, I am closing this issue.