flake: remote: podman cp dir from host to container

[edsantiago]

podman-remote cp tests are flaking; so far only on ubuntu-2004. Symptom is:

# podman-remote --url unix:/tmp/podman.RlDd0r exec cpcontainer cat /usr//hostfile0
#/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#|     FAIL: copy contents of dir to usr/ (cp -> ctr:/usr/)
#| expected: 'random-0-C2gGRg2DgJ'
#|   actual: ''
#\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Instances to date:

sys: podman cp dir from host to container

• gce_instance:prior-ubuntu : sys remote ubuntu-2004 root host
  • PR network: disallow CNI networks with user namespaces #8960
    • 01-13 12:44
  • PR Adding json formatting to --list-tags option in podman search command. #8923
    • 01-11 06:52
  • PR Use HTTPProxy settings from containers.conf #8905
    • 01-11 19:21
  • PR Fix podman logs read partial log lines #8904
    • 01-06 19:44
  • PR Bump github.com/containers/storage from 1.24.3 to 1.24.4 #8813
    • 12-23 06:04
  • PR add pod filter for ps #8804
    • 12-22 12:18

Also seen in nightly cron

Test context:

podman/test/system/065-cp.bats

Line 139 in bbff9c8

run_podman exec cpcontainer cat $dest_fullname/hostfile0

[vrothberg]

Thanks, @edsantiago! Do you know how long it's happening already? (i.e., when it started to occur)

[edsantiago]

The first comment shows all instances known to my cirrus-flake-summarize script. It shows the first instance on 2020-12-22.

[vrothberg]

I will take a look at it once #8281 is fixed.

[vrothberg]

Looking at the failure, I am under the impression that remote exec is the flaky.

The file seems to be copied since we're not getting an ENOENT from the container. So it seems like remote exec is missing the output.

[vrothberg]

The listed flakes have on thing in common: sys remote ubuntu-2004 root host

[edsantiago]

Well, now that you mention it, I searched my flake logs for anything that happens only on ubuntu 2004:

sys: podman cp into a subdirectory matching GraphRoot

• gce_instance:prior-ubuntu : sys remote ubuntu-2004 root host
  • PR Use HTTPProxy settings from containers.conf #8905
    • 01-12 06:50

sys: podman cp into container: weird symlink expansion

• gce_instance:prior-ubuntu : sys remote ubuntu-2004 root host
  • PR system tests (remote): reenable basic exec test #8871
    • 01-04 10:44

[edsantiago]

Here's one with a similar symptom, but it's f33 and not remote??

sys: podman exec - with keep-id

• gce_instance:fedora : sys podman fedora-33 root host
  • PR [ci:docs] Create release notes for V3.0.0 #8992
    • 01-17 09:03

[vrothberg]

It's smelling a like a race with conmon. A tough cookie without an easy reproducer.

I'll skip this one for now in hope to get a better reproducer or for it to magically disappear.

[vrothberg]

My hopes are high the flake has been fixed with the latest conmon v2.0.26 (currently in testing for F3{2,3}).

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago Can you confirm/deny @vrothberg decision, IE Can we close this issue?

[edsantiago]

@vrothberg's comment was on Feb 8. I see one instance since then:

sys: podman cp dir from host to container

• gce_instance:prior-ubuntu : sys remote ubuntu-2004 root host
  • PR Check for supportsKVM based on basename of the runtime #9598
    • 03-03 19:02

...but that was conmon-2.0.22~1-amd64. Unfortunately this is one of those flakes that appears in multiple tests, so I can't know if it's still present unless I go through all flake logs. But I will close for now, and reopen if I see it again.

[edsantiago]

It's not dead. (remote ubuntu 2110 root):

# # podman-remote --url unix:/tmp/podman_tmp_52Zm exec cpcontainer cat /srv/subdir/hostfile0
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #|     FAIL: copy to workdir/subdir (cp -> ctr:subdir)
# #| expected: 'random-0-BpQsfwQPb8'
# #|   actual: ''
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It's the old ubuntu, which we no longer test, so I'm not going to reopen this issue.

[edsantiago]

Reopening, because we just saw this on 36-aarch64:

• fedora-36-aarch64 : sys remote fedora-36-aarch64 root host [remote]
  • PR Reference #16207
    • 10-17 16:10

[edsantiago]

Looks like this is another Ubuntu-only error.