podman build + workdir(?): Error adding network: failed to setup hairpin mode for veth3e067d53: operation not supported

[edsantiago]

This seems to be correlated with podman run of an image built with workdir. (It is not for me to understand why):

# podman build ...
...
STEP 5/25: WORKDIR /emkbn7Pa2r
...
# podman run thatimage ...
time="2021-07-15T13:33:54-05:00" level=error msg="error loading cached network config: network \"podman\" not found in CNI cache"
time="2021-07-15T13:33:54-05:00" level=warning msg="falling back to loading from existing plugins on disk"
Error: error configuring network namespace for container d7934ec1abb74c0fc57d5540482798f665bb41874ed916bc7603645ad7df622f: error adding pod boring_dhawan_boring_dhawan to CNI network "podman": failed to setup hairpin mode for vethb0252915: operation not supported
[ rc=126 (** EXPECTED 0 **) ]

Note the "error adding pod" message, despite no pods anywhere. Note also that "veth" appears in another buildah flake, containers/buildah#2628, although there it's "failed to make veth pair".

Podman network connect and disconnect [It] podman network connect when not running

• fedora-34 : int podman fedora-34 root host
  • PR Scrub podman commands to use report package #10688
    • 06-16 20:43

sys: podman build - workdir, cmd, env, label

• fedora-34 : sys podman fedora-34 root host
  • PR System tests: fix a multiarch problem #10947
    • 07-15 14:46
• fedora-34 : sys remote fedora-34 root host [remote]
  • PR Implemented Until Query Parameter for Containers/logs #10868
    • 07-09 14:12
  • PR Enhance system connection add URL input #10821
    • 06-29 20:38

Podman containers [It] podman kill latest container with SIGTERM

• fedora-34 : Test Bindings
  • PR Bump github.com/onsi/gomega from 1.13.0 to 1.14.0 #10879
    • 07-08 08:37

Also in a int-podman-fedora-34-root-container run that my flake referencer didn't seem to find

Also right now in my own PR, in sys-podman-fedora-34-root-host (not yet indexed by my flake catcher)

[rhatdan]

Coincidence, I just read the following:
#10052 (comment)

@Luap99 Could this be related.

[edsantiago]

The #10052 issue is rootless, whereas this issue (so far) I've only seen in root.

[Luap99]

I don't think this is a podman issue. The bridge cni plugin tries to set the hairpin mode on the veth interface and this fails. It only failed on f34, this could be a kernel problem.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago @Luap99 where do we stand on this one?

[edsantiago]

Seen twice last week:

sys: podman logs - journald log driver requires journald events backend

• fedora-34 : sys podman fedora-34 root host
  • PR podman inspect show exposed ports #11314
    • 08-24 14:09

Podman commit [It] podman commit with volume mounts and --include-volumes

• fedora-34 : int podman fedora-34 root host
  • PR e2e tests: fix overlay: Unknown option vfs.imagestore #11202
    • 08-19 15:34

[tlammi]

I had a similar issue when starting podman on a custom LInux distro built with Yocto project.

The issue was apparently that systemd-networkd had rules that matched the network interfaces created by podman.
The issue disappeared when I deleted all *.network files.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

@edsantiago are we still seeing this flake?

@Luap99 does @tlammi's comment on networkd interfering help?

[edsantiago]

The last instance I see in my flake logs is from September 14, and the log has been GC'ed, so I'll close.