flake? Gating tests: failed to make veth pair

[edsantiago]

Source: gating tests for buildah-1.17.0-0.13.dev.git552cbd3.fc34 (but I've also seen it in other recent builds). (log).

not ok 371 combination-namespaces
[...lots and lots  and lots of messages...]
# buildah from --signature-policy /usr/share/buildah/test/system/policy.json --ipc=container --net=private --pid=host --userns=host --uts=private alpine
# $ /usr/bin/buildah from --signature-policy /usr/share/buildah/test/system/policy.json --quiet --ipc=container --net=private --pid=host --userns=host --uts=private alpine
# alpine-working-container-77
# $ /usr/bin/buildah run alpine-working-container-77 pwd
# /
# $ /usr/bin/buildah run --tty=true alpine-working-container-77 pwd
# the container `buildah-buildah749027279` is not in 'stopped' state
# error running container: error configuring network list if0 for [/bin/pwd]: failed to make veth pair: operation not permitted
# error while running runtime: exit status 1
# [ rc=1 (** EXPECTED 0 **) ]

[edsantiago]

Another one

[edsantiago]

And again in latest build. This is starting to look less and less like a flake.

[edsantiago]

Yeah, definitely not a flake

[rhatdan]

@edsantiago Is this still valid? Or can we close?

[edsantiago]

@rhatdan I haven't even looked at Fedorah Buildah gating tests in months, because there are so many builds and so many failures.

Looking at the last few f33 builds -- 1.19.4-1.fc33 and 1.19.0-1.fc33 -- and a recent f34 -- 1.20.0-0.12.dev.git7f340f9.fc34 -- I don't find a veth failure, so I'll just close and hope for the best.

[edsantiago]

Bummer. This just failed on a manual run (f33):

 ✗ combination-namespaces
   (from function `die' in file /usr/share/buildah/test/system/helpers.bash, line 196,
    from function `run_buildah' in file /usr/share/buildah/test/system/helpers.bash, line 183,
    in test file /usr/share/buildah/test/system/namespaces.bats, line 342)
     `run_buildah run --tty=true  $ctr pwd' failed
   ....
   lots and lots of stuff
   ...
   $ /usr/bin/buildah from --signature-policy /usr/share/buildah/test/system/policy.json --quiet --ipc=host --net=container --pid=container --userns=host --uts=host alpine
   alpine-working-container-18
   $ /usr/bin/buildah run alpine-working-container-18 pwd
   /
   $ /usr/bin/buildah run --tty=true alpine-working-container-18 pwd
   the container `buildah-buildah516995620` is not in 'stopped' state
   error running container: error configuring network list if0 for [/bin/pwd]: failed to make veth pair: operation not permitted
   error while running runtime: exit status 1
   [ rc=1 (** EXPECTED 0 **) ]
   #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
   #| FAIL: exit code is 1; expected 0
   #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I have no idea where buildah-buildah516995620 comes from. It appears only in that one line above, nowhere else in the full log.

buildah-1.19.6-1.fc33

I'm going to rerun the tests, and fully expect them to pass.

[edsantiago]

Nope. Failed again, at a different number:

$ /usr/bin/buildah from --signature-policy /usr/share/buildah/test/system/policy.json --quiet --ipc=host --net=container --pid=private --userns=container --uts=container alpine
alpine-working-container-28
$ /usr/bin/buildah run alpine-working-container-28 pwd
/
$ /usr/bin/buildah run --tty=true alpine-working-container-28 pwd
the container `buildah-buildah999213996` is not in 'stopped' state
error running container: error configuring network list if0 for [/bin/pwd]: failed to make veth pair: operation not permitted
error while running runtime: exit status 1
[ rc=1 (** EXPECTED 0 **) ]

[edsantiago]

And again

   $ /usr/bin/buildah from --signature-policy /usr/share/buildah/test/system/policy.json --quiet --ipc=host --net=private --pid=private --userns=container --uts=host alpine
   alpine-working-container-42
   $ /usr/bin/buildah run alpine-working-container-42 pwd
   /
   $ /usr/bin/buildah run --tty=true alpine-working-container-42 pwd
   the container `buildah-buildah157082686` is not in 'stopped' state
   error running container: error configuring network list if0 for [/bin/pwd]: failed to make veth pair: operation not permitted
   error while running runtime: exit status 1

...but I'm done for the evening.

Reproducer: get a 1minutetip f33 VM, install recent buildah-tests, then

# export BUILDAH_BINARY=/usr/bin/buildah
# bats /usr/share/buildah/test/system/namespaces.bats

[edsantiago]

And again, blocking release #3103: log

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago Better now?

[edsantiago]

Better now?

@rhatdan thank you, you've motivated me to extend my flake-tracking scripts to include buildah.

The most recent instance of this I see is from March 29. Does that mean we think it is magically gone, if we don't see it in thirty days?

[rhatdan]

SGTM

[edsantiago]

OK. I'll reopen if/when I see it again.

[jayaddison]

See also:

• https://bugzilla.redhat.com/show_bug.cgi?id=1956309 (OpenShift CI failures; from inspection of at least one full build log containing the failed to make veth pair message, buildah was in use)
• cri-o 1.21: pod container creation error: failed to make veth pair: operation not supported cri-o/cri-o#4878 (the same error message appearing during creation of pods in a CRI-O backed Kubernetes cluster)