Rawhide: rootless, remote: podman build: failed to create bridge "cni-podman0": could not add "cni-podman0": EPERM

[edsantiago]

Ton of failures in rootless gating tests, on latest rc2 build:

# $ podman-remote build -t nomtab /tmp/podman_bats.WiYj4J/build-test
# STEP 1/2: FROM quay.io/libpod/testimage:20210610
# STEP 2/2: RUN rm /etc/mtab
# error running container: did not get container start message from parent: EOF
# Error: error building at STEP "RUN rm /etc/mtab": plugin type="bridge" failed (add): cni plugin bridge failed: failed to create bridge "cni-podman0": could not add "cni-podman0": operation not permitted
# [ rc=125 (** EXPECTED 0 **) ]

No idea if this is real or a flake. If the latter, it's not one I've ever seen before. I will spin up a virt and try to reproduce today, but I'm still catching up from the weekend.

[edsantiago]

Here's the log from podman system service:

WARN[0072] Failed to load cached network config: network podman not found in CNI cache, falling back to loading network podman from disk
WARN[0073] 1 error occurred:
        * plugin type="bridge" failed (delete): cni plugin bridge failed: mkdir /var/lib/cni: permission denied

[edsantiago]

Reproducer:

$ printf "FROM quay.io/libpod/alpine:latest\nRUN true\n" | podman-remote build -t foo -
STEP 1/2: FROM quay.io/libpod/alpine:latest
STEP 2/2: RUN true
WARN[1035] Failed to load cached network config: network podman not found in CNI cache, falling back to loading network podman from disk
WARN[1035] 1 error occurred:
        * plugin type="bridge" failed (delete): cni plugin bridge failed: mkdir /var/lib/cni: permission denied

error running container: did not get container start message from parent: EOF
Error: error building at STEP "RUN true": plugin type="bridge" failed (add): cni plugin bridge failed: failed to create bridge "cni-podman0": could not add "cni-podman0": operation not permitted

(podman server running in bg, so output is intermingled)

[Luap99]

I think the problem here is that we use code which should only be used as root, we need to set the correct buildah isolation which is only done for local.

[rhatdan]

So the API Handler needs to realize it is running in rootless mode and set the network as such.

[Luap99]

PR #12996

[TomSweeneyRedHat]

So just to verify, is this issue only a problem for remote rootless, or all rootless? @edsantiago

[TomSweeneyRedHat]

Answered off line, this is only a problem with rootless remote build.