Failed to start containers on the agent

[stop5]

Component

agent

Describe the bug

The builds i start all fail even before the clone step. I tested the clone step manually and it worked.
My Problem is that the agent can't even start the container.
Its similar to the error mentioned in #85 (comment)_.
The containers run in an bridged network, because of some issues of podman with nftables.

System Info

{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-e08e1e1b"}

Additional context

journald log:
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"received execution"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"listen for cancel signal"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","Repo":"sebastian/website","Build":"16","ID":"46","time":"2022-07-10T09:27:20Z","message":"Executing 2 stages, in order of:"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","Repo":"sebastian/website","Build":"16","ID":"46","Stage":"0_7617429746464150374_clone","Steps":"0_7617429746464150374_clone","time":"2022-07-10T09:27:20Z","message":"stage"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","Repo":"sebastian/website","Build":"16","ID":"46","Stage":"0_7617429746464150374_stage_0","Steps":"0_7617429746464150374_step_0","time":"2022-07-10T09:27:20Z","message":"stage"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"error","error":"Error response from daemon: no container with name or ID \"0_7617429746464150374_clone\" found: no such container","time":"2022-07-10T09:27:20Z","message":"could not kill container '0_7617429746464150374_clone'"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"error","error":"Error response from daemon: no container with name or ID \"0_7617429746464150374_step_0\" found: no such container","time":"2022-07-10T09:27:20Z","message":"could not kill container '0_7617429746464150374_stage_0'"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"error","error":"Error: No such network: 0_7617429746464150374_default","time":"2022-07-10T09:27:20Z","message":"could not remove network '0_7617429746464150374_default'"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","error":"Error response from daemon: could not find free subnet from subnet pools","exit_code":1,"time":"2022-07-10T09:27:20Z","message":"pipeline complete"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"uploading logs"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"uploading logs complete"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","error":"Error response from daemon: could not find free subnet from subnet pools","exit_code":1,"time":"2022-07-10T09:27:20Z","message":"updating pipeline status"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"error","error":"rpc error: code = Unknown desc = Proc finished with exitcode 1, Error response from daemon: could not find free subnet from subnet pools","time":"2022-07-10T09:27:20Z","message":"grpc error: wait(): code: Unknown: rpc error: code = Unknown desc = Proc finished with exitcode 1, Error response from daemon: could not find free subnet from subnet pools"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"warn","repo":"sebastian/website","build":"16","id":"46","error":"rpc error: code = Unknown desc = Proc finished with exitcode 1, Error response from daemon: could not find free subnet from subnet pools","time":"2022-07-10T09:27:20Z","message":"cancel signal received"}
Jul 10 09:27:20 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:20Z","message":"pipeline done"}
Jul 10 09:27:21 echidna woodpacker-agent-0[100253]: {"level":"debug","repo":"sebastian/website","build":"16","id":"46","time":"2022-07-10T09:27:21Z","message":"updating pipeline status complete"}
Jul 10 09:27:21 echidna woodpacker-agent-0[100253]: {"level":"debug","time":"2022-07-10T09:27:21Z","message":"polling new jobs"}
Jul 10 09:27:21 echidna woodpacker-agent-0[100253]: {"level":"debug","time":"2022-07-10T09:27:21Z","message":"request next execution"}

podman version:
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Validations

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

[stop5]

I used socat to get into the "conversation" between the agent and podman: conn.log
i used this command to start socat and capture the output: socat -t100 -x -v UNIX-LISTEN:podman.sock,mode=777,reuseaddr,fork UNIX-CONNECT:podman.sock.orig 2>&1 | tee conn.log
The post request for creating the network (/v1.41/networks/create) after the volume is a bit strange for me because i set WOODPECKER_BACKEND_DOCKER_NETWORK and from what i understood, it should use the already existing network.
and after that it cleans up everything.

[6543]

look's like a podman issue witch can not create networks 🤔

[stop5]

i fixed the problem by using an non routed default network, using podman 3.0.1 and letting woodpecker create the network. the last one was the final working fix.
After it woodpecker created the network without problems and everything was working.

[6543]

Should that be documented?

[stop5]

that it works with podman 3: yes the documentation can be changed
the behaviour of letting woodpecker create the network or it fails: that should definitly be changed and until its changed it should be documented.

[6543]

@stop5 wana create a pull for the documentation?

[GamePlayer-8]

i fixed the problem by using an non routed default network, using podman 3.0.1 and letting woodpecker create the network. the last one was the final working fix. After it woodpecker created the network without problems and everything was working.

What non-routed default network means? Is it impossible to use Podman inside of Podman?
I'm actually struggling with that.