[RFE]Make docker-compose work with rootless podman

[siretart]

I've tried docker-compose with rootless podman, but that doesn't currently work:

siretart@x1:~/git/awesome-compose/gitea-postgres$ docker-compose -H unix:/run/user/1000/podman/podman.sock up
Pulling gitea (gitea/gitea:latest)...
76146dacacd305f5bbb4af41b6d107c06679db432fd1b5de4dd9577a56789dcf: pulling image () from docker.io/gitea/gitea:latest 
Pulling db (postgres:alpine)...
8c6053d81a45b455a79e7abbc5b4d8410735f060634d345f70b0530b1d50b2b6: pulling image () from docker.io/library/postgres:alpine 
Creating gitea-postgres_gitea_1 ... error
Creating gitea-postgres_db_1    ... 

Creating gitea-postgres_db_1    ... error

ERROR: for gitea-postgres_db_1  network connect is not enabled for rootless containers

ERROR: for gitea  network connect is not enabled for rootless containers

ERROR: for db  network connect is not enabled for rootless containers
ERROR: Encountered errors while bringing up the project.

What would it take to make network connect work for rootless mode? Is that on the roadmap?

Originally posted by @siretart in #9059 (comment)

[baude]

as you can see, the network stack is problematic ... it has not specifically been discussed. I'm not terribly versed on the cni usage for rootless but i'll ask around and see what's involved.

[Luap99]

This is definitely possible. I am currently working on #8910. With the changes there it should be a very trivial change to support network connect/disconnect for rootless.

[zygous]

You've since closed #8910 in favour of #9423; does that impact whether rootless docker-compose is possible?

[Luap99]

@zygous It will work better with #9423 and is already included in there as well.

[q2dg]

I think there's some mental (and technical) gap between supporting docker-compose's configuration format file and Kubernetes's YAML configuration format file. Podman supports latter with generate kube/play kube commands and there's a separate app (podman-compose) which pretends to support the former. However, currently this fact misleads a little because you do not know which way to go

[mheon]

Podman provides full support for the native Docker-compose app now, so podman-compose is no longer the only option for Compose support. I agree that we could clarify this a bit more, but Kube YAML and Compose support serve separate purposes and are quite distinct.

[Draxter]

@siretart How do you start your podman.socket service as a regular user at unix:/run/user/1000/podman/podman.sock ?

[siretart]

Using systemd:

systemctl --user status podman

[Luap99]

@siretart @zygous or anybody else who is interested.
I would love to have some people to build and test PR #9423 with their compose files. It is working great for me. Their could be some distro specific things in the way, like selinux/apparmor so I would l like to know if it works for you, especially on debian based distros since I only tested on fedora33.

[siretart]

@Luap99 that PR doesn't apply to podman 3.0.1. Did you work on latest master? -- if so, any chance that you could provide a version that worked with 3.0.1 to minimize the changes in my test environment?

Otherwise I'd need to package latest HEAD, which I don't have the capacity to keep up with.

[Draxter]

I guess I need to wait for an update for this to be implemented? When running docker-compose in a rootless mode, it fails due to a network error, as by default it uses default cni called 'username_default'. While, from what I understand, in rootless mode, only 'slirp4netns' is supported?

$ podman --version
podman version 3.0.1

$ systemctl --user status podman.socket

● podman.socket - Podman API Socket
   Loaded: loaded (/usr/lib/systemd/user/podman.socket; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-03-23 09:08:38 GMT; 1h 45min ago
     Docs: man:podman-system-service(1)
   Listen: /run/user/1000/podman/podman.sock (Stream)
   CGroup: /user.slice/user-1000.slice/[email protected]/podman.socket

$ systemctl --user status podman

● podman.service - Podman API Service
   Loaded: loaded (/usr/lib/systemd/user/podman.service; static; vendor preset: enabled)
   Active: active (running) since Tue 2021-03-23 10:54:12 GMT; 3s ago
     Docs: man:podman-system-service(1)
 Main PID: 27113 (podman)
   CGroup: /user.slice/user-1000.slice/[email protected]/podman.service
           ├─ 1024 /usr/bin/podman
           ├─27113 /usr/bin/podman --log-level=info system service
           └─27122 /usr/bin/podman --log-level=info system service

$ cat docker-compose.yml
version: "3.8"
services:
  nginx:
    image: nginx:latest
    container_name: nginx
    ports:
      - 1111:80
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 80M

$ docker-compose -H unix:/run/user/1000/podman/podman.sock up -d
Creating nginx ... error

ERROR: for nginx  network connect is not enabled for rootless containers

ERROR: for nginx  network connect is not enabled for rootless containers
ERROR: Encountered errors while bringing up the project.

If I run the podman command manually (with the default/unspecified network), it works fine:

$ podman run -d --name nginx -p 1111:80 nginx:latest
23183655881bbc5007f55dbacc7f2c18640c82837c34de477f10cd5ca0e5f7a3

[Luap99]

@siretart I can make you a version based on 3.0.1.

[Luap99]

@Draxter You have to build #9423 from source in order to test.

[yajo]

Can't we just use the binaries from the CI job? https://github.com/containers/podman/pull/9423/checks?check_run_id=2167287171

[Luap99]

@yajo That should work too. I wasn't aware that binaries are available for each PR.

[Luap99]

@siretart https://github.com/Luap99/libpod/tree/rootless-cni-v3.0.1

[siretart]

@Luap99 sorry for the delay, and thanks for the backport in the branch. I've captured that backport in a patch and pushed the result here: https://salsa.debian.org/debian/libpod/-/tree/master.rootless.compose

With that package built and installed, I tried the initial test-case:

siretart@x1:~/git/awesome-compose/gitea-postgres$ docker-compose  -H unix:/run/user/1000/podman/podman.sock up 
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

This process hung for about a minute and then produced the error above. I suspect this is because podman pull is taking too long to download all of these images. I guess that's understandable, but the way docker-compose surfaces this to the user is kindof surprising. Not sure if there is anything that could be done about this from the podman side.

After trying a couple of times, I got to this result:

siretart@x1:~/git/awesome-compose/gitea-postgres$ docker-compose  -H unix:/run/user/1000/podman/podman.sock up
Starting gitea-postgres_db_1    ... error
Starting gitea-postgres_gitea_1 ... 

ERROR: for gitea-postgres_db_1  error preparing container ebf05efe1fa316b9b67515b45650b99a4dc1a73df6ee0d56f49380abb04dc685 for attach: error configuring network namespace for container ebf05efe1fa316b9b6Starting gitea-postgres_gitea_1 ... error

ERROR: for gitea-postgres_gitea_1  error preparing container 7fc71c590a5c246675b380775a9b712bb6161162f9047c8fda5c08464c57b764 for attach: error configuring network namespace for container 7fc71c590a5c246675b380775a9b712bb6161162f9047c8fda5c08464c57b764: CNI network "gitea-postgres_default" not found

ERROR: for db  error preparing container ebf05efe1fa316b9b67515b45650b99a4dc1a73df6ee0d56f49380abb04dc685 for attach: error configuring network namespace for container ebf05efe1fa316b9b67515b45650b99a4dc1a73df6ee0d56f49380abb04dc685: CNI network "gitea-postgres_default" not found

ERROR: for gitea  error preparing container 7fc71c590a5c246675b380775a9b712bb6161162f9047c8fda5c08464c57b764 for attach: error configuring network namespace for container 7fc71c590a5c246675b380775a9b712bb6161162f9047c8fda5c08464c57b764: CNI network "gitea-postgres_default" not found
ERROR: Encountered errors while bringing up the project.

For reference, I've been testing with the compose file from here: https://github.com/docker/awesome-compose/tree/master/gitea-postgres.

I'll have to try with other compose files, this issue may (or may not) appear with simpler setups. But I'd say this patch is progress! :-)

[Luap99]

@siretart Thanks for the testing. The network not found error is an issue which I fixed for v3.1.0. This is a flake you may not see this every time depending on the system. Version 3.1.0 will be released very soon. I recommend testing again when you updated to 3.1.0 there are more network related fixes in there.

[dirextric]

Hi everyone,

I've been testing rootless docker-compose and it's been working great on Fedora 33... except that it seems containers lose DNS resolution to the outside world (there are no issues accessing plain IP addresses). DNS resolution within networks still works, but has a 10 to 15 second delay. This happens with both docker-compose and the CLI (when connecting the containers to a network), so I used the CLI in my example for a more direct comparison.

With podman 3.0.1, this is what I get:

[user@localhost ~]$ podman network create testnet
/home/user/.config/cni/net.d/testnet.conflist
[user@localhost ~]$ podman run -d --name whoami-one --network testnet docker.io/containous/whoami:latest
b62144bb14d8e2faf6d7c0f72ce627b7e376aa80b33b7bb50534e99f8867b2d8
[user@localhost ~]$ podman run -d --name fedora-one --network testnet docker.io/fedora:latest sleep infinity
6a9a02ea393d12e4a9db9292fc80d88fd439f096fbf28c11677bda3a19763c03
[user@localhost ~]$ podman exec -it fedora-one /bin/bash
[root@6a9a02ea393d /]# curl whoami-one
Hostname: b62144bb14d8
IP: 127.0.0.1
IP: ::1
IP: 10.88.2.2
IP: fe80::4c40:a6ff:fe19:2216
RemoteAddr: 10.88.2.3:52636
GET / HTTP/1.1
Host: whoami-one
User-Agent: curl/7.71.1
Accept: */*

[root@6a9a02ea393d /]# curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

With the latest static build from CI for the PR, this is what I get:

[user@localhost ~]$ podman network create testnet
/home/user/.config/cni/net.d/testnet.conflist
[user@localhost ~]$ podman run -d --name whoami-one --network testnet docker.io/containous/whoami:latest
fe899318882e4187edeae2f1cb3a081b887e4ab350a92fd6f5f9aeeec9b53f16
[user@localhost ~]$ podman run -d --name fedora-one --network testnet docker.io/fedora:latest sleep infinity
7e7118b075a641b39182197cacecb07e5fe0b502c35c93a3dd4330a95f2a6bde
[user@localhost ~]$ podman exec -it fedora-one /bin/bash
[root@7e7118b075a6 /]# curl whoami-one

(long delay)

Hostname: fe899318882e
IP: 127.0.0.1
IP: ::1
IP: 10.88.2.2
IP: fe80::806c:aaff:fe39:b6ca
RemoteAddr: 10.88.2.3:52662
GET / HTTP/1.1
Host: whoami-one
User-Agent: curl/7.71.1
Accept: */*

[root@7e7118b075a6 /]# curl google.com
curl: (6) Could not resolve host: google.com

Is there some solution or workaround for this? Does this happen to anyone else?

[Luap99]

@dirextric Yes this problem is known. The workaround is to set the dns search domain to dns.podman with --dns-search. You can also set this in the compose file. This should not be needed with the 3.1 release.

[dirextric]

@Luap99 Setting the DNS search domain does indeed fix the slow internal resolutions, but it appears the containers still can't resolve to the outside world. Any ideas?

[Luap99]

@dirextric I pushed an update which fixes this. I assume that you only have a local dns server in /etc/resolv.conf, e.g. 127.0.0.53 for systemd-resolved.

[dirextric]

@Luap99 Yes, that is correct. The fix works great, thanks!

[Poweranimal]

Hi there, I'm still facing the same problem and wanted to ask, if I'm missing something here.

Running: docker-compose -H unix:///run/user/1000/podman/podman.sock up
Error: ERROR: for docker_redis_1 network connect is not enabled for rootless containers

podman --version
podman version 3.1.2

[Luap99]

You need version 3.2 (will be released soon).

[Poweranimal]

Got it. Thanks lot

[networkException]

Hello, I'm not sure if this is a regression or some configuration related error but I seem to be experiencing the same problem as #9169 (comment), on version 3.2.2 however. Setting the log level of docker-compose to DEBUG reveals stdout getting spammed with Pending: set() until the 60s timeout gets hit.

Opened a seperate issue: #10857

[Luap99]

@networkException Please open a new issue with steps to reproduce this.