Cannot resolve other containers names in internal network

[yajo]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Containers in internal network are not able to see each other

Steps to reproduce the issue:

1. As rootless user.

2. ❯ podman network create --internal
   WARN[0000] dnsname and --internal networks are incompatible.  dnsname plugin not configured for network cni-podman0 
   /var/home/yajo/.config/cni/net.d/cni-podman0.conflist
   
   ❯ podman container run --rm -d --name a --net cni-podman0 docker.io/library/debian:10 sleep 5 && podman container run --rm -it --name b --net cni-podman0 docker.io/library/debian:10 ping a -c 1
   1e5526ee01a24161d6c135c15a8b2255383b771646338c38cb4790de98f5f66d
   ping: a: Temporary failure in name resolution
   

Describe the results you received:

Containers in same network cannot find each other.

Describe the results you expected:

Containers should see each other.

Additional information you deem important (e.g. issue happens only occasionally):

This works on docker:

❯ docker network create test --internal
09ca0528cc4ab395d64a1d73898c4ee7aa79605e0bb7abe9428e12b490c7a381

❯ docker container run --rm -d --name aaa --net test docker.io/library/debian:10 sleep 5 && docker container run --rm -it --name bbb --net test docker.io/library/debian:10 ping aaa -c 1
440c67d3482ffcba2477c8418b0fe450de5bfe973f2873cc68f2ca4de0460c19
PING aaa (172.23.0.2) 56(84) bytes of data.
64 bytes from aaa.test (172.23.0.2): icmp_seq=1 ttl=64 time=0.118 ms

--- aaa ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.118/0.118/0.118/0.000 ms

Output of podman version:

Version:      3.2.0
API Version:  3.2.0
Go Version:   go1.16.3
Built:        Wed Jun  9 15:24:16 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 12
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: yajolap-tecnativa-com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.12.9-300.fc34.x86_64
  linkmode: dynamic
  memFree: 731549696
  memTotal: 16613015552
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 16959922176
  swapTotal: 16965951488
  uptime: 1h 25m 49.31s (Approximately 0.04 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/yajo/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.3
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/yajo/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 18
  runRoot: /run/user/1000/containers
  volumePath: /var/home/yajo/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.0
  Built: 1623248656
  BuiltTime: Wed Jun  9 15:24:16 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/amd64
  Version: 3.2.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.0-5.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical Fedora Silverblue 34. Trying to migrate from docker-compose project that needs internal network with discovery.

[Luap99]

As you can see by the warning, dnsname is not supported with internal networks. For now you have to use external network if you want to use dnsname.
I tag this as feature, the team wants to rework the network stack in the near future.

[yajo]

Thanks for the feedback. And are there any known workarounds for the time being?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[roberth]

This is important for rootless docker-compose.

[Luap99]

Rootful podman has the same problem, this is not a rootless only issue.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[WhyNotHugo]

This can be easily reproduced with this example:

services:
  nginx:
    image: nginx
  client:
    image: alpine
    command: "sh -c 'apk add curl && curl -v nginx:80 && echo It works'"

And then:

docker-compose up nginx
docker-compose up client

Are there any known workarounds?

[roberth]

For rootfull networks, you can enable the dnsname plugin at the system level https://podman.io/getting-started/network#using-dns-in-container-networks

For the rootless ones, this has not been documented afaik, and I don't know if it's implemented or feasible.

[sthysel]

This is the most commonly used docker-compose functionality. People use docker-compose almost exclusively for the implicit internal network capability, yet this is not supported by podman... I must be missing something here. I cannot imagine that podman, especially v3 with the touted docker-compose compatibility lacks this fundamental basic capability.

[Luap99]

This issue is about internal networks not all networks! Your normal docker-compose networks should work with dns.

[mheon]

For the specific issue here (internal networks), we should be able to support this with the 4.0 network rewrite, though 4.0 is going to be focused on feature parity, so it may slip to 4.1.

[sthysel]

@Luap99

services:
  server:
    image: nginx

  client:
    image: alpine
    depends_on:
      - server
    command: "sh -c 'apk add curl && curl -v server:80 && echo It works'"

docker-compose up does not work in this most basic compose file. 'server' cannot be resolved.

edit: not to derail this thread further, but for other travelers, installing https://github.com/containers/dnsname enabled container <-> container resolution for the sample above.

[mheon]

That's a separate issue. Please file a new issue.

[WhyNotHugo]

A separate issue? I posted the same example 29 days ago. It was my understanding that this issue tracked exactly that example.

[yajo]

In #10672 (comment) the network is not internal (isolated), that's why it's a separate issue.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[baude]

works with podman4

[megakoresh]

works with podman4

Not for me it doesn't:

(default) [sddev@sd-dev-vm ~/test]$ cat /etc/redhat-release
CentOS Stream release 8
(default) [sddev@sd-dev-vm ~/test]$ podman version
Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.17.7

Built:      Tue Mar 15 21:15:06 2022
OS/Arch:    linux/amd64

version: '3.5'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.3
    container_name: elasticsearch
    environment:
    - ELASTIC_PASSWORD=changeme
    - discovery.type=single-node
    - ES_JAVA_OPTS=-Xms128m -Xmx256m
    - xpack.license.self_generated.type=trial
    - xpack.security.enabled=true
    ports:
    - 9200:9200
  kibana:
    image: docker.elastic.co/kibana/kibana:7.16.3
    container_name: kibana
    ports:
    - 5601:5601
    volumes:
    - ./kibana.yml:/usr/share/kibana/config/kibana.yml

(default) [sddev@sd-dev-vm ~/test]$ sudo docker-compose up -d
Creating network "test_default" with the default driver
Creating elasticsearch ... done
Creating kibana        ... done
(default) [sddev@sd-dev-vm ~/test]$ sudo podman exec kibana curl http://elasticsearch:9200
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: elasticsearch

This is quite a significant problem.

[Luap99]

It only works with netavark/aardvark not cni.

[megakoresh]

netavark/aardvark

But I have not opted to use cni, I use default settings which should use arrdvark

[Luap99]

Netavark is only used on new installs (or you have to run podman system reset) and distros that support it.
If you are on RHEL 8 you have to manually install it.

[0x29a]

Another note for Ubuntu travelers: this can be resolved by installing golang-github-containernetworking-plugin-dnsname via apt.

[monodot]

Unfortunately like @megakoresh this doesn't work for me either, on a fresh Fedora 36 install.

• Use case: rootless containers, initialised using podman-compose up. I want to address another container using its docker-compose service name (e.g. http://postgres)
• Networking backend: podman info output says networkBackend: netavark
• Podman version: podman version 4.2.0

Is there a log or somewhere I can look to troubleshoot this?

[mheon]

This is a very old issue. Please file a new bug report.

[daniel-widrick]

While this issue may be old... it remains completely unresolved.

[rhatdan]

Then file a new bug issue, we would prefer new issues with full info on the version of Podman you are seeing the issue with.