Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container IP in /etc/hosts is wrong with docker-compose #12533

Closed
flyingflo opened this issue Dec 7, 2021 · 12 comments · Fixed by #13918
Closed

Container IP in /etc/hosts is wrong with docker-compose #12533

flyingflo opened this issue Dec 7, 2021 · 12 comments · Fixed by #13918
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. network Networking related issue or feature

Comments

@flyingflo
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
On Fedora 35 with stock podman and docker-compose:
When running containers with docker-compose, the container's own IP address in /etc/hosts doesn't match it's network interface address. The address in the network's dnsmasq matches the interface address, however.
Hence, containers can not connect to themselves by hostname, but they can connect to other containers.

If I do virtually the same with podman alone (i.e. create a network, start a container, check the IPs) everything is fine.
So it is some docker-compose effect..

Steps to reproduce the issue:

  1. I created a small reproducer.

  2. Start podman.socket (if not active)

sudo podman build . -t fedora-net
sudo docker-compose up 

Here is what I get from cat /etc/hosts; ifconfig per container:

% sudo docker-compose up        
Creating network "test-network_default" with the default driver
Creating test-network_f2_1 ... done
Creating test-network_f1_1 ... done
Creating test-network_f3_1 ... done
Attaching to test-network_f2_1, test-network_f1_1, test-network_f3_1
f1_1  | 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
f1_1  | ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
f1_1  | 10.89.0.4       a8291ba32e16 test-network_f1_1
f1_1  | 10.89.0.1 host.containers.internal
f1_1  | eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
f1_1  |         inet 10.89.0.6  netmask 255.255.255.0  broadcast 10.89.0.255
f1_1  |         inet6 fe80::5840:9cff:feab:5da1  prefixlen 64  scopeid 0x20<link>
f1_1  |         ether 5a:40:9c:ab:5d:a1  txqueuelen 0  (Ethernet)
f1_1  |         RX packets 7  bytes 1654 (1.6 KiB)
f1_1  |         RX errors 0  dropped 0  overruns 0  frame 0
f1_1  |         TX packets 2  bytes 132 (132.0 B)
f1_1  |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
f1_1  | 
f1_1  | lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
f1_1  |         inet 127.0.0.1  netmask 255.0.0.0
f1_1  |         inet6 ::1  prefixlen 128  scopeid 0x10<host>
f1_1  |         loop  txqueuelen 1000  (Local Loopback)
f1_1  |         RX packets 0  bytes 0 (0.0 B)
f1_1  |         RX errors 0  dropped 0  overruns 0  frame 0
f1_1  |         TX packets 0  bytes 0 (0.0 B)
f1_1  |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
f1_1  | 
f3_1  | 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
f3_1  | ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
f3_1  | 10.89.0.2       e3be6af1f3d4 test-network_f3_1
f3_1  | 10.89.0.1 host.containers.internal
f3_1  | eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
f3_1  |         inet 10.89.0.7  netmask 255.255.255.0  broadcast 10.89.0.255
f3_1  |         inet6 fe80::f450:72ff:fe9b:8b43  prefixlen 64  scopeid 0x20<link>
f3_1  |         ether f6:50:72:9b:8b:43  txqueuelen 0  (Ethernet)
f3_1  |         RX packets 6  bytes 1501 (1.4 KiB)
f3_1  |         RX errors 0  dropped 0  overruns 0  frame 0
f3_1  |         TX packets 2  bytes 132 (132.0 B)
f3_1  |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
f3_1  | 
f3_1  | lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
f3_1  |         inet 127.0.0.1  netmask 255.0.0.0
f3_1  |         inet6 ::1  prefixlen 128  scopeid 0x10<host>
f3_1  |         loop  txqueuelen 1000  (Local Loopback)
f3_1  |         RX packets 0  bytes 0 (0.0 B)
f3_1  |         RX errors 0  dropped 0  overruns 0  frame 0
f3_1  |         TX packets 0  bytes 0 (0.0 B)
f3_1  |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
f3_1  | 
f2_1  | 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
f2_1  | ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
f2_1  | 10.89.0.3       ca2e8f9e76c4 test-network_f2_1
f2_1  | 10.89.0.1 host.containers.internal
f2_1  | eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
f2_1  |         inet 10.89.0.5  netmask 255.255.255.0  broadcast 10.89.0.255
f2_1  |         inet6 fe80::80f:97ff:fec4:d77c  prefixlen 64  scopeid 0x20<link>
f2_1  |         ether 0a:0f:97:c4:d7:7c  txqueuelen 0  (Ethernet)
f2_1  |         RX packets 10  bytes 1828 (1.7 KiB)
f2_1  |         RX errors 0  dropped 0  overruns 0  frame 0
f2_1  |         TX packets 2  bytes 132 (132.0 B)
f2_1  |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
f2_1  | 
f2_1  | lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
f2_1  |         inet 127.0.0.1  netmask 255.0.0.0
f2_1  |         inet6 ::1  prefixlen 128  scopeid 0x10<host>
f2_1  |         loop  txqueuelen 1000  (Local Loopback)
f2_1  |         RX packets 0  bytes 0 (0.0 B)
f2_1  |         RX errors 0  dropped 0  overruns 0  frame 0
f2_1  |         TX packets 0  bytes 0 (0.0 B)
f2_1  |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
f2_1  | 
test-network_f2_1 exited with code 0
test-network_f1_1 exited with code 0
test-network_f3_1 exited with code 0

Describe the results you expected:
The IPs of the same container in

  • in /etc/hosts,
  • the interface,
  • and the DNS resolver
    match.

Additional information you deem important (e.g. issue happens only occasionally):
Issue happens every time, but only with docker-compose.
I attach the journal output of podman.service. Note that, there are two instances of dnsmasq started for that network and containers are added two times, which looks strange to me.

Output of podman version:

% podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.8
Built:        Fri Nov 12 21:25:37 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

% podman info --debug
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    version: "35"
  eventLogger: journald
  hostname: floprimesign.lan
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.6-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 960458752
  memTotal: 16526204928
  ociRuntime:
    name: crun
    package: crun-1.3-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.3
      commit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 2151911424
  swapTotal: 4278185984
  uptime: 60h 13m 58.72s (Approximately 2.50 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/flo/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/flo/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 67
  runRoot: /run/user/1000/containers
  volumePath: /home/flo/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1636748737
  BuiltTime: Fri Nov 12 21:25:37 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

% rpm -q podman
podman-3.4.2-1.fc35.x86_64
% rpm -q docker-compose
docker-compose-1.29.2-3.fc35.noarch

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Dec 7, 2021
@apollo13
Copy link

apollo13 commented Dec 7, 2021

To add more information, this happens only with docker-compose up and not with docker-compose up -d: The main difference here is that without -d compose does the following sequence:

  • create
  • attach logs (this does not happen for detached containers), at this point the oci runtime assigns an ip (visible in the journal)
  • disconnect container from network
  • reconnect container to network
  • start

@Luap99
Copy link
Member

Luap99 commented Dec 7, 2021

Can you share the compose file so we can reproduce?

@flyingflo
Copy link
Author

flyingflo commented Dec 7, 2021

Sorry, I forgot all the attachments that I promised ;)

test-network.tar.gz
podman.service.log

@apollo13
Copy link

apollo13 commented Dec 7, 2021

Even more simplistic example:

services:
  testcontainer:
    image: docker.io/alpine:latest
    command: sh -c "cat /etc/hosts|grep testcontainer; ip a s eth0|grep inet"

@Luap99 Luap99 added the network Networking related issue or feature label Dec 15, 2021
@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Jan 15, 2022

@Luap99 Did you get a chance to look at this?

@mheon mheon self-assigned this Feb 3, 2022
@github-actions
Copy link

github-actions bot commented Mar 6, 2022

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Mar 7, 2022

@mheon Did you ever get anywhere with this one?

@mheon
Copy link
Member

mheon commented Mar 7, 2022 via email

@github-actions
Copy link

github-actions bot commented Apr 7, 2022

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Apr 7, 2022

@Luap99 is this something you are working on?

@Luap99
Copy link
Member

Luap99 commented Apr 7, 2022

Yes

@Luap99 Luap99 assigned Luap99 and unassigned mheon Apr 7, 2022
Luap99 added a commit to Luap99/libpod that referenced this issue Apr 22, 2022
When we connect or disconnect from a network we also have to update
/etc/hosts to ensure we only have valid entries in there.
This also fixes problems with docker-compose since this makes use of
network connect/disconnect.

Fixes containers#12533

Signed-off-by: Paul Holzinger <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. network Networking related issue or feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants