cannot ping services on host, from rootless container

[burhan]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I have a RHEL8 host that is running a postgresql instance. On the same host, I have podman running rootless containers, and in the environment for those containers, I am pointing to the host with port 5432.

The containers (running in a pod), are unable to connect to the host service. Exact error is

{ Error: connect ECONNREFUSED 192.168.28.131:5432
       at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
     errno: 'ECONNREFUSED',
     code: 'ECONNREFUSED',
     syscall: 'connect',
     address: '192.168.28.131',
     port: 5432 },
  original:

This error happens irrespective of using the network IP or 127.0.0.1 as the IP.

Here is the script I use to bring up the hosts:

#!/bin/bash
basedir=$(pwd)
podname="retool"
publish_ip=$(ip route get 1 | awk '{print $7;exit}')
random_key=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
podman pod rm -f ${podname}
podman system prune -a -f --volumes
podman pod create --name ${podname} -p ${publish_ip}:8080:80 -p ${publish_ip}:8443:443
rm ${basedir}/docker.env && touch ${basedir}/docker.env
tee "${basedir}/docker.env" > /dev/null <<'EOF'
NODE_ENV=production
POSTGRES_USER=retool
POSTGRES_PORT=5432
POSTGRES_PASSWORD=super-secret-password-here
EOF
echo "JWT_SECRET=${random_key}" >> ${basedir}/docker.env
echo "POSTGRES_HOST=${publish_ip}" >> ${basedir}/docker.env
buildah bud -t retool:latest -f ${basedir}/Dockerfile .
podman run -d --pod ${podname} --name ${podname}-jobs-runner --env-file=${basedir}/docker.env -e SERVICE_TYPE=JOBS_RUNNER  retool:latest bash -c --add-host postgres:${publish_ip} "chmod -R +x ./docker_scripts; sync; ./docker_scripts/wait-for-it.sh postgres:5432; ./docker_scripts/start_api.sh"
podman run -d --pod ${podname} --name ${podname}-db-connector --env-file=${basedir}/docker.env -e SERVICE_TYPE=DB_CONNECTOR_SERVICE  retool:latest bash -c "./retool_backend"
podman run -d --pod ${podname} --name ${podname}-https-portal --expose 80 --expose 443 --env-file=${basedir}/docker.env -e STAGE=local  --add-host api:127.0.0.1 tryretool/https-portal:latest
#podman run -d --pod ${podname} --name ${podname}-db-ssh-connector --env-file=${basedir}/docker.env -e SERVICE_TYPE=DB_SSH_CONNECTOR_SERVICE -v ssh:/retool_backend/autogen_ssh_keys -v ${basedir}/keys:/retool_backend/keys retool:latest bash -c "./docker_scripts/generate_key_pair.sh; ./retool_backend"
podman run -d --pod ${podname} --name ${podname}-api --env-file=${basedir}/docker.env -e SERVICE_TYPE=MAIN_BACKEND -e DB_CONNECTOR_HOST=http://db-connector -e DB_CONNECTOR_PORT=3002  --add-host db-connector:127.0.0.1 -v ssh:/retool_backend/autogen_ssh_keys -v ${basedir}/keys:/retool_backend/keys retool:latest bash -c "./docker_scripts/wait-for-it.sh postgres:5432; ./docker_scripts/start_api.sh"

The dockerfile comes from the retool project

Output of podman version:

Version:            1.9.3
RemoteAPI Version:  1
Go Version:         go1.13.4
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.13.4
  podmanVersion: 1.9.3
host:
  arch: amd64
  buildahVersion: 1.14.9
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.17-1.module+el8.2.1+6771+3533eb4c.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.17, commit: 3c703d9f178a3a53966e1d5c03d0275ea6cb36a0'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.2"
  eventLogger: file
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 4.18.0-193.14.3.el8_2.x86_64
  memFree: 896569344
  memTotal: 8189599744
  ociRuntime:
    name: runc
    package: runc-1.0.0-66.rc10.module+el8.2.1+6465+1a51e8b6.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  rootless: true
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.0.1-1.module+el8.2.1+6595+03641d72.x86_64
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.3.0
  swapFree: 8493461504
  swapTotal: 8493461504
  uptime: 212h 43m 56.2s (Approximately 8.83 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/retool/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 4
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: fuse-overlayfs-1.0.0-2.module+el8.2.1+6465+1a51e8b6.x86_64
      Version: |-
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/retool/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /tmp/run-1001/containers
  volumePath: /home/retool/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

(there are no updates for podman available in RHEL8)

Additional environment details (AWS, VirtualBox, physical, etc.):

This is a VM on VMWare Fusion

[burhan]

postgresql is listening on 127.0.0.1 by default; but I am not sure how to route 127.0.0.1 from the pods to the host. My understanding is rootless containers in pods use "localhost" and the other mappings to 127.0.0.1 inside the pod are working fine.

firewalld is not running:

[burhan@localhost ~]$ sudo systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

[AkihiroSuda]

If postgresql isn't listening on 192.168.28.131, ECONNREFUSED on 192.168.28.131 is expected.

Starting with Podman 2.1, 127.0.0.1 on the host can be exposed to containers as 10.0.2.2 when allow_host_loopback is set: 7722b58