rootless containers cannot connect using the local host on the same pod

[amnonh]

/kind bug

Description
I'm running version 2.0.3 so maybe it's a new thing because the documentation says it should work.

Running on Fedora 32 with rootless containers.
I'm creating two containers in the same pod. They should connect one another using the local host, but it does not work.

Steps to reproduce the issue:

1. podman run -d --pod=new:test docker.io/library/nginx:latest

2. podman run --rm -it --pod=test fedora curl http://localhost

Describe the results you received:
curl: (7) Failed to connect to localhost port 80: Connection refused

Describe the results you expected:
The default page from the server

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 2.0.3

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.19-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.19, commit: 5dce9767526ed27f177a8fa3f281889ad509fea7'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: ip-10-0-0-111.ec2.internal
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.7-300.fc32.x86_64
  linkmode: dynamic
  memFree: 11909259264
  memTotal: 16780132352
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 0
  swapTotal: 0
  uptime: 5h 58m 39.45s (Approximately 0.21 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/fedora/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 5
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/fedora/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 7
  runRoot: /run/user/1000/containers
  volumePath: /home/fedora/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.3

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.3-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):