Requests to socket activated endpoints on the same host fail when using pasta

[job79]

Issue Description

Containers can't make requests to other containers on the same host when using pasta and systemd socket activation. We encountered this issue when trying to use matrix and other containers with matrix integration on the same host. The problem does not occur when using slirp4netns or host networking.

It's a bit difficult to explain. I hope the reproduce section clarifies the issue.

Steps to reproduce the issue

1. Setup a caddy instance with socket activation (minimal example).
2. Run $ podman run -it --rm alpine wget https://[domain of host] on the same host as the socket activated caddy instance. It outputs:

Connecting to plabble.org ([2a01:4f8:c012:feef::1]:443)
wget: can't connect to remote host: Connection refused

This command does work when executed on any other host.

3. $ podman run -it --net=host --rm alpine wget https://[domain of host] or $ podman run -it --net=slirp4netns --rm alpine wget https://[domain of host] does work:

Connecting to plabble.org ([2a01:4f8:c012:feef::1]:443)
Connecting to git.plabble.org ([2a01:4f8:c012:feef::1]:443)
saving to 'index.html'
index.html           100% |**********************************************************************************************************************************************************************| 42935  0:00:00 ETA
'index.html' saved

Describe the results you received

I can't make requests to the same host when using caddy, pasta and systemd socket activation.

Describe the results you expected

A successful requests.

podman info output

host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 94.88
    systemPercent: 3.37
    userPercent: 1.74
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "41"
  eventLogger: journald
  freeLocks: 2047
  hostname: nexus
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.11.5-300.fc41.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1296412672
  memTotal: 2054799360
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc41.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc41.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.18-1.fc41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.18
      commit: 8656b2548509fcc69ea7e8823a870564360a57a1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20241030.gee7d0b6-1.fc42.x86_64
    version: |
      pasta 0^20241030.gee7d0b6-1.fc42.x86_64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-3.fc41.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 1027076096
  swapTotal: 1027076096
  uptime: 0h 5m 38.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 31607205888
  graphRootUsed: 7664345088
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.5
  Built: 1729209600
  BuiltTime: Fri Oct 18 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.5

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

The host is running Fedora CoreOS 41.20241027.3.0.

Additional information

Tried with pasta 0^20240906.g6b38f07-1.fc41.aarch64-pasta and pasta 0^20241030.gee7d0b6-1.fc42.x86_64-pasta . I wasn't able to get podman 5.3 to install on CoreOS.

[Luap99]

If your domain resolves to the ip of the host and this is the main interface then this ip is copied by pasta into namesapce. As such if you connect to it from within the namesapce it will stay local to that namesapce and routed to the host.

https://blog.podman.io/2024/03/podman-5-0-breaking-changes-in-detail/ (see the pasta section)
https://blog.podman.io/2024/10/podman-5-3-changes-for-improved-networking-experience-with-pasta/ (see the map-guest-addr section)

So you should be able to connect to host.containers.internal instead but I think this is only connecting to ipv4 not ipv6. So you may need to specify your own --map-guest-addr option with an ipv6 address but I haven't tested that so far.

[job79]

Interesting. The systemd socket is binding to both ipv4 and ipv6, so doing curl host.containers.internal -H "Host: nexus.plabble.org" inside a container does indeed work and fix the issue. Thanks!

Only problem is that I would want my domain (nexus.plabble.org in this case) to map to host.containers.internal. Like $ podman run -it --add-host nexus.plabble.org:172.16.0.1 --rm alpine wget nexus.plabble.org but then without the hardcoded ip address. Is there a way to do this which I am overlooking?

[Luap99]

--add-host nexus.plabble.org:host-gateway should work. host-gateway being the magic string that is mapped to same ip as host.containers.internal.

[job79]

That does indeed work, thanks! Couldn't find that anywhere ;)

So I think this issue can be closed? I still don't fully understand why it was working without the socket activation, but I might understand after taking a closer look tomorrow.

[job79]

I still don't fully understand why it was working without the socket activation, but I might understand after taking a closer look tomorrow.

So I figured it out. It was working before switching to socket activation because the target container has a shared network with caddy. After switching it still has the shared network, but caddy doesn't accept the requests anymore so they fail.

Because adding --add-host nexus.plabble.org:host-gateway fixes the issue I am closing this. Appreciate your help!