Mac: current HEAD errors out on --add-host using host-gateway with "host containers internal IP address is empty"

[jfrantzius]

Issue Description

With current HEAD (Git hash bbd4476), an error occurs when using the --add-host option with host-gateway special IP designator, even though this is supposed to be supported with #14390

Steps to reproduce the issue

Steps to reproduce the issue

1. export CONTAINERS_MACHINE_PROVIDER=applehv
2. podman machine init --rootful
3. podman run --add-host foobar:host-gateway -it busybox

Describe the results you received

podman run --add-host foobar:host-gateway -it busybox
Error: preparing container a90605b73f90a2cab9db7142643a111f010ebabc6d8a74eb0bfeda0f525f97cf for attach: failed to create new hosts file: unable to replace "host-gateway" of host entry "foobar:host-gateway": host containers internal IP address is empty

Describe the results you expected

Expected a running container with a mapping for domain foobar in its hosts file

podman info output

(installed using `brew install podman --HEAD`, resulting in Git hash bbd4476)

podman info
host:
  arch: arm64
  buildahVersion: 1.33.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 98.79
    systemPercent: 0.66
    userPercent: 0.54
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2044
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.7.4-200.fc39.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 172539904
  memTotal: 2048000000
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc39.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.1-5.fc39.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.1
  ociRuntime:
    name: crun
    package: crun-1.14-1.fc39.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.14
      commit: 667e6ebd4e2442d39512e63215e79d693d0780aa
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231230.gf091893-1.fc39.aarch64
    version: |
      pasta 0^20231230.gf091893-1.fc39.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 10m 18.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 2611372032
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.9.0
  Built: 1706090829
  BuiltTime: Wed Jan 24 11:07:09 2024
  GitCommit: ""
  GoVersion: go1.21.6
  Os: linux
  OsArch: linux/arm64
  Version: 4.9.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[grozan]

Hi, issue is still there with podman 5.0.0 installed with brew

$ podman version
Client:       Podman Engine
Version:      5.0.0
API Version:  5.0.0
Go Version:   go1.22.1
Git Commit:   e71ec6f1d94d2d97fb3afe08aae0d8adaf8bddf0
Built:        Tue Mar 19 10:47:46 2024
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      5.0.0-dev-8a643c243
API Version:  5.0.0-dev-8a643c243
Go Version:   go1.21.8
Built:        Mon Mar 18 01:00:00 2024
OS/Arch:      linux/amd64

$ podman run --add-host foobar:host-gateway -it busybox
Error: preparing container 0cbcd3af7f916dde233efbe6410fef54f439395aa14797fe165ecec223a4428d for attach: failed to create new hosts file: unable to replace "host-gateway" of host entry "foobar:host-gateway": host containers internal IP address is empty

is it fixable at least?
Thanks

[Luap99]

Sure it is fixable but it is not so trival, the machine networking works differently. We do not use /etc/hosts but rather rely on gvproxy to serve the host.contianers.internal hostname so podman itself doesn't know the ip first hand.

However it should be possible to make a dns request from podman for this name so that we can then add the specified entry to /etc/hosts with the returned ip.

[cfergeau]

However it should be possible to make a dns request from podman for this name so that we can then add the specified entry to /etc/hosts with the returned ip.

This is also exposed by gvproxy over the /services/dns/all HTTP endpoint, but if I remember correctly, when podman starts gvproxy, this endpoint is not accessible from the host.

[CodingAnarchy]

Is there any workaround to resolve this?

[tobega]

I found a workaround so I can handle both docker and podman users:

host_gateway=$(podman network inspect -f "{{range .Subnets}}{{.Gateway}}{{end}}" podman 2>/dev/null || echo "host-gateway")

docker run --add-host foobar:"$host_gateway" -it busybox

[Luap99]

I found a workaround so I can handle both docker and podman users:

host_gateway=$(podman network inspect -f "{{range .Subnets}}{{.Gateway}}{{end}}" podman 2>/dev/null || echo "host-gateway")

docker run --add-host foobar:"$host_gateway" -it busybox

This returns the ip of the subnet gateway which is not the host ip for podman machine nor would this ip makes sense for rootless podman so I don't see how this would be a replacement.

[tobega]

I found a workaround so I can handle both docker and podman users:

host_gateway=$(podman network inspect -f "{{range .Subnets}}{{.Gateway}}{{end}}" podman 2>/dev/null || echo "host-gateway")

docker run --add-host foobar:"$host_gateway" -it busybox

This returns the ip of the subnet gateway which is not the host ip for podman machine nor would this ip makes sense for rootless podman so I don't see how this would be a replacement.

Well, it works for my case where I need several containers to communicate over the host network. I use it to replace --add-host host.docker.internal:host-gateway

[GiGurra]

I suspect this is the reason for k3d not working on podman, at least not on macos
k3d-io/k3d#1447 + k3d-io/k3d#1379

[cyangalaxy]

I have the same (or similar) problem, when I create/start a container without root.

Specifically, I'm on Fedora and creating & trying to start a Open WebUI container (a GUI for Ollama, local LLM).

Error: unable to start container: failed to create new hosts file: unable to replace "host-gateway" of host entry "host.docker.internal:host-gateway": host containers internal IP address is empty

What fixes the issue, is creating the container as Root. But I wish Non-Root as that allows me to use a nice graphical Podman manager app called "Pods"

[Luap99]

@cyangalaxy Are you using podman machine? If not it is not the same issue.

[cyangalaxy]

@cyangalaxy Are you using podman machine? If not it is not the same issue.

No I'm not (I'm on Linux which doesn't have that).

Should I create a new issue for my problem? I haven't found a solution for my problem (other than running as Root).

[Luap99]

@cyangalaxy If you are using pasta (default rootless networking since 5.0) then this is expected unless you have a second host ip available, see #22653
As mentioned there we are working towards better fixes for that for now this is expected

[andreaippo]

@cyangalaxy If you are using pasta (default rootless networking since 5.0) then this is expected unless you have a second host ip available, see #22653 As mentioned there we are working towards better fixes for that for now this is expected

Ok but is there any workaround right now for users working with:

• rootless
• bridge network
• linux
  ?

Thanks

[Luap99]

@andreaippo The quoted issue lists plenty of workarounds.

[andreaippo]

@andreaippo The quoted issue lists plenty of workarounds.

I tried the first one mentioned here, to no avail:

#22653 (comment)

I created ~/.config/containers/containers.conf with the following:

[network]
default_rootless_network_cmd = "pasta"
pasta_options = ["--map-gw"]

How can I be sure this change is effective?

Next, I have started my container on a bridge network:

docker run -p 8090:8080 --name=my-service --replace --net=my-bridge_default -d org.me/my-service

When I send a specific curl to my-service:8090, it contacts another service running on my host at :8080, using host.containers.internal. However, from the logs of the my-service container I can still see:

Caused by: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'host.containers.internal' [A(1)] and search domain query for configured domains failed as well: [dns.podman]

What am I doing wrong?

Thanks

Update: since this is OT, opened a new issue #23776

[Luap99]

@andreaippo This is off topic to this issue (as this is about macos), please create a new discussion if you need support.