podman 4.0.0-dev on M1 mac CNI error when publishing ports

[jfchevrette]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

On Apple M1 mac, running a container & publishing a port causes a CNI error

Steps to reproduce the issue:

1. Install podman from HEAD

brew install podman --HEAD

2. Init the machine

$ podman machine init --now
Downloading VM image: fedora-coreos-35.20220116.2.0-qemu.aarch64.qcow2.xz: done
Extracting compressed file
Image resized.
Machine init complete
INFO[0000] waiting for clients...
INFO[0000] new connection from  to /var/folders/cz/5z51srk94356dfhdhh0jvhl40000gn/T/podman/qemu_podman-machine-default.sock
Waiting for VM ...
Machine "podman-machine-default" started successfully

3. Attempt to run a container while exposing a port

$ podman run -it -P docker.io/nginx
Error: error preparing container 0e0d3dff78e806497b50791aaa6f480b47a12b3919252fc7ff62f2bfcb9dd293 for attach: error configuring network namespace for container 0e0d3dff78e806497b50791aaa6f480b47a12b3919252fc7ff62f2bfcb9dd293: error adding pod pedantic_allen_pedantic_allen to CNI network "podman": Post "http://host.crc.testing:7777/services/forwarder/expose": dial tcp 192.168.127.254:7777: connect: connection refused

Describe the results you received:

See error above

Describe the results you expected:

Container should run and port should be exposed to the client

Additional information you deem important (e.g. issue happens only occasionally):

The host host.crc.testing does not resolve

$ host host.crc.testing
Host host.crc.testing not found: 3(NXDOMAIN)

The IP 192.168.127.254 does not seem to be reachable

$ ping 192.168.127.254
PING 192.168.127.254 (192.168.127.254): 56 data bytes
ping: sendto: No route to host

$ nc 192.168.127.254:7777

$ curl -v 192.168.127.254:7777
*   Trying 192.168.127.254:7777...
* Immediate connect fail for 192.168.127.254: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server

Output of podman version:

$ podman version
Client:       Podman Engine
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.17.6
Git Commit:   9f8c0975eda7fe3acd91992f8285b68b731929e0
Built:        Tue Jan 25 19:49:44 2022
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.8

Built:      Wed Dec  8 16:48:10 2021
OS/Arch:    linux/arm64

Output of podman info --debug:

$ podman info --debug
host:
  arch: arm64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc35.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.10-200.fc35.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1646661632
  memTotal: 2048368640
  networkBackend: ""
  ociRuntime:
    name: crun
    package: crun-1.4-1.fc35.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.4
      commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.aarch64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 1m 15.5s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: ""
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1639000090
  BuiltTime: Wed Dec  8 16:48:10 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/arm64
  Version: 3.4.4

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[mheon]

The problem is that your server is on v3.4.4 - you need to match major version of client and server.

[jfchevrette]

@mheon I thought this might be the case. Is there a way to get the 4.0.0 server installed as part of podman machine init if 4.0.0 is not out yet? I thought it would install the correct version automatically

[mheon]

@baude @ashley-cui Is there any easy way to override the Podman package in use by FCOS?

[Luap99]

I think you can run rpm-ostree override replace <PODMAN4.0.rpm> I think you can get the rc2 rpm from koji here https://koji.fedoraproject.org/koji/buildinfo?buildID=1899046

[jfchevrette]

When I run the rpm-ostree override replace command I get

error: Non-local replacement overrides not implemented yet

I'm not familiar with rpm-ostree and my google-fu failed ...

[zeha]

This worked for me:

podman machine ssh
curl -O https://kojipkgs.fedoraproject.org//packages/podman/4.0.0/0.2.rc2.fc35/aarch64/podman-4.0.0-0.2.rc2.fc35.aarch64.rpm
sudo -i
rpm-ostree override replace /home/core/podman-4.0.0-0.2.rc2.fc35.aarch64.rpm
systemctl reboot

[jfchevrette]

@zeha thanks this worked for me as well. Closing as this is not an issue

[nbenns]

Just adding in case anyone else is having trouble, here are the steps that worked for me to install podman and kind on Mac and get them working together.

brew install podman --head

# make sure docker desktop is deleted
sudo unlink /usr/local/bin/docker
sudo ln -s /usr/local/bin/podman /usr/local/bin/docker

podman machine init
podman machine start
podman machine ssh

# for intel:
curl -O https://kojipkgs.fedoraproject.org//packages/podman/4.0.0/0.2.rc2.fc35/x86_64/podman-4.0.0-0.2.rc2.fc35.x86_64.rpm

# for arm:
curl -O https://kojipkgs.fedoraproject.org//packages/podman/4.0.0/0.2.rc2.fc35/aarch64/podman-4.0.0-0.2.rc2.fc35.aarch64.rpm

sudo -i

# for intel:
rpm-ostree override replace /home/core/podman-4.0.0-0.2.rc2.fc35.x86_64.rpm

# for arm:
rpm-ostree override replace /home/core/podman-4.0.0-0.2.rc2.fc35.aarch64.rpm

echo ip6_tables > /etc/modules-load.d/ip6_tables.conf

systemctl reboot

brew install kind kubectl
kind create cluster