Latest podman on F30 Silverblue errors on iptables

[ashcrow]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

After upgrading to podman-1.3.1-1.git7210727.fc30.x86_64 and containernetworking-plugins-0.7.4-2.fc30.x86_64 in Fedora 30 (Silverblue) podman no longer works as expected.

Steps to reproduce the issue:

1. Create and run a container. EG: sudo podman run --rm -ti registry.fedoraproject.org/fedora:29 /bin/bash

Describe the results you received:

$ sudo podman run --rm -ti registry.fedoraproject.org/fedora:29 /bin/bash
ERRO[0000] Error adding network: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 3: iptables v1.8.2 (legacy): can't initialize iptables table `nat': Permission denied
                                                    Perhaps iptables or your kernel needs to be upgraded.
                                                                                                         
ERRO[0000] Error while adding pod to CNI network "podman": failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 3: iptables v1.8.2 (legacy): can't initialize iptables table `nat': Permission denied
                                                                              Perhaps iptables or your kernel needs to be upgraded.
                                                                                                                                   
Error: error configuring network namespace for container a28103429bb14f323b106ec8f7712d3f2d5788e154b2bb9a5bfede113af811ec: failed to list chains: running [/usr/sbin/iptables -t nat -S --wait]: exit status 3: iptables v1.8.2 (legacy): can't initialize iptables table `nat': Permission denied
Perhaps iptables or your kernel needs to be upgraded.
$ 

Describe the results you expected:
I expected a bash shell in the Fedora image I had locally.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:            1.3.1
RemoteAPI Version:  1
Go Version:         go1.12.2
OS/Arch:            linux/amd64

Output of podman info --debug:

$ podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.2
  podman version: 1.3.1
host:
  BuildahVersion: 1.8.2
  Conmon:
    package: podman-1.3.1-1.git7210727.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: c9a4c48d1bff85033b7fc9b62d25961dd5048689'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 12273864704
  MemTotal: 16691343360
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 8417964032
  SwapTotal: 8417964032
  arch: amd64
  cpus: 4
  hostname: work
  kernel: 5.1.6-300.fc30.x86_64
  os: linux
  rootless: true
  uptime: 44m 24.21s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.centos.org
store:
  ConfigFile: /home/steve/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /var/home/steve/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /run/user/1000
  VolumePath: /var/home/steve/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

• Physical
• Fedora 30 Silverblue

[mheon]

@mccv1r0 Mind taking a look? I think this is CNI, not Podman's IPTables code

[mccv1r0]

Ack. Quick test... I can't reproduce on fedora 30 server using same versions.

@ashcrow : Is firewalld running?

[ashcrow]

@mccv1r0 it is not. It failed to parse it's libvirt.xml file from libvirt-daemon-driver-network-5.1.0-6.fc30.x86_64

[mccv1r0]

@ashcrow So firewalld is meant to be used? Or is the intent to not use firewalld at all?

I ask because I'm trying to get to the bottom of what iptables v1.8.2 (legacy): means given I can't reproduce on fedora 30 server. In my case, firewalld was masked to look at a different issue.

[ashcrow]

@mccv1r0 I think it is meant to be running, but it is not. Looking at the packages for iptables I see:

$ rpm -qa | grep iptables
iptables-1.8.2-1.fc30.x86_64
iptables-libs-1.8.2-1.fc30.x86_64

It looks like the switch over to nftables is available already as well (maybe why iptables is legacy?)

$ rpm -q nftables
nftables-0.9.0-5.fc30.x86_64

When I brought this up earlier in a chat session @vrutkovs noted that he had to downgrade to an earlier version to get podman working on F30 Silverblue again.

[vrutkovs]

I was hitting #3187 (comment), downgrade to podman 1.2 worked. Not sure if its related to this issue

[ashcrow]

FWIW I am able to run the commands podman is trying to run manually without issue.

[zerodayz]

I have noticed that the userdata folder within the overlay container storage:

/var/home/$USERNAME/.local/share/containers/storage/overlay-containers/2508f7cad2af2b8d1ebaa6e286610fad03b1f70bbc30f69bf9cb12a19ef784fd/userdata

Was created with permissions

drwx------. 3  100000  100000 4096 Jun  6 17:40 userdata

But mine user ID was 1000 not 100000 so I have changed the perms and now I can connect to the container with

toolbox enter

Now even creating and entering new container works fine

toolbox create --container test
toolbox enter --container test

Removal of the container:

$ podman stop fedora-toolbox-30
2508f7cad2af2b8d1ebaa6e286610fad03b1f70bbc30f69bf9cb12a19ef784fd
$ podman ps -a
CONTAINER ID  IMAGE                                             COMMAND               CREATED         STATUS                      PORTS  NAMES
2508f7cad2af  registry.fedoraproject.org/f30/fedora-toolbox:30  toolbox --verbose...  27 minutes ago  Exited (143) 3 seconds ago         fedora-toolbox-30
$ toolbox rm fedora-toolbox-30
$ podman ps -a
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES

[ashcrow]

This ended up being a bad SELinux policy. Following the steps at https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/#_selinux_problems allowed things to work properly once more. Thanks for looking into this @mccv1r0 and @mheon. Thank you to @miabbott for pointing me at the SB troubleshooting page.