podman images command slow, wants a kernel 4.8 or newer

[keestux]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

$ podman --log-level info images

Describe the results you received:

First I'm getting this output.

$ podman --log-level info images
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
INFO[0000] /usr/bin/podman filtering at log level info  
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: opaque flag erroneously copied up, consider update to kernel 4.8 or later to fix 
INFO[0000] Setting parallel job count to 25             

Describe the results you expected:

I expect the list of images within a reasonable amout of time. Now it takes about three minutes for 49 images

$ time podman --log-level info images
INFO[0000] podman filtering at log level info           
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: opaque flag erroneously copied up, consider update to kernel 4.8 or later to fix 
INFO[0000] Setting parallel job count to 25             
REPOSITORY                                 TAG                    IMAGE ID      CREATED       SIZE
...
real	1m34,135s
user	2m56,262s
sys	0m6,911s

Output of podman version:

$ podman version
Version:      3.2.1
API Version:  3.2.1
Go Version:   go1.16.7
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 8
  distribution:
    distribution: ubuntu
    version: "21.10"
  eventLogger: journald
  hostname: moorsterbeek
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.0-39-generic
  linkmode: dynamic
  memFree: 2736304128
  memTotal: 16568578048
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.1-0ubuntu2
      spec: 1.0.2-dev
      go: go1.16.5
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.4.0
  swapFree: 1027600384
  swapTotal: 1027600384
  uptime: 78h 16m 17.07s (Approximately 3.25 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/kees/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/kees/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 316
  runRoot: /run/user/1000/containers
  volumePath: /home/kees/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/impish,now 3.2.1+ds1-2ubuntu3 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

[giuseppe]

Podman performs some checks to verify whether native diff can be used. In your case, one of the tests fail (the opaque flag is copied up), so it cannot use the faster native diff.

I'd expect that version of the kernel on ubuntu to not have this issue though.

Is it a regression? Have you experienced it before?

[keestux]

I don't know if this is a regression. I only recently switched from docker to podman.

Just checking, on Ubuntu 20.04 I don't have that INFO message. There I've installed podman from https://download.opensuse.org because podman is not directly available on Ubuntu 20.04. Version 3.4.2

[keestux]

@giuseppe would you be able to give me a set of commands that I can run from the shell to repeat what is done in check.go:doesSupportNativeDiff. I'm not familiar with Go, but it seems doable to do the same check on the command line.

It's something to do with CONFIG_OVERLAY_FS_REDIRECT_DIR, right? which is not set in the Ubuntu kernel.

[giuseppe]

it is already fixed with containers/storage#948 (duplicate of containers/storage#947)

[giuseppe]

you'll need to upgrade to Podman 4

[keestux]

Upgrade to Podman 4 is easier said than done. I'm on a Ubuntu system. Even in 22.04 they have Podman v3.4.4