Podman v4.3.1 to 4.4.4 introduced breaking change for unsupported filesystems #1546

tis-rpage · 2023-03-31T02:58:07Z

Issue Description

On 4.3.1 and earlier, unsupported filesystems, such as 0x53464846 for WSLFS would not abort the command and would complete the action.

Steps to reproduce the issue

Run an unsupported filesystem, like WSL1 WLFS
Install podman 4.3.1
sudo podman images
view successful image list
Install podman 4.4.4
sudo podman images
view nothing, because error handling aborted on the unsupported filesystem that used to be functional yet unsupported

Describe the results you received

podman images
Error: filesystem type 0x53464846 reported for /var/lib/containers/storage is not supported with 'overlay': backing file system is unsupported for this graph driver

Describe the results you expected

podman images
REPOSITORY                                                        TAG                                           IMAGE ID      CREATED        SIZE
<none>                                                            <none>                                        b177afe92189  7 months ago   559 MB
<none>                                                            <none>                                        10a9f9f6f494  7 months ago   559 MB
quay.io/keycloak/keycloak                                         19.0.1                                        49756595a9cb  7 months ago   559 MB
gcr.io/k8s-minikube/kicbase                                       v0.0.33                                       b7ab23e98277  8 months ago   1.15 GB

podman info output

podman info --log-level=trace
INFO[0000] podman filtering at log level trace
DEBU[0000] Called info.PersistentPreRunE(podman info --log-level=trace)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Unknown filesystem type 0x53464846 reported for /var/lib/containers/storage
Error: filesystem type 0x53464846 reported for /var/lib/containers/storage is not supported with 'overlay': backing file system is unsupported for this graph driver
DEBU[0000] Shutting down engines

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Running as root on WSL1 alpine linux.

Additional information

Issue seems related to this pull request:
Update c/storage after #1436

As the issue affects the latest versions of buildah and skopeo, suggesting a change in error handling in c/images is impacting all 3 products. A possible fix would be to add type 0x53464846 as "WSLFS" as a supported filesystem to bypass the error handling.

The text was updated successfully, but these errors were encountered:

tis-rpage · 2023-03-31T03:14:07Z

The operating environment is WSL1 Alpine Linux 3.17 with /var/lib/containers on WSLFS. Similar issues suggest fuse-overlayfs as a workaround, however v4.3.1 works fine without that package available and v4.4.4 fails to even work with podman info.

Due to some concerns with running without systemd, PROPAGATION was set to shared, but likely was unnecessary.

sudo mount --make-rshared /
findmnt -o PROPAGATION /
PROPAGATION
shared

Working v4.3.1 podman info:

podman info --log-level=trace
INFO[0000] podman filtering at log level trace
DEBU[0000] Called info.PersistentPreRunE(podman info --log-level=trace)
TRAC[0000] Reading configuration file "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
TRAC[0000] &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.50.1 Annotations:[] BaseHostsFile: CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[nproc=32768:32768] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:k8s-file LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ConmonRsPath:[/usr/libexec/podman/conmonrs /usr/local/libexec/podman/conmonrs /usr/local/lib/podman/conmonrs /usr/bin/conmonrs /usr/sbin/conmonrs /usr/local/bin/conmonrs /usr/local/sbin/conmonrs /run/current-system/sw/bin/conmonrs] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath: EventsLogFileMaxSize:1000000 EventsLogger:file graphRoot:/var/lib/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/libexec/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] SSHConfig:/root/.ssh/config RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/var/lib/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/libpod VolumePath:/var/lib/containers/storage/volumes VolumePluginTimeout:5 VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:10.88.0.0/16 DefaultSubnetPools:[{Base:10.89.0.0/16 Size:24} {Base:10.90.0.0/15 Size:24} {Base:10.92.0.0/14 Size:24} {Base:10.96.0.0/11 Size:24} {Base:10.128.0.0/9 Size:24}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}
TRAC[0000] Reading configuration file "/etc/containers/containers.conf"
DEBU[0000] Merged system config "/etc/containers/containers.conf"
TRAC[0000] &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.50.1 Annotations:[] BaseHostsFile: CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[nproc=32768:32768] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:k8s-file LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ConmonRsPath:[/usr/libexec/podman/conmonrs /usr/local/libexec/podman/conmonrs /usr/local/lib/podman/conmonrs /usr/bin/conmonrs /usr/sbin/conmonrs /usr/local/bin/conmonrs /usr/local/sbin/conmonrs /run/current-system/sw/bin/conmonrs] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath: EventsLogFileMaxSize:1000000 EventsLogger:file graphRoot:/var/lib/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/libexec/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] SSHConfig:/root/.ssh/config RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/var/lib/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/libpod VolumePath:/var/lib/containers/storage/volumes VolumePluginTimeout:5 VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:10.88.0.0/16 DefaultSubnetPools:[{Base:10.89.0.0/16 Size:24} {Base:10.90.0.0/15 Size:24} {Base:10.92.0.0/14 Size:24} {Base:10.96.0.0/11 Size:24} {Base:10.128.0.0/9 Size:24}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Unknown filesystem type 0x53464846 reported for /var/lib/containers/storage
DEBU[0000] Unknown filesystem type 0x53464846 reported for /var/lib/containers/storage
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=<unknown>, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
TRAC[0000] found runtime "/usr/bin/crun"
TRAC[0000] found runtime "/usr/bin/runc"
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 37
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - devices
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.7-r0
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: unknown'
  cpuUtilization:
    idlePercent: 86.27
    systemPercent: 4.27
    userPercent: 9.47
  cpus: 12
  distribution:
    distribution: alpine
    version: 3.17.3
  eventLogger: file
  hostname: DESKTOP
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.4.0-22621-Microsoft
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 17150640128
  memTotal: 34266050560
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.7.2-r0
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/crun
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: false
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-r0
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 97427050496
  swapTotal: 103079215104
  uptime: 120h 59m 8.00s (Approximately 5.00 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 255465615360
  graphRootUsed: 229053087744
  graphStatus:
    Backing Filesystem: <unknown>
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 19
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1676457208
  BuiltTime: Wed Feb 15 05:33:28 2023
  GitCommit: ""
  GoVersion: go1.19.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

DEBU[0000] Called info.PersistentPostRunE(podman info --log-level=trace)

giuseppe · 2023-03-31T09:08:11Z

I think the issue is caused by d9da335.

I am fine with reverting the change, I am usually against these overzealous checks that block valid use cases.

The overlay driver previously raised an error when encountering an unsupported filesystem. This commit changes the error message to a debug log, allowing the overlay driver to continue its operation even with unsupported filesystems, without causing a failure. [NO NEW TESTS NEEDED] Introduced-by: containers#1374 Closes: containers#1546 Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe · 2023-03-31T09:16:26Z

opened a PR: #1547

aarontomlin · 2023-03-31T11:31:09Z

b

I think the issue is caused by d9da335.

I am fine with reverting the change, I am usually against these overzealous checks that block valid use cases.

Hi Giuseppe,

If I understand correctly, since the backing filesystem is not supported with OverlayFS it would not be safe to continue.
Why not provide actual support instead - assuming this is possible?

tis-rpage · 2023-03-31T13:31:31Z

b

I think the issue is caused by d9da335.
I am fine with reverting the change, I am usually against these overzealous checks that block valid use cases.

Hi Giuseppe,

If I understand correctly, since the backing filesystem is not supported with OverlayFS it would not be safe to continue. Why not provide actual support instead - assuming this is possible?

@aarontomlin I'm running on WSL1 which doesn't support overlayfs, yet skopeo, podman, and buildah previously were all able to work with copying, syncing, inspecting, building images. With the change in these new versions, that capability is broken.

I'm not sure if filesystem type 0x53464846 aka WSLFS should be whitelisted, or if the diagnosis should be a soft error. At the very least, for my use case a flag to allow the check to be ignored would let the tools be functional.

TomSweeneyRedHat · 2023-03-31T14:09:05Z

@giuseppe @rhatdan is this something we need to get into ZeroDay for RHEL 8.8/9.2? It smells like it.....

… not supported with OverlayFS" This reverts commit d9da335. Closes: containers#1546 Signed-off-by: Giuseppe Scrivano <[email protected]>

tis-rpage added the kind/bug label Mar 31, 2023

This was referenced Mar 31, 2023

Buildah v1.28.2 to 1.29.1 introduced breaking change for unsupported filesystems containers/buildah#4695

Closed

Skopeo v1.10.0 to 1.11.1 introduced breaking change for unsupported filesystems containers/skopeo#1956

Closed

giuseppe transferred this issue from containers/podman Mar 31, 2023

giuseppe mentioned this issue Mar 31, 2023

drivers/overlay: Notify the user if the backing filesystem is not supported with OverlayFS #1374

Merged

giuseppe mentioned this issue Mar 31, 2023

overlay: change error to debug log for unsupported fs #1547

Merged

rhatdan closed this as completed in #1547 Apr 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman v4.3.1 to 4.4.4 introduced breaking change for unsupported filesystems #1546

Podman v4.3.1 to 4.4.4 introduced breaking change for unsupported filesystems #1546

tis-rpage commented Mar 31, 2023

tis-rpage commented Mar 31, 2023 •

edited

Loading

giuseppe commented Mar 31, 2023

giuseppe commented Mar 31, 2023

aarontomlin commented Mar 31, 2023

tis-rpage commented Mar 31, 2023

TomSweeneyRedHat commented Mar 31, 2023

Podman v4.3.1 to 4.4.4 introduced breaking change for unsupported filesystems #1546

Podman v4.3.1 to 4.4.4 introduced breaking change for unsupported filesystems #1546

Comments

tis-rpage commented Mar 31, 2023

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

tis-rpage commented Mar 31, 2023 • edited Loading

giuseppe commented Mar 31, 2023

giuseppe commented Mar 31, 2023

aarontomlin commented Mar 31, 2023

tis-rpage commented Mar 31, 2023

TomSweeneyRedHat commented Mar 31, 2023

tis-rpage commented Mar 31, 2023 •

edited

Loading