`system df` fails with `error locating item named "manifest"` #751

Hi-Angel · 2021-08-30T18:58:31Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman system df results in

Error: error locating item named "manifest" for image with ID "a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad": file does not exist

I don't have any unusual setups, nothing I'd deem to be responsible for odd behavior, so I figured it's a bug worth reporting.

Steps to reproduce the issue:

Run podman system df

Describe the results you received:

Error: error locating item named "manifest" for image with ID "a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad": file does not exist

Describe the results you expected:

Information of how much disk space podman images take.

Output of podman version:

Version:      3.3.0
API Version:  3.3.0
Go Version:   go1.17
Git Commit:   98f252a3a1a8f1ee00f9f96c6ba00500954b5093-dirty
Built:        Sat Aug 21 14:48:18 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

Details

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.29-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 7e6de6678f6ed8a18661e1d5721b81ccee293b9b'
  cpus: 4
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: constantine-N61Ja
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.13.13-zen1-1-zen
  linkmode: dynamic
  memFree: 458891264
  memTotal: 8229945344
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 0.21-1
    path: /usr/bin/crun
    version: |-
      crun version 0.21
      commit: c4c3cdf2ce408ed44a9e027c618473e6485c635b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 6585036800
  swapTotal: 8053059584
  uptime: 51h 42m 15.1s (Approximately 2.12 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /home/constantine/.config/containers/storage.conf
  containerStore:
    number: 144
    paused: 0
    running: 1
    stopped: 143
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.7.1-1
      Version: |-
        fusermount3 version: 3.10.4
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.4
        using FUSE kernel interface version 7.31
  graphRoot: /home/constantine/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 127
  runRoot: /run/user/1000/containers
  volumePath: /home/constantine/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.0
  Built: 1629546498
  BuiltTime: Sat Aug 21 14:48:18 2021
  GitCommit: 98f252a3a1a8f1ee00f9f96c6ba00500954b5093-dirty
  GoVersion: go1.17
  OsArch: linux/amd64
  Version: 3.3.0

Package info (e.g. output of rpm -q podman or apt list podman):

Name            : podman
Version         : 3.3.0-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  device-mapper  iptables  libseccomp  crun  slirp4netns  libsystemd  fuse-overlayfs  libgpgme.so=11-64
Optional Deps   : podman-docker: for Docker-compatible CLI
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support
                  crun: support for unified cgroupsv2 [installed]
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 71.69 MiB
Packager        : Morten Linderud <[email protected]>
Build Date      : Сб 21 авг 2021 14:48:18
Install Date    : Сб 28 авг 2021 18:05:13
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

The text was updated successfully, but these errors were encountered:

mheon · 2021-08-30T18:59:55Z

Can you do a podman inspect a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad and provide the output?

Hi-Angel · 2021-08-30T19:00:59Z

Can you do a podman inspect a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad and provide the output?

Sure, it is:

 λ podman inspect a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad
WARN[0000] error determining if an image is a manifest list: error locating item named "manifest" for image with ID "a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad": file does not exist, ignoring the error 
Error: error locating item named "manifest" for image with ID "a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad": file does not exist

mheon · 2021-08-30T19:45:14Z

Do you know what the image is? Does it appear in podman images?

If you don't know what it is, I would recommend trying to podman rmi -f it and see if that resolves things.

@vrothberg PTAL

Hi-Angel · 2021-08-30T19:50:36Z

Do you know what the image is? Does it appear in podman images?

Heh, it turns out podman images doesn't work either:

Error: error locating item named "manifest" for image with ID "a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad": file does not exist

By ID alone I can't identify what that image is.

If you don't know what it is, I would recommend trying to podman rmi -f it and see if that resolves things.

I presume I could, because the only image that matters to me is running right now, which I don't think I'd lose. But note that in this case the bug would be closed, and can re-appear on someone else's machine. If you're okay with that, I can try removing that image.

vrothberg · 2021-08-31T08:13:57Z

Thanks for reaching out, @Hi-Angel.

This error can occur when the manifest of an image isn't present in the local storage. I do not know how that can happen other than a podman-{pull,build} (i.e., or any other process copying an image to the storage) getting killed at a very specific time.

I agree that podman rmi -f $ID is the best way to resolve that issue. Can you try doing that and report back if it resolved your issue? We should probably improve the error messaging there and ask users to remove (or repull) the image. We have improved the errors for similar situations as well where images could be corrupted.

Hi-Angel · 2021-08-31T08:17:54Z

Thanks! Doing podman rmi -f $ID helped to get rid of this error.

vrothberg · 2021-08-31T08:21:14Z

Thanks for confirming, @Hi-Angel. I will reopen the issue, so that we can improve the error messaging.

vrothberg · 2021-08-31T08:30:32Z

Moving it over to containers/common since it's something for libimage.

openshift-ci · 2021-08-31T08:30:45Z

@Hi-Angel: The label(s) kind/bug cannot be applied, because the repository doesn't have them.

In response to this:

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman system df results in

Error: error locating item named "manifest" for image with ID "a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad": file does not exist

I don't have any unusual setups, nothing I'd deem to be responsible for odd behavior, so I figured it's a bug worth reporting.

Steps to reproduce the issue:

Run podman system df

Describe the results you received:

Error: error locating item named "manifest" for image with ID "a249f971778eb31c10a8fb97f7a3c54f8549731ccac39accd0c02f358cb589ad": file does not exist

Describe the results you expected:

Information of how much disk space podman images take.

Output of podman version:

Version:      3.3.0
API Version:  3.3.0
Go Version:   go1.17
Git Commit:   98f252a3a1a8f1ee00f9f96c6ba00500954b5093-dirty
Built:        Sat Aug 21 14:48:18 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

Details

host:
 arch: amd64
 buildahVersion: 1.22.3
 cgroupControllers: []
 cgroupManager: systemd
 cgroupVersion: v2
 conmon:
   package: /usr/bin/conmon is owned by conmon 1:2.0.29-1
   path: /usr/bin/conmon
   version: 'conmon version 2.0.29, commit: 7e6de6678f6ed8a18661e1d5721b81ccee293b9b'
 cpus: 4
 distribution:
   distribution: arch
   version: unknown
 eventLogger: journald
 hostname: constantine-N61Ja
 idMappings:
   gidmap:
   - container_id: 0
     host_id: 1000
     size: 1
   - container_id: 1
     host_id: 165536
     size: 65536
   uidmap:
   - container_id: 0
     host_id: 1000
     size: 1
   - container_id: 1
     host_id: 165536
     size: 65536
 kernel: 5.13.13-zen1-1-zen
 linkmode: dynamic
 memFree: 458891264
 memTotal: 8229945344
 ociRuntime:
   name: crun
   package: /usr/bin/crun is owned by crun 0.21-1
   path: /usr/bin/crun
   version: |-
     crun version 0.21
     commit: c4c3cdf2ce408ed44a9e027c618473e6485c635b
     spec: 1.0.0
     +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
 os: linux
 remoteSocket:
   path: /run/user/1000/podman/podman.sock
 security:
   apparmorEnabled: false
   capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
   rootless: true
   seccompEnabled: true
   seccompProfilePath: /etc/containers/seccomp.json
   selinuxEnabled: false
 serviceIsRemote: false
 slirp4netns:
   executable: /usr/bin/slirp4netns
   package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
   version: |-
     slirp4netns version 1.1.12
     commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
     libslirp: 4.6.1
     SLIRP_CONFIG_VERSION_MAX: 3
     libseccomp: 2.5.1
 swapFree: 6585036800
 swapTotal: 8053059584
 uptime: 51h 42m 15.1s (Approximately 2.12 days)
registries:
 search:
 - docker.io
 - registry.fedoraproject.org
 - quay.io
 - registry.access.redhat.com
 - registry.centos.org
store:
 configFile: /home/constantine/.config/containers/storage.conf
 containerStore:
   number: 144
   paused: 0
   running: 1
   stopped: 143
 graphDriverName: overlay
 graphOptions:
   overlay.mount_program:
     Executable: /usr/bin/fuse-overlayfs
     Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.7.1-1
     Version: |-
       fusermount3 version: 3.10.4
       fuse-overlayfs: version 1.7.1
       FUSE library version 3.10.4
       using FUSE kernel interface version 7.31
 graphRoot: /home/constantine/.local/share/containers/storage
 graphStatus:
   Backing Filesystem: btrfs
   Native Overlay Diff: "false"
   Supports d_type: "true"
   Using metacopy: "false"
 imageStore:
   number: 127
 runRoot: /run/user/1000/containers
 volumePath: /home/constantine/.local/share/containers/storage/volumes
version:
 APIVersion: 3.3.0
 Built: 1629546498
 BuiltTime: Sat Aug 21 14:48:18 2021
 GitCommit: 98f252a3a1a8f1ee00f9f96c6ba00500954b5093-dirty
 GoVersion: go1.17
 OsArch: linux/amd64
 Version: 3.3.0

Package info (e.g. output of rpm -q podman or apt list podman):

Name            : podman
Version         : 3.3.0-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  device-mapper  iptables  libseccomp  crun  slirp4netns  libsystemd  fuse-overlayfs  libgpgme.so=11-64
Optional Deps   : podman-docker: for Docker-compatible CLI
                 btrfs-progs: support btrfs backend devices [installed]
                 catatonit: --init flag support
                 crun: support for unified cgroupsv2 [installed]
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 71.69 MiB
Packager        : Morten Linderud <[email protected]>
Build Date      : Сб 21 авг 2021 14:48:18
Install Date    : Сб 28 авг 2021 18:05:13
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Make sure to check an image for corruption before running disk usage on it. Such checks are already applied on various execution paths but not yet on disk usage. Further update the corrupted-image error to include that the image should be removed to resolve the error. This should ultimately guide users to resolve the issue. Fixes: containers#751 Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg · 2021-08-31T09:11:50Z

Opened #752

Make sure to check an image for corruption before running disk usage on it. Such checks are already applied on various execution paths but not yet on disk usage. Further update the corrupted-image error to include that the image should be removed to resolve the error. This should ultimately guide users to resolve the issue. Fixes: containers#751 Signed-off-by: Valentin Rothberg <[email protected]>

Hi-Angel closed this as completed Aug 31, 2021

vrothberg reopened this Aug 31, 2021

vrothberg self-assigned this Aug 31, 2021

vrothberg transferred this issue from containers/podman Aug 31, 2021

vrothberg mentioned this issue Aug 31, 2021

libimage: disk usage: catch corrupted images #752

Merged

rhatdan closed this as completed in #752 Aug 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`system df` fails with `error locating item named "manifest"` #751

`system df` fails with `error locating item named "manifest"` #751

Hi-Angel commented Aug 30, 2021

mheon commented Aug 30, 2021

Hi-Angel commented Aug 30, 2021 •

edited

Loading

mheon commented Aug 30, 2021

Hi-Angel commented Aug 30, 2021

vrothberg commented Aug 31, 2021

Hi-Angel commented Aug 31, 2021

vrothberg commented Aug 31, 2021

vrothberg commented Aug 31, 2021

openshift-ci bot commented Aug 31, 2021

vrothberg commented Aug 31, 2021

system df fails with error locating item named "manifest" #751

system df fails with error locating item named "manifest" #751

Comments

Hi-Angel commented Aug 30, 2021

mheon commented Aug 30, 2021

Hi-Angel commented Aug 30, 2021 • edited Loading

mheon commented Aug 30, 2021

Hi-Angel commented Aug 30, 2021

vrothberg commented Aug 31, 2021

Hi-Angel commented Aug 31, 2021

vrothberg commented Aug 31, 2021

vrothberg commented Aug 31, 2021

openshift-ci bot commented Aug 31, 2021

vrothberg commented Aug 31, 2021

`system df` fails with `error locating item named "manifest"` #751

`system df` fails with `error locating item named "manifest"` #751

Hi-Angel commented Aug 30, 2021 •

edited

Loading