Inconsistent treatment of manifest lists in "podman run"

[avikivity]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

A podman run with the image pointing to a local manifest list will fail.

A podman run with the image pointing to a manifest list in a registry will succeed.

At best this is confusing. At worst, it means there is no way to extend a manifest list, since it will be treated differently if created locally or pulled, and I have to remove it in order to use it.

Steps to reproduce the issue:

Reproducer script:

#!/bin/bash -ex

M=docker.io/avikivity/test-manifest

podman pull docker.io/fedora:32
podman manifest create "$M"
podman manifest add --all "$M" docker.io/fedora:32
podman manifest push --all "$M" docker://"$M"
podman run --rm "$M" echo y || echo failed running image from manifest
podman rmi "$M"
podman run --rm "$M" echo y && echo succeeded running image from manifest

1. Edit the reproducer, substituting an unused name that you can push to for M.

2. Run the reproducer

3. Wonder why the same podman run command fails or succeeds depending on local state

Describe the results you received:

failed running image from manifest
succeeded running image from manifest

Describe the results you expected:

succeeded running image from manifest

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.0.4
API Version:  1
Go Version:   go1.14.6
Built:        Thu Jan  1 02:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.19-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.19, commit: 5dce9767526ed27f177a8fa3f281889ad509fea7'
  cpus: 12
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: avi
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.8-200.fc32.x86_64
  linkmode: dynamic
  memFree: 2141855744
  memTotal: 33542152192
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 15166337024
  swapTotal: 16869486592
  uptime: 359h 10m 34.1s (Approximately 14.96 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/avi/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/avi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 33
  runRoot: /run/user/1000/containers
  volumePath: /home/avi/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 02:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.6
  OsArch: linux/amd64
  Version: 2.0.4

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.4-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

@QiWang19 PTAL

[QiWang19]

Create & Run using a manifest returns error var ErrImageIsBareList = stderrors.New("image contains a manifest list or image index, but no runnable image") , is this as expected?
@rhatdan @nalind ?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I would think not, but will wait for @nalind to respond.

[QiWang19]

Podman haven't implemented to pick one image from the list of images from a manifest list, just have checked with @nalind that had been supported in cri-o, will look into it.

[nalind]

No, it isn't expected. It falls out of the current implementation, where the name is associated with the manifest list and not the images that are mentioned in it. I think we have most of what would be needed to make this work if the runnable image was in local storage (I think the only missing piece is tagging the runnable image with a digested reference that matches the name that's been assigned to the list, e.g. repository@digest), but the reproducer script isn't pulling them down (manifest add doesn't pull layers), so there's no local image to run. I don't have a guess as to whether or not we want to be pulling the image in that case.

[QiWang19]

if the manifest list exists locally, try with --pull always can success podman run --pull always --rm "$M", but is this enough for this issue?

[avikivity]

--pull always works, but I can't tell my users to to use that.

I can rmi the local manifest as a workaround.