Unable to inspect image manifests in private repository on Mac

[Charleeboyle]

Issue Description

podman manifest inspect <image>
Error: getting content of manifest list or image <image> reading image "docker://<image>": Requesting bearer token: invalid status code from registry 400 (Bad Request)

The image is hosted in a private image repository. This image is mirrored to a public image repository using skopeo. I am able to run podman manifest inspect <image> on the image hosted in the public repository, but not on the image hosted in the private repository.

Note: I am able to pull the image from the private repository with podman pull

This seems to be an issue with running the command on Mac, as the command works inside of a Linux VM.

podman logout <repository> followed by podman login <repository> does not resolve the issue.

podman version

podman version
Client:       Podman Engine
Version:      4.5.1
API Version:  4.5.1
Go Version:   go1.20.4
Git Commit:   9eef30051c83f62816a1772a743e5f1271b196d7
Built:        Fri May 26 16:10:12 2023
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      4.5.1
API Version:  4.5.1
Go Version:   go1.20.4
Built:        Fri May 26 18:58:48 2023
OS/Arch:      linux/amd64

Steps to reproduce the issue

Steps to reproduce the issue

1. Run podman manifest inspect <image> where is hosted in a private image repository. On MacOS.

Describe the results you received

Error: getting content of manifest list or image reading image "docker://": Requesting bearer token: invalid status code from registry 400 (Bad Request)

Describe the results you expected

{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
......
]
}

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 94.33
    systemPercent: 3.78
    userPercent: 1.89
  cpus: 1
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.3.8-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1226952704
  memTotal: 2048794624
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 9m 50.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 2399244288
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685123928
  BuiltTime: Fri May 26 18:58:48 2023
  GitCommit: ""
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

Please update you podman versions, both the client and server are out of date. When you have a new client you likely need to do podman machine rm followed by podman machine init to recreate the VM with a newer podman version.

[Charleeboyle]

My apologies, I thought I had the most up-to-date version. The error still persists on 4.5.1. I have updated the podman version and podman info command output above.

[Luap99]

Thanks for checking.

@vrothberg @mtrmac PTAL

[vrothberg]

I didn´t double check in the code but my suspicion is that manifest inspect doesn't set the auth header.

[mtrmac]

Yes; pkg/bindings/manifests/manifests.go:InspectListData does not invoke podman/pkg/auth. Same for the server-side handler.

---

“invalid status code from registry 400 (Bad Request)” seems a bit unexpected in this situation, but, meh, we know registry error reporting is inconsistent, and anyway we need to pass the credentials through before we can worry about other possible problem causes. And the report says that it works fine from inside the Linux VM, so it really seems that the cause is missing credentials.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Luap99]

I think this was fixed in #19180?