Podman search does not use token authentication

[dcermak]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Podman search does not use token authentication when contacting a registry that requires authentication for searching.

Steps to reproduce the issue:

1. Add registry.suse.com to unqualified-search-registries in /etc/containers/storage.conf

2. podman search bci-base

Describe the results you received:

ERRO[0000] error getting search results from v2 endpoint "registry.suse.com": unable to retrieve auth token: invalid username/password: errors:
denied: requested access to the resource is denied
unauthorized: authentication required 
Error: 1 error occurred:
        * couldn't search registry "registry.suse.com": unable to retrieve auth token: invalid username/password: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

Describe the results you expected:

A search result.

Additional information you deem important (e.g. issue happens only occasionally):

The issue seems to be related to the fact that registry.suse.com requires authentication for all users, even for anonymous searches. It responds to an initial request with a www-authenticate header, but maybe it is ignored by podman?

🕙[ 12:24:19 ] ❯ curl -v https://registry.suse.com/v2/
*   Trying 35.156.195.197:443...
* Connected to registry.suse.com (35.156.195.197) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=registry.suse.com
*  start date: Nov  2 00:00:00 2021 GMT
*  expire date: Dec  1 23:59:59 2022 GMT
*  subjectAltName: host "registry.suse.com" matched cert's "registry.suse.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x560220f66dc0)
> GET /v2/ HTTP/2
> Host: registry.suse.com
> user-agent: curl/7.79.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 401 
< date: Mon, 28 Feb 2022 11:25:00 GMT
< content-type: application/json
< content-length: 87
< docker-distribution-api-version: registry/2.0
< www-authenticate: Bearer realm="https://registry.suse.com/auth",service="SUSE Linux Docker Registry"
< x-content-type-options: nosniff
< 
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host registry.suse.com left intact

Output of podman version:

Client:       Podman Engine
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.18beta1
Git Commit:   c39dffe83db9fa3cfa6897b971956821f1bbcce2
Built:        Mon Feb 28 12:21:51 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 12
  distribution:
    distribution: fedora
    version: "35"
  eventLogger: journald
  hostname: Boreas
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 100000
      size: 65536
  kernel: 5.16.10-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 8078229504
  memTotal: 33319403520
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 12h 18m 23.12s (Approximately 0.50 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
  - registry.suse.com
store:
  configFile: /home/dan/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /home/dan/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1388
  runRoot: /run/user/1000/containers
  volumePath: /home/dan/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.0-dev
  Built: 1646047311
  BuiltTime: Mon Feb 28 12:21:51 2022
  GitCommit: c39dffe83db9fa3cfa6897b971956821f1bbcce2
  GoVersion: go1.18beta1
  OsArch: linux/amd64
  Version: 4.0.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.4-1.fc35.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[rhafer]

@dcermak You might want to take a look here: containers/image#1079

[rhatdan]

@mtrmac PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@vrothberg @mtrmac thoughts?

[vrothberg]

containers/image#1478

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@vrothberg any movement on this?

[vrothberg]

Nope. Let's keep the conversation in containers/image#1478 where it needs to be fixed first.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mtrmac]

For the record, this should be fixed in containers/image#1640 (after Podman updates c/image).

[vrothberg]

Closing as containers/image#1640 has been merged into Podman.