docker login with docker.io creds "successfully" logs-into registry.fedoraproject.org then fails to push to registry-1.docker.io

[azdle]

Issue Description

First off, I'm not sure if this is a podman issue or a fedora package issue, but I figured this might be helpful here since there are a lot of closed, but unresolved, issues here that seem like they could be caused by this. Also, this is on 4.9.4, I haven't tried v5.

Doing a fresh setup with podman-docker installed on fedora and following docker.io's instructions for logging in and pushing my first image, I'm told my "Login Succeeded!", but when I try to push I get a "resource denied" error.

It seems running docker login -u $USER stores the access token as a token for registry.fedoraproject.org (not what I expected), but when I try to push it tries to push to registry-1.docker.io (is what I expected).

Steps to reproduce the issue

1. Install podman-docker
2. Follow docker.io's instructions to login
   a. Create access token
   b. run docker login -u $USER
   c. enter token

Describe the results you received

$ podman login -u $USER
Password:
Login Succeeded!
$ cat /run/user/1000/containers/auth.json
{
    "auths": {
	    "registry.fedoraproject.org": {
		    "auth": "cGF0cmljazpkY2tyX3BhdF84X2xvbHRoaXNhaW50cmVhbAo="
	    }
    }
}⏎
$ docker push $USER/$IMAGE
[...]
Error: writing blob: initiating layer upload to /v2/$USER/$IMAGE/blobs/uploads/ in registry-1.docker.io: requested access to the resource is denied

Describe the results you expected

$ docker login -u $USER
Password: 
Login Succeeded!
$ cat /run/user/1000/containers/auth.json
{
    "auths": {
	    "registry-1.docker.io": {
		    "auth": "cGF0cmljazpkY2tyX3BhdF84X2xvbHRoaXNhaW50cmVhbAo="
	    }
    }
}⏎
$ docker push $USER/$IMAGE
Writing manifest to image destination

^ exit success

or

Failure for the login command

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 94.97
    systemPercent: 1.49
    userPercent: 3.54
  cpus: 20
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "39"
  eventLogger: journald
  freeLocks: 2039
  hostname: brick
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.7.11-200.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 40281542656
  memTotal: 67076845568
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.x86_64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 25h 7m 53.00s (Approximately 1.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/$USER/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 2
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/$USER/.local/share/containers/storage
  graphRootAllocated: 4095101370368
  graphRootUsed: 37661007872
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 26
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/$USER/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4
  Built: 1711445992
  BuiltTime: Tue Mar 26 04:39:52 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

No response

Additional information

Workaround

For anyone else having this issue, either edit your auth.json file to say "registry-1.docker.io" or just run docker login -u $USER registry-1.docker.io instead.

[rhatdan]

@mtrmac PTAL

[mtrmac]

Not saying which registry you want to work with can only guess so much. And using Docker documentation can only be relevant to a point.

I strongly recommend specifying the registry in all operations, or maybe using specifically configured aliases. Alternatively, changing unqualified-search-registries to only contain docker.io would probably make the behavior closer to (but not identical to) Docekr’s.

(The cause here is that login defaults to the registry from https://src.fedoraproject.org/rpms/containers-common/blob/rawhide/f/registries.conf#_21 , in a a sense sense consistent with how podman pull works; but podman push does not use the same search list, and it wouldn’t make sense for it to do so. And those behaviors are very long-standing, back from projectatomic/docker, and not something we can viably change.)

[rhatdan]

Since I don't see this as an issue at this point, I have converted it to a discussion.

[mtrmac]

I think it would be a reasonable RFE to include the credential scope (registry[/repo]) in the interactive Password: prompt.

That would not help non-interactive users using --password-stdin and the like, those simply need to be careful.