skopeo copy, using podman credentials, unauthorized

[edsantiago]

New failure in podman gating tests on rawhide:

 ✗ podman login - shares credentials with skopeo - default auth file
   (from function `is' in file /usr/share/podman/test/system/helpers.bash, line 491,
    from function `_test_skopeo_credential_sharing' in file /usr/share/podman/test/system/150-login.bats, line 237,
    in test file /usr/share/podman/test/system/150-login.bats, line 266)
     `_test_skopeo_credential_sharing' failed
   # podman rm --all --force
   # podman ps --all --external --format {{.ID}} {{.Names}}
   # podman images --all --format {{.Repository}}:{{.Tag}} {{.ID}}
   quay.io/libpod/testimage:20210610 9f9ec7f2fdef
   /usr/bin/skopeo
   # podman login --tls-verify=false --username userg1An --password 9cJHxAGPVTmlk6n localhost:5428
   Login Succeeded!
   # skopeo copy ...
   time="2021-09-22T20:15:22-04:00" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
   Getting image source signatures
   time="2021-09-22T20:15:22-04:00" level=fatal msg="trying to reuse blob sha256:f36118df491fbfd96093731809941d7bb881136415ccc114bc26d6bf10499a0e at destination: checking whether a blob sha256:f36118df491fbfd96093731809941d7bb881136415ccc114bc26d6bf10499a0e exists in localhost:5428/skopeo-ok-mwz3wpt25j-ok: unauthorized: authentication required"
   #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
   #|     FAIL: skopeo copy - exit status
   #| expected: '0'
   #|   actual: '1'
   #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

skopeo-1.4.1-1.fc36 , podman-3.4.0-0.3.rc1.fc36 , kernel 5.14.0-0.rc5.20210813gitf8e6dfc64f61.46.fc36

Seems to be 100% reproducible. Does not reproduce on my f34 laptop (kernel 5.12.something).

Incomplete bug report because it's very late in the day. I can try to narrow it down tomorrow.

[mtrmac]

Thanks for your report.

I remember seeing similar failures in Skopeo’s CI from time to time, but they usually go away on a test restart. I haven’t tried to look into them yet, I’m afraid.

If this is 100% reproducible, that’s even more interesting.

From a quick check, AFAICS the last time the credential storage locations might have changed is (via c/common/pkg/auth) back in April. The registry images also don’t seem to have changed.

[edsantiago]

Ugh. This is podman, not skopeo, and I'm 99% sure that it's containers/podman#11195 because it mucked with XDG. Now what? Should I refile this under podman? Should skopeo make a similar change? @Luap99 PTAL.

[mtrmac]

Yeah, at a first glance, if I understand #10745 correctly, Podman/Conmon was manipulating XDG_RUNTIME_DIR so that it was different between network creation/deletion, and the fix was… to manipulate the value even harder? My first instinct would be to work to preserve the environment value so that the network creation/deletion uses the same values; if that were impossible, to restrict the hard-coding of XDG_RUNTIME_DIR to the problematic DNS (Conmon?) environments, rather than make a process-wide change.

But I also don’t have a good handle on all the various code paths and what happens where (e.g. containers/podman#10745 (comment) points at GetRuntimeDir() returning "" with no documentation, at a first glance, what does that even mean?!), and I appreciate there are genuine difficulties WRT /run/user, non-systemd sessions, in-container environments with no sessions at all…

[edsantiago]

This is a podman bug. Filed containers/podman#11725

[mtrmac]

@edsantiago thanks for that reproducer — there are legitimate objections to the built-in /run/containers/0 default, but it seems clear enough that when the external environment points to a valid XDG_RUNTIME_DIR, the credentials should be stored there and nowhere else.