CentOS Stream 9 and ssh-rsa

[lsm5]

Yep, I tried this again today using a CentOS Stream 9 VM trying to podman image scp to a RHEL 9 Beta machine using a 2048 bit RSA key.

CentOS Stream 9 machine info

[jwboyer@localhost ~]$ cat /etc/os-release 
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
[jwboyer@localhost ~]$ rpm -q podman
podman-4.0.3-1.el9.x86_64
[jwboyer@localhost ~]$

SSH connection with the key working:

[jwboyer@localhost ~]$ ssh -A 192.168.122.170
Warning: Permanently added '192.168.122.170' (ED25519) to the list of known hosts.
Web console: https://localhost:9090/ or https://192.168.122.170:9090/

Last login: Mon Apr 25 13:02:35 2022 from 192.168.122.1
[jwboyer@localhost ~]$ exit
[jwboyer@localhost ~]$ update-crypto-policies --show
DEFAULT
[jwboyer@localhost ~]$ 

Podman connection add

[jwboyer@localhost ~]$

podman image scp failing with handshake issue

[jwboyer@localhost ~]$ podman pull ubi8
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 4eef1fa1f1c1 done  
Copying blob eb24191cef20 done  
Copying config c54243b588 done  
Writing manifest to image destination
Storing signatures
c54243b58814cd424740dfebb046f356ba3acc23f04e04ffba60004eb1e8b0ea
[jwboyer@localhost ~]$ podman image scp ubi8 CONNECTION::
Copying blob 30adffdbd388 done  
Copying blob 0804b3644b85 done  
Copying config c54243b588 done  
Writing manifest to image destination
Storing signatures
Key Passphrase: 
Error: failed to connect: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
[jwboyer@localhost ~]$ 

RHEL 9 VM info

[jwboyer@localhost ~]$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="9.0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.0"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.0 Beta (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.0 Beta"
[jwboyer@localhost ~]$ rpm -q podman
podman-4.0.2-4.el9_0.x86_64
[jwboyer@localhost ~]$ update-crypto-policies --show
DEFAULT
[jwboyer@localhost ~]$ 

As you can see, an SSH works fine between the machines with the same key, but podman image scp fails with the handshake issue. RSA keys aren't deprecated in CS9/RHEL9 and the regular ssh connection works fine.

Originally posted by @jwboyer in #8323 (comment)

[lsm5]

alright, so looks like GCE's c9s instances have crypto-policies set to LEGACY by default. I can reproduce the issue in podman system connection add by switching to DEFAULT.

[lsm5]

@jwboyer This seems to be resolved in main branch of podman (I suspect in the recent update to golang.org/x/crypto/ssh which also included support for sha2). Our rhcontainerbot/podman-next copr builds the latest main branch for c9s. When you install podman from there, you might notice some hiccups with the catatonit package, but allowerasing should work fine.

[lsm5]

so afaict, backporting the right commit to v4.0-rhel branch should suffice.

[lsm5]

Fixed in commit 9049500 in v4.0-rhel branch.

@jnovy any idea when we can have newer builds of v4.0-rhel with this commit in centos stream 9 ?

[jnovy]

Hi @lsm5 CentOS Stream 9 is following latest releases of podman which means it is now podman-4.0.3. It doesn't follow maintenance branches as CentOS Stream 9 is a rolling stream. Assuming @mheon is going to release 4.1rc1 within days. Once done it will appear in c9s too.

[lsm5]

Closing this. Issue is resolved upstream and fix will land in CentOS whenever we're ready to build.