Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Newer OpenSSH clients are dropping support for ssh-rsa-cert-v01 #10918

Closed
webvictim opened this issue Mar 7, 2022 · 18 comments · Fixed by #19579
Closed

Newer OpenSSH clients are dropping support for ssh-rsa-cert-v01 #10918

webvictim opened this issue Mar 7, 2022 · 18 comments · Fixed by #19579
Assignees
Labels
bug c-10 Internal Customer Reference c-bl Internal Customer Reference c-cc Internal Customer Reference c-ju Internal Customer Reference c-m Internal Customer Reference c-sn Internal Customer Reference c-te Internal Customer Reference OpenSSH For customers using Teleport and OpenSSH sales-onboarding Issues related to prospects server-access

Comments

@webvictim
Copy link
Contributor

webvictim commented Mar 7, 2022

Description

What happened:

Fedora 35 bundles OpenSSH 8.7 and the default system crypto policy has been changed to remove ssh-rsa and ssh-rsa-cert-v01 from the approved list of PubKeyAcceptedAlgorithms:

[vagrant@fedora35 ~]$ grep PubkeyAcceptedAlgorithms /etc/crypto-policies/back-ends/openssh.config | tr "," "\n"
PubkeyAcceptedAlgorithms ecdsa-sha2-nistp256
[email protected]
[email protected]
[email protected]
ecdsa-sha2-nistp384
[email protected]
ecdsa-sha2-nistp521
[email protected]
ssh-ed25519
[email protected]
[email protected]
[email protected]
rsa-sha2-256
[email protected]
rsa-sha2-512
[email protected]

This breaks ssh connectivity to Teleport clusters when using tsh config with an auto-generated config file:

[vagrant@fedora35 ~]$ ssh ubuntu@ip-10-0-131-171
[email protected]: Permission denied (publickey).
ERROR: exit status 255

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Expanding the ProxyCommand out to run ssh manually and using -v, we can see that the underlying error is actually no mutual signature algorithm:

[vagrant@fedora35 ~]$ ssh -v -p 3023 [email protected] -s proxy:ip-10-0-131-171:3022
OpenSSH_8.7p1, OpenSSL 1.1.1l  FIPS 24 Aug 2021
debug1: Reading configuration data /home/vagrant/.ssh/config
debug1: /home/vagrant/.ssh/config line 6: Applying options for example.teleport.com
debug1: /home/vagrant/.ssh/config line 13: Skipping Host block because of negated match for example.teleport.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/vagrant/.ssh/config
debug1: /home/vagrant/.ssh/config line 6: Applying options for example.teleport.com
debug1: /home/vagrant/.ssh/config line 13: Skipping Host block because of negated match for example.teleport.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to example.teleport.com [1.2.3.4] port 3023.
debug1: Connection established.
debug1: identity file /home/vagrant/.tsh/keys/example.teleport.com/gus-local type 0
debug1: certificate file /home/vagrant/.tsh/keys/example.teleport.com/gus-local-ssh/purple-cert.pub type 4
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version Teleport
debug1: compat_banner: no match: Teleport
debug1: Authenticating to example.teleport.com:3023 as 'ubuntu'
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: [email protected]
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: [email protected] need=64 dh_need=64
debug1: kex: [email protected] need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host certificate: [email protected] SHA256:3Sl9kSR4U+ovsjM6OKaD4JPuAxFBcWlGelnAQOm6P0E, serial 0 ID "" CA ssh-rsa SHA256:s4nqYP8iLSHc9veGtJEie8ndQM6JDDxUvf6EuvCfFCo valid after 2022-01-28T17:59:23
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: checking without port identifier
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'example.teleport.com' is known and matches the RSA-CERT host certificate.
debug1: Found CA key in /home/vagrant/.tsh/known_hosts:1
debug1: found matching key w/out port
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/vagrant/.tsh/keys/example.teleport.com/gus-local-ssh/purple-cert.pub RSA-CERT SHA256:s8tjExcDL3bXMT9Wv9tKkO08tI8ZUZas0ZWd1rNpMmI explicit
debug1: Will attempt key: /home/vagrant/.tsh/keys/example.teleport.com/gus-local RSA SHA256:s8tjExcDL3bXMT9Wv9tKkO08tI8ZUZas0ZWd1rNpMmI explicit agent
debug1: Will attempt key: teleport:gus-local RSA-CERT SHA256:s8tjExcDL3bXMT9Wv9tKkO08tI8ZUZas0ZWd1rNpMmI agent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/vagrant/.tsh/keys/example.teleport.com/gus-local-ssh/purple-cert.pub RSA-CERT SHA256:s8tjExcDL3bXMT9Wv9tKkO08tI8ZUZas0ZWd1rNpMmI explicit
-debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: /home/vagrant/.tsh/keys/example.teleport.com/gus-local RSA SHA256:s8tjExcDL3bXMT9Wv9tKkO08tI8ZUZas0ZWd1rNpMmI explicit agent
-debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: teleport:gus-local RSA-CERT SHA256:s8tjExcDL3bXMT9Wv9tKkO08tI8ZUZas0ZWd1rNpMmI agent
-debug1: send_pubkey_test: no mutual signature algorithm
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).

Workaround

The current workaround is to add these two lines to the first section of the generated ~/.ssh/config file:

#
# Begin generated Teleport configuration for example.teleport.com:443 from `tsh config`
#

# Common flags for all purple hosts
Host *.purple example.teleport.com
    UserKnownHostsFile "/home/vagrant/.tsh/known_hosts"
    IdentityFile "/home/vagrant/.tsh/keys/example.teleport.com/gus-local"
    CertificateFile "/home/vagrant/.tsh/keys/example.teleport.com/gus-local-ssh/purple-cert.pub"
+   HostKeyAlgorithms [email protected]
+   PubkeyAcceptedAlgorithms [email protected]

# Flags for all purple hosts except the proxy
Host *.purple !example.teleport.com
    Port 3022
    ProxyCommand "/usr/local/bin/tsh" proxy ssh --cluster=purple --proxy=example.teleport.com %r@%h:%p

# End generated Teleport configuration

On Fedora, you can also run sudo update-crypto-policies --set LEGACY to switch back to the older crypto policies which still trust ssh-rsa-cert-v01 and ssh-rsa.

What you expected to happen: Connectivity should work out of the box. Either Teleport should upgrade to a newer SSH certificate format, or automatically add the PubkeyAcceptedKeyTypes line to generated configs with tsh config.

Reproduction Steps

Run above steps on a Fedora 35 machine.

Server Details

  • Teleport version (run teleport version): Teleport v8.3.4 git:v8.3.4-0-g010bea1 go1.17.3
  • Server OS (e.g. from /etc/os-release): Fedora 35
  • Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): Vagrant
  • Additional details:

Client Details

  • Tsh version (tsh version): Teleport v8.3.4 git:v8.3.4-0-g010bea1 go1.17.3
@webvictim webvictim added bug OpenSSH For customers using Teleport and OpenSSH labels Mar 7, 2022
@stevenGravy stevenGravy added the sales-onboarding Issues related to prospects label Mar 7, 2022
@ArunNadda ArunNadda added the c-te Internal Customer Reference label Mar 16, 2022
@LocalOracularBunny
Copy link

Same problem/fix on Ubuntu Jammy

@frozenice
Copy link

Just encountered this too (same errors) with:

  • Teleport v9.1.1 (using Machine ID / tbot)
  • Alpine Linux 3.15.4 with ssh OpenSSH_8.8p1, OpenSSL 1.1.1n 15 Mar 2022

It worked when I added PubkeyAcceptedKeyTypes [email protected] to the generated /opt/machine-id/ssh_config file (appended to the first Host block, or when appending a new Host * block with that entry to the end of the file).

Currently wondering, how I can actually use that workaround from my CI pipeline, as /opt/machine-id is mounted ro (I could mount it rw for now, until this is fixed).

Maybe a config setting for tbot would be good, so one can add more options to the first Host block, something like:

...
destinations
  - directory: ...
    configs:
      - ssh_client:
          additional_options:
            - "PubkeyAcceptedKeyTypes [email protected]"

@webvictim
Copy link
Contributor Author

@frozenice AFAIK the machine ID config block is the same every time it’s generated, so you could potentially take a copy of the file, make edits and then mount it as a secret/similar to use the edited version rather than the one in the output directory for now.

@ArunNadda ArunNadda added the c-10 Internal Customer Reference label May 3, 2022
@adam-azarchs
Copy link

Yes, the immediate problem here is that they've dropped support for known-insecure key signing algorithms in favor of sha2 variants, but the broader problem is that RSA in general is obsolete. While for the time being attacks on correctly-implemented RSA are still mostly theoretical (for anyone without a quantum computer, anyway, so for maybe another 3 years), it would be good to have support for more up-to-date algorithms, specifically ed25519, before some future update to OpenSSH drops RSA entirely.

@KasperSkytte
Copy link

Just wanted to +1. Encountering the same issue on Pop!_OS (based on Ubuntu Jammy). Suggested workaround fixes it

@0SkillAllLuck
Copy link

Running into this issue on MacOS Ventura swell. Workaround is sadly not working

@tbenz9
Copy link

tbenz9 commented Sep 27, 2022

I also ran into this issue on Ubuntu 22.04. The workaround above didn't exactly work but adding the following lines to my /etc/ssh/sshd_config file did work.

PubkeyAcceptedAlgorithms [email protected]
CASignatureAlgorithms +ssh-rsa

@jakule jakule self-assigned this Oct 4, 2022
@jakule
Copy link
Contributor

jakule commented Oct 10, 2022

I also ran into this issue on Ubuntu 22.04. The workaround above didn't exactly work but adding the following lines to my /etc/ssh/sshd_config file did work.

PubkeyAcceptedAlgorithms [email protected]
CASignatureAlgorithms +ssh-rsa

@tbenz9 You should add these lines to your ssh config, not sshd. Your local ssh config should be located in ~/.ssh/config and you can generate it by using tsh config >> ~/.ssh/config. This command will append the correct config to the file.

@webvictim
Copy link
Contributor Author

/etc/ssh/ssh_config is also the system-wide SSH client config (notably a different file to sshd_config)

@FrankWeissmann FrankWeissmann added the c-sn Internal Customer Reference label Oct 25, 2022
@Kilhog
Copy link

Kilhog commented Oct 27, 2022

Running into this issue on MacOS Ventura swell. Workaround is sadly not working

For macOS Ventura you need to edit the file : /private/etc/ssh/ssh_config and add the following lines in Host * block :

PubkeyAcceptedAlgorithms [email protected]
CASignatureAlgorithms +ssh-rsa

@kylern kylern added the c-m Internal Customer Reference label Oct 27, 2022
@2pravin7
Copy link

Running into this issue on MacOS Ventura swell. Workaround is sadly not working

For macOS Ventura you need to edit the file : /private/etc/ssh/ssh_config and add the following lines in Host * block :

PubkeyAcceptedAlgorithms [email protected]
CASignatureAlgorithms +ssh-rsa

This worked like a charm. Thanks!

@hookbot
Copy link

hookbot commented Nov 3, 2022

Running into this issue on MacOS Ventura swell. Workaround is sadly not working

For macOS Ventura you need to edit the file : /private/etc/ssh/ssh_config and add the following lines in Host * block :
...

After upgrading MacOSX to Ventura, I was suddenly unable into ssh to my RSA SSHD servers.

$ ssh 10.0.0.1
Unable to negotiate with 10.0.0.1 port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss
$

I ran your fix to allow me to connect:

echo 'HostKeyAlgorithms +ssh-rsa' | sudo tee -a /etc/ssh/ssh_config

Sure, this is better, but it still doesn't honor my id_rsa file. :-/ It still requires a password (which I do NOT have configured on the SSH server for security reasons).

$ ssh 10.0.0.1
[email protected]'s password:

So then I also had to run this:

echo 'PubkeyAcceptedKeyTypes +ssh-rsa' | sudo tee -a /etc/ssh/ssh_config

And then everything suddenly started working perfectly again:

$ ssh 10.0.0.1
Last login: Thu Nov  3 04:26:00 2022 from 10.0.0.2
[hookbot@server ~]$

Sure, I realize that RSA isn't as secure, but upgrading the SSHD server on this device is not an option for me.

@pschisa pschisa added the c-cc Internal Customer Reference label Nov 16, 2022
@tcannonfodder
Copy link

Running into this issue on MacOS Ventura swell. Workaround is sadly not working

For macOS Ventura you need to edit the file : /private/etc/ssh/ssh_config and add the following lines in Host * block :

PubkeyAcceptedAlgorithms [email protected]
CASignatureAlgorithms +ssh-rsa

This worked like a charm. Thanks!

This should only be added to ~/.ssh/config to help isolate the fix; since it's inherently not as secure. That way, you don't misplace the patch once it can be removed

@flo-compbio
Copy link

Can confirm workaround works if you include the following in ssh/ssh_config:

Host *
  HostKeyAlgorithms [email protected]
  PubkeyAcceptedAlgorithms [email protected]

@pschisa pschisa added the c-bl Internal Customer Reference label Mar 14, 2023
@fghamsary
Copy link

But this is not really resolved! Has it?
This was just a work around to be using an obsolete certificate type, when are you going to really resolve the problem by adding new certificates which are not obsolete, or if it's possible just point this out here on this issue.

@zoni
Copy link

zoni commented Aug 1, 2023

@fghamsary it should have been resolved, see #15149 (comment)

@fghamsary
Copy link

fghamsary commented Aug 1, 2023

@fghamsary it should have been resolved, see #15149 (comment)

We already have later version than 12 and it is not the case. The ticket was that teleport doesn't use standard SSH clients now and use their own fork version, but the connection created is still with RSA-Cert which is not secure anymore!

Here is the connection to our proxy:
$ tsh version
Teleport v12.4.12 git:api/v12.4.12-0-gdea6fce go1.20.6
Proxy version: 12.3.1

@webvictim
Copy link
Contributor Author

but the connection created is still with RSA-Cert which is not secure anymore!

This is a misunderstanding. RSA keys themselves are secure, but connecting with RSA keys using a SHA1 hash signing algorithm is not.

See the "Imminent deprecation notice" section of the OpenSSH 8.7 release notes for clarity:

"Note that the deactivation of ssh-rsa signatures does not necessarily require cessation of use for RSA keys. In the SSH protocol, keys may be capable of signing using multiple algorithms. In particular, ssh-rsa keys are capable of signing using rsa-sha2-256 (RSA/SHA256), rsa-sha2-512 (RSA/SHA512) and ssh-rsa (RSA/SHA1). Only the last of these is being turned off by default."

Teleport now uses rsa-sha2-512 by default and also supports rsa-sha2-256.

ssh-rsa is no longer used by default, but Teleport users who have upgraded from earlier versions may have to perform a CA rotation to migrate their CA away from SHA1. This is not done by default as it is a disruptive operation which may require longer-lived user and host certificates to be re-issued.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug c-10 Internal Customer Reference c-bl Internal Customer Reference c-cc Internal Customer Reference c-ju Internal Customer Reference c-m Internal Customer Reference c-sn Internal Customer Reference c-te Internal Customer Reference OpenSSH For customers using Teleport and OpenSSH sales-onboarding Issues related to prospects server-access
Projects
None yet