upload_forcing to orcinus and optimum fails "Private key file is encrypted"

[douglatornell]

This has often been due to a network stack issue on one of the orcinus login nodes, but that case is transient.
This is happening consistently and reproducibly.
Appeared in new env created 2-Mar-2022 for emergency deployment after /SalishSeaCast SSD failure.
paramiko version that shows issue is 2.9.2.
pandas version in previous working env was likely 2.7.2.

Traceback:

2022-03-07 09:55:40,940 CRITICAL [upload_forcing] unhandled exception:
Traceback (most recent call last):
  File "/data/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work
    checklist = self.worker_func(
  File "/data/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 92, in upload_forcing
    ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key)
  File "/data/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 121, in sftp
    ssh_client = ssh(host, key_filename, ssh_config_file)
  File "/data/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 68, in ssh
    ssh_client.connect(
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/client.py", line 435, in connect
    self._auth(
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/client.py", line 766, in _auth
    raise saved_exception
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/client.py", line 736, in _auth
    key = self._key_from_filepath(
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/client.py", line 588, in _key_from_filepath
    key = klass.from_private_key_file(key_path, password)
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/pkey.py", line 249, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/rsakey.py", line 64, in __init__
    self._from_private_key_file(filename, password)
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/rsakey.py", line 190, in _from_private_key_file
    data = self._read_private_key_file("RSA", filename, password)
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/pkey.py", line 322, in _read_private_key_file
    data = self._read_private_key(tag, f, password)
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/pkey.py", line 348, in _read_private_key
    data = self._read_private_key_pem(lines, end, password)
  File "/data/SalishSeaCast/nowcast-env/lib/python3.10/site-packages/paramiko/pkey.py", line 400, in _read_private_key_pem
    raise PasswordRequiredException("Private key file is encrypted")
paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted

[douglatornell]

Root cause appears to be that, as of paramiko=2.9.0, it's ssh client expects to expects to receive a server-sig-algs message from the server telling it what key signature algorithms it supports. If it doesn't get server-sig-algs, the client defaults to using rsa-sha2-512. Unfortunately, optimum and orcinus appear to be running verisons of OpenSSH which don't send server-sig-algs, and don't support rsa-sha2-512. server-sig-algs appears to have been implemented in OpenSSH v7.4.

optimum is running OpenSSH v5.3.
orcinus is running OpenSSH v6.4.

Adding a disabled_algorithms={'pubkeys': ['rsa-sha2-512', 'rsa-sha2-256']} kwarg to paramiko.client.SSHClient().connect() appears to resolve the issue.