Intermittent failures with SSH_AUTH_SOCK and many keys loaded in agent

[jschwartz-cray]

/kind bug

Description

podman intermittently fails with:

$ podman info
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: Connection to bastion host (ssh://podman@podman:22/run/user/1001/podman/podman.sock) failed.: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures

when using SSH_AUTH_SOCK, an agent with many different ssh keys loaded, e.g. 12 different ones for various different envs, servers, source control systems, etc.:

$ ssh-add -l
2048 SHA256:<.............redacted....................> <user>@<company> (RSA)
2048 SHA256:<.............redacted....................> <user>@<company> (RSA)
4096 SHA256:<.............redacted....................> <user>@<company> (RSA)
4096 SHA256:<.............redacted....................> <user>@<company> (RSA)
2048 SHA256:<.............redacted....................> <user>@<company> (RSA)
4096 SHA256:<.............redacted....................> <user>@<company> (RSA)
2048 SHA256:<.............redacted....................> <user>@<company> (RSA)
4096 SHA256:<.............redacted....................> <user>@<company> (RSA)
4096 SHA256:<.............redacted....................> <user>@<company> (RSA)
4096 SHA256:<.............redacted....................> <user>@<company> (RSA)
1024 SHA256:<.............redacted....................> <user>@<company> (RSA)
256  SHA256:<.............redacted....................> <user>@<company> (ED25519)

and a podman remote server with a relatively low MaxAuthTries set (not uncommon for security reasons, some hardening even sets this to 1).

The usual solution for this is to add IdentitiesOnly=yes to your ~/.ssh/config which I have done for my podman remote server:

Host podman
    Hostname <...>
    StrictHostKeyChecking no
    IdentitiesOnly yes
    IdentityFile ~/.ssh/podman_rsa
    User podman
    UserKnownHostsFile /dev/null

but podman does not respect this setting and I'm guessing it tries keys in varying orders resulting in the intermittent behavior of this issue.

As a workaround, specifying --identity when creating the podman system connection and removing SSH_AUTH_SOCK from the env seems to eliminate the issue, forcing just that single identity to be used (though this is undesirable due to #7806 and the workaround #7806 (comment) via #8676):

env -u SSH_AUTH_SOCK podman info

Steps to reproduce the issue:

1. setup a remote connection with no --identity
2. set MaxAuthTries on the server to a low value
3. create and ssh-copy-id an ssh key to the server
4. start an ssh-agent, setting SSH_AUTH_SOCK to point to it, and load numerous ssh keys (>MaxAuthTries, e.g. 12) into the agent including the one you copied to the server
5. repeatedly run podman info and see that it intermittently fails

Describe the results you received:

Roughly half the time when I run a podman command it fails with:

ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures

Describe the results you expected:

podman remote commands to be nearly 100% reliable provided the server is up and accessible.

Additional information you deem important (e.g. issue happens only occasionally):

N/A

Output of podman version:

$ podman version
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: Connection to bastion host (ssh://podman@podman:22/run/user/1001/podman/podman.sock) failed.: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures

(sigh)

$ podman version
Client:
Version:      3.4.1
API Version:  3.4.1
Go Version:   go1.17.2
Built:        Tue Oct 19 15:14:42 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Wed Dec 31 17:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 16
  distribution:
    distribution: ubuntu
    version: "21.04"
  eventLogger: journald
  hostname: <redacted>
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.11.0-40-generic
  linkmode: dynamic
  logDriver: ""
  memFree: 11103854592
  memTotal: 16463306752
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: <redacted>
plugins:
  log: null
  network: null
  volume: null
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1001/containers
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 0
  BuiltTime: Wed Dec 31 17:00:00 1969
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ brew info podman
podman: stable 3.4.1 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/usr/local/Cellar/podman/3.4.1 (170 files, 39.5MB) *
  Poured from bottle on 2021-10-22 at 14:28:11
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✘, go-md2man ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d
==> Analytics
install: 14,015 (30 days), 34,910 (90 days), 62,794 (365 days)
install-on-request: 14,021 (30 days), 34,912 (90 days), 62,728 (365 days)
build-error: 2 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

N/A

[mheon]

@baude @jwhonce @ashley-cui PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Emilgardis]

An easy way to fix this is to increase the

MaxAuthTries in /etc/ssh/sshd_config

$ SSH_AUTH_SOCK="" podman machine ssh <machine>

# We're now inside the machine
sudo sh -c 'echo "MaxAuthTries 30" >> /etc/ssh/sshd_config'

[cdoern]

I think this one will be fixed with containers/common#1094

[cdoern]

podman PR open using new SSH interface should fix this