Issues using podman machine with ssh agent

[sanyer]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When ssh agent is running, podman machine connection sometimes fails because of Too many failures:

$ podman --log-level debug play kube myapp.yaml --down
INFO[0000] podman filtering at log level debug
DEBU[0000] Called kube.PersistentPreRunE(podman --log-level debug play kube myapp.yaml --down)
DEBU[0000] SSH Ident Key "/Users/<username>/.ssh/podman-machine-default" SHA256:Y<sha> ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.23Vo8gvmZn/Listeners", ssh-agent signer(s) enabled
DEBU[0000] SSH Agent Key SHA256:<sha> ssh-rsa
DEBU[0000] SSH Agent Key SHA256:<sha> ssh-ed25519
DEBU[0000] SSH Agent Key SHA256:<sha> ssh-ed25519
DEBU[0000] SSH Agent Key SHA256:<sha>ssh-ed25519
DEBU[0000] SSH Agent Key SHA256:<sha> ssh-rsa
DEBU[0000] SSH Agent Key SHA256:<sha> ssh-ed25519
DEBU[0000] SSH Agent Key SHA256:<sha> ssh-ed25519
DEBU[0000] SSH Agent Key SHA256:<sha> ssh-rsa
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: Connection to bastion host (ssh://core@localhost:61933/run/user/501/podman/podman.sock) failed.: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures

Executing this command again and again helps.

Steps to reproduce the issue:

1. Execute any command which needs connection to a podman machine

Describe the results you received:

ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures

Describe the results you expected:

Command executed successfully.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.0.3
API Version:  4.0.3
Go Version:   go1.18
Built:        Fri Apr  1 17:28:59 2022
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.0.3
API Version:  4.0.3
Go Version:   go1.18
Built:        Fri Apr  1 20:22:39 2022
OS/Arch:      linux/arm64

Output of podman info --debug:

host:
  arch: arm64
  buildahVersion: 1.24.3
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 2
  distribution:
    distribution: fedora
    variant: coreos
    version: "36"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.17.3-300.fc36.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 615960576
  memTotal: 2052497408
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc36.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.aarch64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 26h 5m 42.86s (Approximately 1.08 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/501/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.3
  Built: 1648837359
  BuiltTime: Fri Apr  1 20:22:39 2022
  GitCommit: ""
  GoVersion: go1.18
  OsArch: linux/arm64
  Version: 4.0.3

Package info (e.g. output of rpm -q podman or apt list podman):

$ brew info podman
podman: stable 4.0.3 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/4.0.3 (172 files, 46.2MB) *
  Poured from bottle on 2022-04-05 at 02:23:33
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✔, go-md2man ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
fish completions have been installed to:
  /opt/homebrew/share/fish/vendor_completions.d
==> Analytics
install: 19,820 (30 days), 52,160 (90 days), 128,153 (365 days)
install-on-request: 19,823 (30 days), 52,143 (90 days), 128,127 (365 days)
build-error: 8 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[flouthoc]

Hi @sanyer , This looks duplicate of #12289 therefore i'm closing it could you please bump original issue. Please feel free to re-open if you think this is duplicate. Thanks

[thw0rted]

Hi @flouthoc , I'm having this same issue on a local (not remote) machine, with only one key in the agent. Do you think the issue you considered as duplicate will address my problem as well? If not, should this issue be re-opened, or should I file my own? (I'm able to work around the problem by unsetting SSH_AUTH_SOCK but I believe that shouldn't be necessary in the long term.)

ETA: it might be relevant that I'm on Windows, and my SSH_AUTH_SOCK is pointing to Pageant (the PuTTY agent), not ssh-agent. Maybe this needs its own issue?

[eveerman]

Hi @flouthoc , I'm having this same issue on a local (not remote) machine, with only one key in the agent. Do you think the issue you considered as duplicate will address my problem as well? If not, should this issue be re-opened, or should I file my own? (I'm able to work around the problem by unsetting SSH_AUTH_SOCK but I believe that shouldn't be necessary in the long term.)

ETA: it might be relevant that I'm on Windows, and my SSH_AUTH_SOCK is pointing to Pageant (the PuTTY agent), not ssh-agent. Maybe this needs its own issue?

@thw0rted I see very similar seeming issues with podman remote client on windows - which have existed since, I think, at least v2 (?) just opened #15121 if you want to follow or expand upon that one also.
I think that the issues as reported are different, e.g. on windows I certainly don't see it as an intermittent issue. Further in my setup SSH_AUTH_SOCK is being used and correctly picked up by OpenSSH-Portable which is the "windows native ssh client" and in general works fine. i.e. not simply an issue of many keys.

It is certainly possible that a move to an internal ssh client (from containers/common#1094 will resolve it but the bug reports themselves are different,

[thw0rted]

Just to make sure this doesn't get lost: #12289 just closed because #15094 was merged, adding a flag for golang vs native SSH connectivity. This needs to be tested on Windows, in both modes, and ideally in an environment that is configured to use Pageant as an agent. If it works, great; if not, this issue should be re-opened (or I guess a new one filed).

(@cdoern -- maybe you know whether that scenario has been tested already?)