macos: podman asks for login even when remote does not allow password authentications

[ssbarnea]

/kind bug

Description

It seams that podman-remote goes into interactive password prompts regardless if remote servers allow them or not, which is different from ssh implementation and also breaks automation.

Please note that this problem seams to be particular to "podman" cli on macos (which is in fact podman-remote), but when trying to reproduce the same login prompt from a linux host while using the podman-remote command, it did fail correctly with a "no supported methods remain". Both versions were the same (2.1.1).

Steps to reproduce the issue:

# proof server does not allow password authentication
ssbarnea@imac: ~
$ SSH_AUTH_SOCK= ssh -i /dev/null -F /dev/null -o PubkeyAuthentication=no root@leno
root@leno: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
FAIL: 255

# proof that podman asks for password when it should not do so
ssbarnea@imac: ~
$ CONTAINER_HOST=ssh://root@leno/run/podman/podman.sock CONTAINER_SSHKEY=/Users/ssbarnea/.ssh/id_rsa_insecure
 podman --log-level=debug ps
INFO[0000] podman filtering at log level debug
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=debug ps)
Login password:
Time: 0h:00m:06s

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:
podman version 2.1.1

(paste your output here)

Output of podman info --debug:
Irellevant because it does not work due to current bug

Login password:

Package info (e.g. output of rpm -q podman or apt list podman):

2.1.1 installed with brew on macos

Related: #7806

[afbjorklund]

Podman should only fall back to asking for the password, if it can't find any of:

• identity (in variable)
• SSH_AUTH_SOCK
• password (in url)

        if len(authMethods) == 0 {
                pass, err := terminal.ReadPassword("Login password:")
                if err != nil {
                        return Connection{}, err
                }
                authMethods = append(authMethods, ssh.Password(string(pass)))
        }

In your case you are passing an identity (CONTAINER_SSHKEY), so it should not ask for password.

But your log output is missing a vital line, public key signer enabled for identity, after parsing it...

[jwhonce]

@ssbarnea I'm sorry, I have been unable to reproduce this issue.

I'm using:

• macOS Big Sur 11.0.1
• podman 2.1.1 installed via brew
• CONTAINER_HOST=ssh://[email protected]:22/run/user/1000/podman/podman.sock CONTAINER_SSHKEY=~/.ssh/id podman --log-level=debug ps

I have tried:

• id files that both are authorized and unauthorized on the server, not prompted
• Setting CONTAINER_SSHKEY any value should either result in a error (invalid/missing files) or a log output when valid
• Setting CONTAINER_SSHKEY="" does cause the Login prompt, as expected
• I have tried both bash and zsh

/cc @ashley-cui

[ssbarnea]

I suspect this is highly dependent on key type and size and apparently there is no debug option to enable to make it more verbose.

Confirmed that my both keys are 4096 rsa, one of them is encrypted at rest (loaded into agent) and the other one is unencrypted. podman refuses to connect to the bastion host with both, but ssh has no problems connecting.

The bastion is a fedora 33 machine but i doubt that matters much because ssh works, with RSA keys too!

Update

Now I do have the confirmation that podman is unable to establish ssh connection using RSA keys, regardless if they are loaded by itself or via ssh-agent, it fails to establish a handshake with them:

Error: Failed to create sshClient: Connection to bastion host (ssh://root@leno:22/var/run/podman/podman.sock) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Following https://www.ssh.com/ssh/keygen/ I created two other keys ecdsa and ed25519 and they both worked. I skiped testing dsa as I mostly considered unsafe/not-recommended/deprecated.

[ssbarnea]

@afbjorklund the logic in the mentioned code snipped seams flawed at best. An app should not prompt user for a password just because it failed to load other authentication methods. Interactive (challenge) SSH authentication should be taken care of the ssh library and should establish a real connection with the server, not to ask user about a password.

This is confusing at best, but I would even add that it has some security implications. Also it breaks because it does not support any authentication variations, for example some systems may require multiple answers, like OTP tokens.

I personally do not see much use for that prompt and I would not mind seeing it removed, or at least making it appear only when user explicitly asked for this "feature" to be enabled, like a --prompt kind of option.

At this moment if I force podman to run with a non-tty by adding </dev/null, I endup with another confusing error, side effect from current implementation:

Error: Failed to create sshClient: EOF
FAIL: 125

If I try to pipe the std stream to something else, I still get the interactive prompt, yet another reason for causing breakage in production or CI.

podman --log-level=debug ps 2>&1 | cat
Login password:

[afbjorklund]

@afbjorklund the logic in the mentioned code snipped seams flawed at best. An app should not prompt user for a password just because it failed to load other authentication methods. Interactive (challenge) SSH authentication should be taken care of the ssh library and should establish a real connection with the server, not to ask user about a password.

I didn't write the code, I was just looking at it... (It was added in cbca625 and 6ff4239)

https://github.com/containers/podman/blob/v2.2.0/pkg/bindings/connection.go#L185_L215

Still don't know why any of it is used though, if you are indeed providing an identity (or agent) ?

Maybe $SSH_AUTH_SOCK is misbehaving ?

I personally do not see much use for that prompt and I would not mind seeing it removed, or at least making it appear only when user explicitly asked for this "feature" to be enabled, like a --prompt kind of option.

Seems reasonable, and this is how ssh works I suppose. (PasswordAuthentication option)

[ssbarnea]

I would not mind removing that code myself but I am waiting for some podman core developer to comment on this as I do not want to propose an unwanted change.

[afbjorklund]

But it seems like the main difference between and Mac and Linux is the missing ssh-agent (SSH_AUTH_SOCK) ?

Here it has either 2 (identity+agent) or 1 (agent only) auth method, so it never falls back to asking for a password...

[afbjorklund]

It is possible to change the code to use PasswordCallback (when needed) instead of Password (up front).

You might still want to have an option to enable/disable, but at least wouldn't ask if not supported by server.

[afbjorklund]

After PR #8596 :

Now it goes directly to failure mode, if server doesn't support passwords. If it does, it will prompt (like before).

Error: failed to create sshClient: Connection to bastion host (ssh://[email protected]:39147) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain

Normally passwords are disabled for root. If they are allowed, but the wrong one is given, it looks more like:

Login password:
Error: failed to create sshClient: Connection to bastion host (ssh://[email protected]:39147) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

---

For this particular test, I did:
unset CONTAINER_SSHKEY
unset SSH_AUTH_SOCK

Normally would suggest keys...

[afbjorklund]

I think it's a bit weird that it is called the "bastion host", since as far as I know it only supports unix sockets ?

https://en.wikipedia.org/wiki/Bastion_host

So it's not so much about jumping to another server on the network, the Podman service is probably local...

But it was only the message, not important.