Expected PROXY_READY, received ‘PROXY_ERROR: Remote SSH QSys ‘SSH’ is not allowed’

[kablag]

I do not understand how to setup SSH cluster and user guide link https://github.com/mschubert/clustermq/wiki#setting-up-the-scheduler is broken…

I installed clustermq to my local (MacOS) computer and to the cluster. It works on both.
Then I use on my local computer

options(
  clustermq.scheduler = "ssh",
  clustermq.ssh.host = "myuser@myhost", # use your user and host, obviously
  clustermq.ssh.log = "~/cmq_ssh.log" # log for easier debugging
)

fx = function(x) x * 2
Q(fx, x=1:3, n_jobs=1)

and get

Sending common data ...
Error in .subset2(public_bind_env, "initialize")(...) : 
  Expected PROXY_READY, received ‘PROXY_ERROR: Remote SSH QSys ‘SSH’ is not allowed’

Do I need special setup on my cluster?

ssh -R 54709:localhost:6687 myuser@myhost R --vanilla works.

[mschubert]

The user guide link should be: https://mschubert.github.io/clustermq/articles/userguide.html#ssh. This is already fixed in the current develop branch.

The relevant error message for your case is:

PROXY_ERROR: Remote SSH QSys ‘SSH’ is not allowed

It looks like you set

options(clustermq.scheduler = "ssh")

not only on your local machine but also your remote cluster. On your cluster, this should be your scheduler (or multicore) and not ssh (otherwise you'd ssh into your cluster to then ssh somewhere else).

[mschubert]

Closing this, because https://mschubert.github.io/clustermq/articles/userguide.html#ssh-connector already states that the remote scheduler should be set up and tested prior to using it via SSH.

[wlandau]

I am getting this same error when I try to send jobs to a Docker container. The container has clustermq and ssh keys correctly configured:

$ sudo docker run -it wlandau/cmq-ssh /bin/bash
$ service ssh start
$ mkdir /root/.ssh
$ echo PUBLIC_KEY_CONTENT > /root/.ssh/authorized_keys

Outside the container:

options(
  clustermq.scheduler = "ssh",
  clustermq.ssh.host = "[email protected]"
)
library(clustermq)
f <- function(x) x * 2
Q(f, x = seq_len(3), n_jobs = 1)
#> Connecting [email protected] via SSH ...
#> Sending common data ...
#> Error in .subset2(public_bind_env, "initialize")(...) : 
#>   Expected PROXY_READY, received ‘PROXY_ERROR: Remote SSH QSys ‘LOCAL’ is not allowed’

[mschubert]

You need to select a scheduler that works via sockets in the container.

So,

options(clustermq.scheduler = "multiprocess")

should do?

I will clarify this in the docs.

[wlandau]

Thanks, it worked! I didn't realize options(clustermq.scheduler = "multiprocess") needed to be in the worker's .Rprofile file so any new R instance would have the correct scheduler.

[wlandau]

I am trying this same approach to connect to my company's SGE cluster, and I am running into issues. Nothing gets written to the log. ssh CENSORED-USER@CENSORED-HOST in my local terminal works just fine. Any ideas?

options(
  clustermq.scheduler = "ssh",
  clustermq.ssh.host = "CENSORED-USER@CENSORED-HOST",
  clustermq.ssh.log = "~/cmq.log"
)
f <- function(x) 2*x
library(clustermq)
Q(fun = f, x = 1:3, n_jobs = 3)
#> Connecting CENSORED-USER@CENSORED-HOST via SSH ...
#>
#> This system is for AUTHORIZED use ONLY!
#> 
#> 
#> Error: remote port forwarding failed for listen port 52425
#> 
#> Error in .subset2(public_bind_env, "initialize")(...) : 
#>   Remote R process did not respond after 5000 seconds. Check your SSH server log.

[wlandau]

Traceback:

6: stop("Remote R process did not respond after ", init_timeout, 
       " seconds. ", "Check your SSH server log.")
5: .subset2(public_bind_env, "initialize")(...)
4: qsys$new(data = data, reuse = reuse, ...)
3: workers(n_jobs, data = data, reuse = FALSE, template = template, 
       log_worker = log_worker, verbose = verbose)
2: Q_rows(fun = fun, df = df, const = const, export = export, pkgs = pkgs, 
       seed = seed, memory = memory, template = template, n_jobs = n_jobs, 
       job_size = job_size, rettype = rettype, fail_on_error = fail_on_error, 
       workers = workers, log_worker = log_worker, chunk_size = chunk_size, 
       timeout = timeout, max_calls_worker = max_calls_worker, verbose = verbose)
1: Q(fun = f, x = 1:3, n_jobs = 3)

[wlandau]

I also tried setting options(clustermq.ssh.timeout = 60), but I still get the same error.

[mschubert]

The problem here is that the reverse tunnel fails:

#> Error: remote port forwarding failed for listen port 52425

We need the tunnel to be able to piggyback the local<->worker connection on the existing SSH connection.

Maybe your admins disabled that? (details here)

[wlandau]

Looks that way now that you mention it.

debug1: Remote: Server has disabled port forwarding.
debug1: remote forward failure for: listen 54709, connect localhost:6687

Full log:

$ ssh -v -R 54709:localhost:6687 USER@HOST source .bashrc && module load R-qualified && R --vanilla
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to HOST port 22.
debug1: Connection established.
debug1: identity file /Users/USER/.ssh/id_rsa type 0
debug1: identity file /Users/USER/.ssh/id_rsa-cert type -1
debug1: identity file /Users/USER/.ssh/id_dsa type -1
debug1: identity file /Users/USER/.ssh/id_dsa-cert type -1
debug1: identity file /Users/USER/.ssh/id_ecdsa type -1
debug1: identity file /Users/USER/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/USER/.ssh/id_ed25519 type -1
debug1: identity file /Users/USER/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/USER/.ssh/id_xmss type -1
debug1: identity file /Users/USER/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to HOST:22 as 'USER'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: HOSTKEY
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: HOSTKEY SHA256:HOSTSHA
debug1: Host 'HOST' is known and matches the ECDSA host key.
debug1: Found key in /Users/USER/.ssh/known_hosts:16
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/USER/.ssh/id_rsa RSA SHA256:OTHERSHA
debug1: Will attempt key: /Users/USER/.ssh/id_dsa 
debug1: Will attempt key: /Users/USER/.ssh/id_ecdsa 
debug1: Will attempt key: /Users/USER/.ssh/id_ed25519 
debug1: Will attempt key: /Users/USER/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received

This system is for AUTHORIZED use ONLY!


debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/USER/.ssh/id_rsa RSA SHA256:OTHERSHA
debug1: Server accepts key: /Users/USER/.ssh/id_rsa RSA SHA256:OTHERSHA
debug1: Authentication succeeded (publickey).
Authenticated to HOST ([40.1.29.175]:22).
debug1: Remote connections from LOCALHOST:54709 forwarded to local address localhost:6687
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: Remote: Server has disabled port forwarding.
debug1: remote forward failure for: listen 54709, connect localhost:6687
Warning: remote port forwarding failed for listen port 54709
debug1: All remote forwarding requests processed
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: source .bashrc
TERM environment variable not set.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype [email protected] reply 0
TERM environment variable not set.
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3588, received 3544 bytes, in 0.2 seconds
Bytes per second: sent 20610.4, received 20357.6
debug1: Exit status 1