Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SSHManager support for invoking Windows workers via cmd.exe #38353

Merged
merged 1 commit into from
Nov 23, 2020

Conversation

mgkuhn
Copy link
Contributor

@mgkuhn mgkuhn commented Nov 9, 2020

Distributed.addprocs() now supports four new keyword arguments shell,
ssh, env and cmdline_cookie.

Specifying shell=:wincmd now makes it possible to start workers on a
Windows machine with an sshd server that invokes cmd.exe as the shell
(e.g. Microsoft's OpenSSH port does that by default). Previously
SSHManager only supported ssh connections to a POSIX shell.

Specifying ssh="/usr/bin/ssh" makes it possible to specify the ssh
client that SSHManager will use (useful for debugging and where a
custom-version of ssh is required).

The new env parameter now allows to pass arbitrary environment
variables to workers.

Specifying cmdline_cookie=true is a workaround for an ssh problem
with Windows workers that run older (pre-ConPTY) version of Windows,
Julia, or OpenSSH.

(This is a revised version of PR #30614)

@musm
Copy link
Contributor

musm commented Nov 19, 2020

LGTM pending #38352

@jebej
Copy link
Contributor

jebej commented Nov 20, 2020

Is it possible to make this work for when ssh conects to powershell instead of cmd? Maybe by first starting cmd.exe?

@mgkuhn
Copy link
Contributor Author

mgkuhn commented Nov 20, 2020

Adding PowerShell command-invocation support is on my todo list, but will be a future, separate PR. (Proper meta-character escaping in PowerShell sadly can be even more complicated than in cmd.exe. Neither has an escaping mechanism as simple and clean as we are used to from the POSIX shell.)

@jebej
Copy link
Contributor

jebej commented Nov 20, 2020

So it wouldn't be possible to just start cmd.exe from powershell and then use the work from this PR?

@musm
Copy link
Contributor

musm commented Nov 20, 2020

Looks like this needs a rebase to remove the cmd.exe escaping functions since we've merged #38352

Distributed.addprocs() now supports four new keyword arguments `shell`,
`ssh`, `env` and `cmdline_cookie`.

Specifying `shell=:wincmd` now makes it possible to start workers on a
Windows machine with an sshd server that invokes `cmd.exe` as the shell
(e.g. Microsoft's OpenSSH port does that by default). Previously
SSHManager only supported ssh connections to a POSIX shell.

Specifying `ssh="/usr/bin/ssh"` makes it possible to specify the ssh
client that SSHManager will use (useful for debugging and where a
custom-version of ssh is required).

The new `env` parameter now allows to pass arbitrary environment
variables to workers.

Specifying `cmdline_cookie=true` is a workaround for an ssh problem
with Windows workers that run older (pre-ConPTY) version of Windows,
Julia, or OpenSSH.
@mgkuhn
Copy link
Contributor Author

mgkuhn commented Nov 20, 2020

Rebased after merge of #38352

@musm
Copy link
Contributor

musm commented Nov 22, 2020

Failures look unrelated.
One is a linear algebra failure: https://build.julialang.org/#/builders/52/builds/5721/steps/5/logs/stdio
The other is Profile error: https://build.julialang.org/#/builders/4/builds/6152/steps/5/logs/stdio

Will merge tomorrow sans objections.

@musm musm merged commit bb6a48e into JuliaLang:master Nov 23, 2020
simonbyrne added a commit that referenced this pull request Nov 24, 2020
I made this change to allow the set to be expanded for my own package, but I noticed this also helps unify #38353 and existing ssh-only options.
@simonbyrne simonbyrne added the parallelism Parallel or distributed computation label Nov 24, 2020
simonbyrne added a commit that referenced this pull request Nov 26, 2020
I made this change to allow the set to be expanded for my own package, but I noticed this also helps unify #38353 and existing ssh-only options.
JeffBezanson pushed a commit that referenced this pull request Nov 30, 2020
…8570)

I made this change to allow the set to be expanded for my own package, but I noticed this also helps unify #38353 and existing ssh-only options.
Keno pushed a commit that referenced this pull request Jun 5, 2024
Distributed.addprocs() now supports four new keyword arguments `shell`,
`ssh`, `env` and `cmdline_cookie`.

Specifying `shell=:wincmd` now makes it possible to start workers on a
Windows machine with an sshd server that invokes `cmd.exe` as the shell
(e.g. Microsoft's OpenSSH port does that by default). Previously
SSHManager only supported ssh connections to a POSIX shell.

Specifying `ssh="/usr/bin/ssh"` makes it possible to specify the ssh
client that SSHManager will use (useful for debugging and where a
custom-version of ssh is required).

The new `env` parameter now allows to pass arbitrary environment
variables to workers.

Specifying `cmdline_cookie=true` is a workaround for an ssh problem
with Windows workers that run older (pre-ConPTY) version of Windows,
Julia, or OpenSSH.
Keno pushed a commit that referenced this pull request Jun 5, 2024
…8570)

I made this change to allow the set to be expanded for my own package, but I noticed this also helps unify #38353 and existing ssh-only options.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
parallelism Parallel or distributed computation system:windows Affects only Windows
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants