Use PowerShell for REPL shell mode on Windows

[davidanthoff]

It would be great if the julia shell mode on Windows used PowerShell as the shell. That is after all the default shell on Windows, and it is much more powerful cmd.exe.

This has some info on how to host PowerShell. Not sure how one would pull this of technically, maybe have a mixed DLL that is compiled with VC as a bridge between julia and the CLR?

Clearly not a high priority item.

[yuyichao]

Dup of #7267 ?

[davidanthoff]

No, this is a different thing. What I'm suggesting here is that command entered in REPL mode of the julia shell (i.e. after you have pressed ;) are executed in a hosted PowerShell environment. #7267 is about the console that hosts the julia REPL. Two different things.

[tkelman]

It is however contained in #11478. I think it would be better to do this as a package rather than making it default, since there's a substantial initial latency in running powershell commands.

[davidanthoff]

Yes, #11478 has some discussion, but it really is a PR about something else, so might make sense to keep this issue here open?

I agree performance will not be good if you start powershell.exe whenever you execute a command via ;. I think the way to do this would be via the hosting API for PowerShell. Also agree that one could experiment with that in a package and then see how performances looks. If it is ok, it could go into base?

[tkelman]

I don't think we're likely to move much more functionality from packages into base, we're doing a lot of the reverse.

[tkelman]

If it's useful enough (and it probably would be for doing a more capable version of Base.download and others) then it could be a good candidate for a "default package," once we get the mechanism for doing that worked out.

[davidanthoff]

Did some more research on how this might be done. I think the easiest might be this: create a dll written in C# that interacts with the PowerShell hosting API. That dll would then export native functions via the UnmanagedExports package and could be called directly from julia via ccall. I just tried this and it all works. That seems by far the simplest way to get the whole CLR hosting etc. done, essentially one just gets that for free with this approach.

[PallHaraldsson]

"on Windows". Note, PowerShell is now portable to Linux and OS X. I guess it doesn't change this issue as can't depend on it installed..

[Bebotron]

Have there been any updates on this? I use Julia on Windows and it would be nice to use Powershell commands on the REPL in shell mode.

[StefanKarpinski]

We don't use a shell in any of the "shell" modes, so I'm inclined to say no on this.

[Bebotron]

Could you elaborate? I'm not quite familiar as to how shell mode works on the REPL. I just mean that just as we can use bash commands in shell mode running the REPL on Linux, is there no way to be able to use PS commands in shell mode on the REPL as well?

Edit: I guess I should clarify, it's not important to me that I can specifically use PS commands in shell mode. It would be perfectly ok to use bash commands. But the problem seems to be that when running the Julia REPL from Powershell, none of the usual commands work in shell mode (ls, pwd, cd, tec...)

[ViralBShah]

Has anyone tried if they work in git-bash?

[StefanKarpinski]

You can’t use bash commands on Linux: shell mode is not a shell, it’s just a syntax for doing run(`some stuff`);. This is the same on all platforms, we don’t use a shell on any of them.

[ViralBShah]

I was referring to things like ls and pwd. If you use the Windows console which is what Julia uses by default, run(`ls`); won't work. But, if you start it in git-bash, it does work on Windows, which makes using Windows so much better. I am doing this on Windows 7. Also note the unix style paths.

Viral@Viral-PC MINGW64 ~
$ ./AppData/Local/Programs/Julia/Julia-1.4.1/bin/julia.exe
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.1 (2020-04-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> run(`pwd`)
/c/Users/Viral
Process(`pwd`, ProcessExited(0))

[ViralBShah]

I think we can close this with Windows Terminal being out.

[davidanthoff]

I don't think this issue here has anything to do with the Windows Terminal: it is about the experience one gets in the ; REPL mode.

[StefanKarpinski]

Yes, it has nothing to do with the Windows Terminal. The request is for the shell mode on Windows to run an actual shell, unlike what it currently does on all platforms. I really don't think we want to get into the game of shell mode invoking real shells, but if we do, then it should be the same on all platforms and not different on Windows.

[Bebotron]

I guess this goes back to me not understanding how shell mode works, so I don't think I'm articulating this properly. It doesn't matter (to me) that shell more doesn't run an actual shell. It matters that when I run the Julia REPL from Windows Terminal, shell mode doesn't work (running ;ls, ;cat, etc..). It's seeming that this is a completely different issue that what was originally suggested, so I don't know if opening a new issue titled "Shell mode not working on Windows".

[StefanKarpinski]

One option would be to ship (bundle) busybox along with Julia on Windows and include the path to busybox in the path (probably at the end of the path) in shell mode so that commands like that do actually work. Busybox is GPL, but this this is bundling rather than creating a derived work—if you delete busybox, everything still works and you could swap in any implementation of the same commands to the same effect.

[davidanthoff]

I think if we ever wanted to support PowerShell in the way I asked for at the top (and I should say, I don't think this is a high priority issue), then today one could also think about something like this:

• Add an API to Julia base that allows packages to hook into the ; shell mode, i.e. allow a package to say "I can provide a shell experience"
• Create a PowerShell.jl package that hooks into this API, i.e. if PowerShell.jl is loaded, then ; mode is handled by PowerShell.jl in some way. Users could add a using PowerShell to their startup.jl then.
• PowerShell.jl itself could actually use artifacts to deploy PowerShell 7. PowerShell 7 is a) cross platform and b) xcopy deployable. So at that point it would not be a Windows solution at all, but just PowerShell for who wants this.

That way this problem would be outsourced from Julia itself.

Could we reopen this issue? Even if it isn't high priority, I think it is a valid request and maybe at some point someone wants to take it up.

I think a distinct issue is that shell mode on Windows right now is just more or less useless, but fixing that (in a non-PowerShell way) seems a topic for a different issue than this one here.

[StefanKarpinski]

These approaches aren't mutually exclusive: shipping busybox would make the shell mode on Windows not useless by default and would make many examples in the wild work on Windows. Allowing packages to hook into the shell mode is also a cool idea, but we could do both.

[davidanthoff]

Yes, agreed, I think doing both would be ideal. Fixing shell mode (via busybox or whatever) seems the higher priority to me of the two.

[ViralBShah]

I see the difference. Yes, it would be nice that it drops you into something meaningful, when you press ;.

[PallHaraldsson]

[EDIT: I assumed same solution as for BusyBox would work for Toybox, but it's not been ported, and WSL only may (or may not) work: https://github.com/landley/toybox/issues/61 ]

Busybox is GPL

As you say the GPL not a problem, but I would suggest the much smaller 814 KB (archived):
https://en.wikipedia.org/wiki/Toybox

include ls, cp, mv, and about 150 others. [..] became a BSD-licensed BusyBox alternative.[3][4] Toybox is included with Android 6.0 "Marshmallow"[5]

If we ever support Android, then it's already there. Both boxes are for Unix (and KornShell Android uses), so I wouldn't know if they work on Windows, I guess the solution works for both.

[StefanKarpinski]

I would suggest the much smaller 814 KB (archived): https://en.wikipedia.org/wiki/Toybox

I wouldn't know if they work on Windows, I guess the solution works for both.

Since UNIX systems already have all the expected commands, we only need this for Windows, so something that doesn't appear to work on Windows isn't super helpful.

[IlyaOrson]

Just as a practical workaround for the time being, it is possible to use ReplMaker.jl to register a powershell mode that works (at least for the basic stuff) in Windows, taking parts from #31490 as shown here.

[jgardona]

I use msys2 as shell for julia. It works as expected. Or just use rust alternatives as lsd, fd, ripgrep... Just put them on path and you will have a shell working.