Issues observed with PowerShell on Server-Side

[heywoodlh]

I use PowerShell Core (currently 7.3.2) on my servers (NixOS, Arch Linux) running OpenSSH version OpenSSH_9.2p1, OpenSSL 3.0.8 7 Feb 2023.

When I use mosh over ssh with PowerShell I observe two things I don't encounter with a normal ssh client:

• It takes around 10 seconds before it actually connects
• A bunch of characters are injected into my virtual terminal upon connection (see the attached screenshot, I did not press anything) -- I can clear them out, so it's not a huge issue

Is there any way to get mosh to play more nicely with PowerShell (or vice versa) -- I'd love to have these issues resolved. I am not sure what further information to provide, so please let me know what additional information would be useful in diagnosing the exact issue.

[achernya]

What terminal emulator are you using? In theory the specific shell should not matter, but I admit I don't know how powershell interacts with its terminal emulator.

[heywoodlh]

@achernya I have observed the issues I described while using mosh in the following terminal emulators:

• Kitty (MacOS, Linux)
• Guake (Linux)
• iTerm2 (MacOS)
• Blink (i[Pad]OS)

[achernya]

Do you have screenshots from, say, xterm, rxvt-unicode, gnome-terminal, or Terminal.app? Preferably showing side-by-side of ssh and mosh? Those are the terminals I am most familiar with. I'd normally ask for the output of echo $TERM ... but I have no idea if powershell respects $TERM. Also do the artifacts go away if you run powershell under screen/tmux?

[eminence]

Also, what version of mosh, on both the client and server?

[heywoodlh]

So I just found that adding --no-ssh-pty seemingly resolves the issue. Is there a way to make that a default in my PowerShell environment, i.e. with an environment variable (worst case scenario, I could just create a PowerShell alias for mosh --no-ssh-pty)?

---

Do you have screenshots from, say, xterm, rxvt-unicode, gnome-terminal, or Terminal.app? Preferably showing side-by-side of ssh and mosh?

If you'd like me to, I can send these screenshots but with figuring out the --no-ssh-pty flag resolves this issue for me, I'm guessing that it isn't a terminal emulator specific issue?

what version of mosh, on both the client and server?

1.4.0 (on both)

Also do the artifacts go away if you run powershell under screen/tmux

Funny enough, I use PowerShell in tmux, and it does not go away with it (not counting my recent discovery of --no-ssh-pty).

[eminence]

I haven't been able to reproduce this, using kitty v0.21.2 and PowerShell Core 7.3.2 on Debian. I don't have PowerShell yet as my default shell, so I tried to reproduce with:

mosh [email protected] pwsh

I don't really understand why no-ssh-pty has an impact here. I'd be interested in capturing what powershell is sending to your client. Can you capture that? Perhaps with something like:

mosh [email protected] -- script /tmp/pwsh.script -c pwsh

(Hopefully the same stuff gets injected to your client). Then exit powershell and send us the pwsh.script file

[heywoodlh]

Attached is pwsh.script (had to rename it to .txt to upload it to GitHub)
pwsh.txt

[heywoodlh]

To me, it looked like the same stuff was injected after script ran

[achernya]

Interestingly enough the script recording shows that ^[[48;1R^[[48;1RPowerShell 7.3.2 is verbatim a caret and a open square bracket, not a \[ escape sequence. This feels like a powershell bug to me, because if I replace it with an escape sequence character it's correct.

[heywoodlh]

This feels like a powershell bug to me

Yeah, that would make sense to me. I was thinking it could be my PowerShell profile, but upon logging into a server with PowerShell Core as it's default shell without any of my PowerShell profile stuff setup (so just using the default PowerShell profile), I seem to experience the same issue.

In the meantime, as a workaround I have created a PowerShell function to inject --no-ssh-pty into my mosh commands: https://github.com/heywoodlh/conf/blob/1d1e4b32d10983ddfafb3358b7a58a40bbf2c159/dotfiles/config/powershell/not-windows.ps1#L33-L37

---

If there's anymore information I can provide to help drill into this further please let me know and I'd be happy to provide it. I'm not too sure how to move forward with this on my end.

[eminence]

I guess it could also be something that runs before PowerShell, but if you don't see these weird characters with another shell (say, bash), then that does kind of point toward PowerShell.

I'm a bit confused as to what ^[[48;1R actually is. I can't find any CSI functions that end in R (except for XTREPORTCOLORS but that doesn't have the same form). I do notice that your original screenshot had ^[[21;1R and your latest script has ^[[48;1R -- I wonder if is supposed to be a DECSTBM, which ends with a lower-case r (do the numbers change based on your terminal size @heywoodlh ?). Even if so, this would be a weird bug, and doesn't explain why we're seeing a verbatim ^[[ instead of CSI

Anyway, I recognize that you may no longer want to spend time continuing to debug an issue for which you have a workaround. I would be curious to know if you see this issue with bash or how your terminal size might impact the output, if you don't mind.

[heywoodlh]

I would be curious to know if you see this issue with bash or how your terminal size might impact the output, if you don't mind.

Nothing on my client side seems to contribute to this. I only see this when PowerShell is my default shell on the server-side. When PowerShell is not my default shell, I do not experience this issue.

---

I recognize that you may no longer want to spend time continuing to debug an issue for which you have a workaround

I definitely would love to help find a fix to this rather than having to workaround this, so keep the suggestions coming. :)

[eminence]

How about getting an strace dump to confirm (or refute) the theory that it's pwsh that's printing this stuff?

[eminence]

also, you said specifically that you see this when PowerShell is your default shell:

• What if PowerShell is your default shell, but you use bash instead?
• What if bash is your default shell, but you use PowerShell instead?

[heywoodlh]

So here's how it looks when I login with mosh to a server which has BASH as its default shell (no weird symbols injected, looks normal):

---

What if PowerShell is your default shell, but you use bash instead?

Same issues as I described:

• Delay (around 10 seconds)
• Characters injected into shell

To test this, I ran this command (let me know if there is a better way):

mosh <host> bash

---

What if bash is your default shell, but you use PowerShell instead?

No issues:

To test this, I ran this command (let me know if there is a better way):

mosh <host> pwsh

---

How about getting an strace dump to confirm (or refute) the theory that it's pwsh that's printing this stuff?

Can you provide an example command? I'm not sure if you want strace on the server-side or client-side (my suspicion is that it's something with PowerShell Core + SSH on the server side).

[eminence]

ok, this is even more mysterious. If we see the weird stuff even when powershell isn't used, then maybe it's not powershell's fault at all.

I think we need an strace on the server, to see what's printing this (assuming that someone on the server-side is printing it).

So something like:

mosh <host> -- strace -ff -o /tmp/strace.output --string-limit=128 bash

Then send all the strace output files. What I'm looking to do here is identify exactly what process is writing the ^[[48;1R^[[48;1R string

[eminence]

@heywoodlh Any news on this?

[heywoodlh]

@eminence sorry for the delay, got distracted with life 😄

Just FYI, this is me mosh-ing into a NixOS server, which is why the file paths for binaries and such might be a bit odd.

strace.output.849741.txt
strace.output.849740.txt
strace.output.849739.txt
strace.output.849670.txt