winpty-agent.exe hangs on Windows 10 upon terminal resize

[aleherb]

E.g. within mintty, execute

console.exe cmd

then resize the terminal window -> winpty-agent.exe hangs in

(gdb) bt
#0 0x77c28c3c in ntdll!ZwDeviceIoControlFile () from C:\Windows\SYSTEM32\ntdll.dll
#1 0x76e8c8e9 in KERNELBASE!GetConsoleMode () from C:\Windows\SYSTEM32\KernelBase.dll
#2 0x76e8c7fa in KERNELBASE!GetConsoleMode () from C:\Windows\SYSTEM32\KernelBase.dll
#3 0x76f03230 in SetConsoleScreenBufferSize () from C:\Windows\SYSTEM32\KernelBase.dll
#4 0x0040485c in Win32Console::resizeBuffer(Coord const&) ()
#5 0x00404a97 in Win32Console::reposition(Coord const&, SmallRect const&) ()
#6 0x004033b1 in Agent::resizeWindow(int, int) ()
#7 0x00402dff in Agent::handleSetSizePacket(ReadBuffer&) ()
#8 0x0040295d in Agent::handlePacket(ReadBuffer&) ()
#9 0x004028d1 in Agent::pollControlSocket() ()
#10 0x0040275c in Agent::onPipeIo(NamedPipe*) ()
#11 0x0040154a in EventLoop::run() ()
#12 0x004064e6 in main () 

Workaround: Enable legacy console in cmd > Properties.

The following patch seems to prevent the hang:

diff --git a/agent/Agent.cc b/agent/Agent.cc
index 5983ab9..52c5c4b 100644
--- a/agent/Agent.cc
+++ b/agent/Agent.cc
@@ -392,8 +392,8 @@ void Agent::resizeWindow(int cols, int rows)
         markEntireWindowDirty();
     m_dirtyWindowTop = newWindowRect.top();

-    m_console->reposition(newBufferSize, newWindowRect);
     unfreezeConsole();
+    m_console->reposition(newBufferSize, newWindowRect);
 }

 void Agent::scrapeOutput()

[trofimander]

Confirm: that fixes the hang on Windows 10.

[leycec]

Could someone post that as a pull request? Pleaaaase.

[rprichard]

I think I understand what's happening. The new Windows 10 console rewraps the console lines when the console is resized. When the console is "frozen" (i.e. the "Select All" system command is active), writes to the console block. Presumably, with Windows 10, attempts to resize the console also block.

I really need to test with Windows 10, though.

I'm wondering what happens if one of the console programs prints new lines before/during the Win32Console::reposition call.

I suspect Windows 10's line rewrapping is going to confuse winpty. Normally, winpty tries to detect changes and send only changed lines, but with rewrapping, I think it's going to lose track of what's new.

[aleherb]

Presumably, with Windows 10, attempts to resize the console also block.

No, it doesn't, but the selection is cleared. This is different to a Windows 7 console.

I'm wondering what happens if one of the console programs prints new lines before/during the Win32Console::reposition call.

I wonder what freezing the console during the reposition should actually protect. Printing lines does not affect window or buffer size.

I suspect Windows 10's line rewrapping is going to confuse winpty.

The only bad impact I have seen so far is, that a resize confuses editing of the current input line in cmd.exe, in particular history navigation. It seems to work fine with e.g. bash (+readline).

[rprichard]

I tested programmatically resizing the console buffer while selection was in-progress, and it apparently hangs the Windows console, such that the conhost.exe process must be killed from the Task Manager. This actually seems like a Windows bug to me.

See https://github.com/rprichard/winpty/blob/win10/misc/Win10ResizeWhileFrozen.cc

[rprichard]

I reported the conhost.exe hang to Microsoft: https://wpdev.uservoice.com/forums/266908-command-prompt/suggestions/9941292-conhost-exe-hangs-in-win10-if-setconsolescreenbuff

[rprichard]

Fixed by d464089. Unfortunately, conhost.exe can still hang if something other than winpty resizes the console.