Terminal hangs when opening new windows or new tabs on Windows Server 2022

[wkbrd]

After installing either Terminal 1.10 or Terminal Preview 1.11 (after its dependencies and winget), opening it the first time (or subsequent times) causes the window to hang indefinitely (the process shows as 'Not Responding' in Task Manager). A similar problem happens when clicking the + to open a new tab.

The problem happens regardless if the default terminal is either Powershell or others (like Command Prompt).

A partial workaround is to launch a second copy of the app while the instance is hung, then use Task Manager to kill the hung app. Once the hung process closes, the application opens as expected. The workaround does not address the new tab problem, but just the initial window opening.

If there is any logging or debug that can be enabled, please let me know.

[zadjii-msft]

A similar problem happens when clicking the + to open a new tab.

That quells my fear that it's related to #4472.

What shell are you trying to open in the new tab?

[wkbrd]

The default shell on a fresh install (PowerShell). I also changed the default to Command Prompt and it behaved the same.

[frugecn]

I'm also seeing it in Windows 10 for Terminal 1.10

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[wkbrd]

This is still a problem and I have replied. How does one remove the labels the bot mentions? I do not seem to have that ability.

Now the bot removed the labels but 8 days ago it did not.

[zadjii-msft]

Now the bot removed the labels but 8 days ago it did not.

Sorry about that, you probably replied in a weird window where the bot was down.

I've honestly got no idea what's going on here - could you grab a dump of the Terminal process while it's in the hung state? You should be able to with Task Manager:

There's not a ton of logging around this scenario, but that's partially because it's not something that's really been problematic in the past.

[wkbrd]

@zadjii-msft : How do I transfer the dump file to you securely? I cannot seem to upload .zip files or .dmp files here.

[zadjii-msft]

My email is on my github profile, you can send it there ☺️

[wkbrd]

@zadjii-msft : The dump file zipped is 76 MB which is larger than my e-mail quota allows. Is there another way to share it with you? I could request a temporary place via my IT organization, but it may take several days for the area to be provisioned.

[negf]

I have the same problem. Windows Terminal Preview v1.10.1933.0 works, and Windows Terminal v1.10.2383.0 hangs. I am on Windows 10 (19043.1237).

[wkbrd]

@zadjii-msft : What other methods do you have to provide Microsoft the dump file? I am unable to attach it here or send via e-mail?

[zadjii-msft]

@wkbrd Honestly, not sure I have a good solution. @DHowett Do we have another way other than email for something like this?

[wkbrd]

Perhaps if you have a private OneDrive folder that could be shared with me and then I could upload into it? (And hopefully the upload is not blocked by OneDrive)

[Falco20019]

Similar to #11252 btw.

[wkbrd]

@Falco20019 : Yes, this appears similar.

[Sankgreall]

#11252 appears to be related to Windows 10, whereas this issue appears to be related to Windows Server. I'm not sure they are duplicates, though #11252 has been marked as a duplicate of this issue now.

[Sankgreall]

@zadjii-msft If it helps, I have uploaded a memory dump with Windows Terminal in its hang state and shared it with your email address. You can access it through OneDrive here. https://srminform-my.sharepoint.com/:u:/g/personal/j_jackson_s-rminform_com/EfptE2hZQ2xErdBCGJN967MBjQavxnnXQQx8JtaNyc8_uQ?email=migrie%40microsoft.com&e=9IKHlN

Hope that's okay but let me know if you would like me to host it somewhere else

[wkbrd]

Hi @Sankgreall , was your dump from Windows 10 or Server 2022?

[miniksa]

My GH name at the usual suspect corporate domain.

[Sankgreall]

Hosted here.

I will note that one of my debug sessions actually spun up the Terminal correctly. I just wasn't thinking and deleted it, but this is also expected behaviour as others have noted if you kill the process manually and start a new console, it sometimes appears to work and bypasses this issue.

[miniksa]

Sounds super race condition-y to me. I hope this trace is insightful. Opening now.

[miniksa]

Oh my goodness. There is an exception thrown on a thread inside of this trace. We might be onto something! Digging!

[miniksa]

This is going to sound really weird. Do you have a touchscreen?

[Sankgreall]

Well. Technically yes? The machines we install Windows Terminal on get shipped out to users with Surface Laptops, which have touch screens. I am running a Dell XPS, which also has touchscreen. The touchscreen functionality is not used for any business processes, however... If that makes sense?

[miniksa]

An access violation happens (0xC0000005) as we attempt to get the position of the cursor in ScreenInfoUiaProviderBase::GetSelection in response to a query right as we're starting up.

Partial stack of throwing callsite

0:000> k
 # Child-SP          RetAddr               Call Site
00 (Inline Function) --------`--------     Microsoft_Terminal_Control!Cursor::GetPosition [C:\a\_work\1\s\src\buffer\out\cursor.cpp @ 42] 
01 000000f0`c2d0e7a0 00007ffc`b1473643     Microsoft_Terminal_Control!Microsoft::Console::Types::UiaTextRangeBase::RuntimeClassInitialize+0x5c [C:\a\_work\1\s\src\types\UiaTextRangeBase.cpp @ 47] 
02 (Inline Function) --------`--------     Microsoft_Terminal_Control!Microsoft::Terminal::TermControlUiaTextRange::RuntimeClassInitialize+0x25 [C:\a\_work\1\s\src\types\TermControlUiaTextRange.cpp @ 23] 
03 000000f0`c2d0e7f0 00007ffc`b147378f     Microsoft_Terminal_Control!Microsoft::WRL::Details::MakeAndInitialize<Microsoft::Terminal::TermControlUiaTextRange,Microsoft::Terminal::TermControlUiaTextRange,Microsoft::Console::Types::IUiaData * &,IRawElementProviderSimple * const &,Cursor const &,std::basic_string_view<wchar_t,std::char_traits<wchar_t> > const &>+0xcf [C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\winrt\wrl\implements.h @ 2478] 
04 000000f0`c2d0e860 00007ffc`b14725a9     Microsoft_Terminal_Control!Microsoft::Terminal::TermControlUiaProvider::CreateTextRange+0x6f [C:\a\_work\1\s\src\types\TermControlUiaProvider.cpp @ 155] 
05 000000f0`c2d0e8d0 00007ffc`b1461b2e     Microsoft_Terminal_Control!Microsoft::Console::Types::ScreenInfoUiaProviderBase::GetSelection+0x139 [C:\a\_work\1\s\src\types\ScreenInfoUiaProviderBase.cpp @ 245] 
06 000000f0`c2d0e970 00007ffc`b1461a7b     Microsoft_Terminal_Control!winrt::Microsoft::Terminal::Control::implementation::TermControlAutomationPeer::GetSelection+0x3e [C:\a\_work\1\s\src\cascadia\TerminalControl\TermControlAutomationPeer.cpp @ 171] 
07 000000f0`c2d0e9b0 00007ffc`c1d3988a     Microsoft_Terminal_Control!winrt::impl::produce<winrt::Microsoft::Terminal::Control::implementation::TermControlAutomationPeer,winrt::Windows::UI::Xaml::Automation::Provider::ITextProvider>::GetSelection+0x2b [C:\a\_work\1\s\src\cascadia\TerminalControl\Generated Files\winrt\Windows.UI.Xaml.Automation.Provider.h @ 1470] 

terminal/src/types/ScreenInfoUiaProviderBase.cpp

Line 243 in aec7392

const Cursor& cursor = _getTextBuffer().GetCursor();

_getTextBuffer() yields nullptr as it's not set up yet, the GetCursor() method resolves to +0x20 from that location and we try to read the cursor information out of 0x0000000000000020 which is completely wrong and the access violation is thrown.

The tablet input service (the soft keyboard that shows up on touch screens... also known as tabtip.exe) is making a query of the Terminal through the UIA framework while it's still loading. Before it even loaded that text buffer and got it ready.

But the tablet input service is being sneaky. They know they're calling other people's code and that it could contain bugs. So they registered a Structured Exception Handler to catch anything that could go wrong. Up to and including an access violation that would normally crash the application. The C++ Frame Handler we had installed will not catch an access violation... but an SEH one will!

Time Travel Position: 11C7B0:328 [Unindexed] Index
tiptsf!`CImmersiveFocusTracker::HandleAutomationEvent'::`1'::filt$0:
00007ffc`bcd70e08 4055            push    rbp

The touch screen keyboard handler then just marks the call as failed and gives up. But it ends that whole thread right where it stood... and doesn't let our unlock call go off in the scope exit.

terminal/src/types/ScreenInfoUiaProviderBase.cpp

Lines 221 to 244 in aec7392

	IFACEMETHODIMP ScreenInfoUiaProviderBase::GetSelection(_Outptr_result_maybenull_ SAFEARRAY** ppRetVal)
	{
	_LockConsole();
	auto Unlock = wil::scope_exit([&]() noexcept {
	_UnlockConsole();
	});
	RETURN_HR_IF_NULL(E_INVALIDARG, ppRetVal);
	*ppRetVal = nullptr;
	HRESULT hr = S_OK;
	// make a safe array
	*ppRetVal = SafeArrayCreateVector(VT_UNKNOWN, 0, 1);
	if (*ppRetVal == nullptr)
	{
	return E_OUTOFMEMORY;
	}
	WRL::ComPtr<UiaTextRangeBase> range;
	if (!_pData->IsSelectionActive())
	{
	// return a degenerate range at the cursor position
	const Cursor& cursor = _getTextBuffer().GetCursor();
	hr = CreateTextRange(this, cursor, _wordDelimiters, &range);

Then when the rest of the Terminal is given a chance to load... (see #11135 (comment) in the main thread)... it tries to take the lock to set things up. But the SEH-killed thread from the tiptsf never let it go. So it's deadlocked.

In short, it's an easy repro. Run tabtip.exe or right click the task bar to bring up the touch/tablet keyboard and then start the Terminal with that open. Or use a touch screen where it's already open all the time. That's what got us. We don't have touchscreens on our dev boxes. :(

So the culprit is #10971 (and 1.10 backport #11042) because it allows queries from the UIA before everything is ready.

Now our team will debate whether that fix is appropriate and we should revert it... whether we should patch up this particular call site and/or audit the others from tabtip.exe to see if we can guard against this circumstance... and whether we should maybe install our own SEH handler to make sure that locks get unlocked from the UIA context if everything goes sideways before someone kills the thread on us (LOOKING AT YOU tiptsf.dll!)

Thank you so much for working with us today and helping us track this down. This looks like it is going to be a big deal and a big priority for us to service. I have to head home for this evening and @zadjii-msft is already off for today, but we'll cycle through with @carlos-zamora and @DHowett in the morning and start cranking on fixes and working out our servicing story. So much appreciated that you went above and beyond in providing us all the logging, diagnostics, and tracing we could possibly need today. You're awesome!

[Sankgreall]

Ah amazing. Very happy I could help, and also want to say thank you and the rest of the team for servicing such an incredible application that we use literally every single day. Your effort is really appreciated!

[miniksa]

OK here's the plan:

We're going to prepare two fixes:

1. We're going to prepare the quick fix by guarding the nullptr dereference callsites just inside the UIA provider's public methods so if something makes a query early in the process, we should detect that the structure is empty and can answer "sorry, nothing selected" nicely (and the like) instead of having the crash.

Here's the structure output to make it easier to see which members need guarding:

0:000> dx -r1 ((Microsoft_Terminal_Control!Microsoft::Console::Types::IUiaData *)0x233fc65cfb8)
((Microsoft_Terminal_Control!Microsoft::Console::Types::IUiaData *)0x233fc65cfb8)                 : 0x233fc65cfa0 [Type: Microsoft::Terminal::Core::Terminal * (derived from Microsoft::Console::Types::IUiaData *)]
    [+0x020] _pfnWriteInput   : empty [Type: std::function<void __cdecl(std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > &)>]
    [+0x060] _pfnWarningBell  : empty [Type: std::function<void __cdecl(void)>]
    [+0x0a0] _pfnTitleChanged : empty [Type: std::function<void __cdecl(std::basic_string_view<wchar_t,std::char_traits<wchar_t> >)>]
    [+0x0e0] _pfnCopyToClipboard : empty [Type: std::function<void __cdecl(std::basic_string_view<wchar_t,std::char_traits<wchar_t> >)>]
    [+0x120] _pfnScrollPositionChanged : empty [Type: std::function<void __cdecl(int,int,int)>]
    [+0x160] _pfnBackgroundColorChanged : empty [Type: std::function<void __cdecl(til::color)>]
    [+0x1a0] _pfnCursorPositionChanged : empty [Type: std::function<void __cdecl(void)>]
    [+0x1e0] _pfnTabColorChanged : empty [Type: std::function<void __cdecl(std::optional<til::color>)>]
    [+0x220] _pfnTaskbarProgressChanged : empty [Type: std::function<void __cdecl(void)>]
    [+0x260] _stateMachine    : empty [Type: std::unique_ptr<Microsoft::Console::VirtualTerminal::StateMachine,std::default_delete<Microsoft::Console::VirtualTerminal::StateMachine> >]
    [+0x268] _terminalInput   : empty [Type: std::unique_ptr<Microsoft::Console::VirtualTerminal::TerminalInput,std::default_delete<Microsoft::Console::VirtualTerminal::TerminalInput> >]
    [+0x270] _title           : nullopt [Type: std::optional<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > >]
    [+0x298] _startingTitle   : "" [Type: std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >]
    [+0x2b8] _tabColor        : nullopt [Type: std::optional<til::color>]
    [+0x2c0] _startingTabColor : nullopt [Type: std::optional<til::color>]
    [+0x2c8] _colorTable      : { size=256 } [Type: std::array<unsigned long,256>]
    [+0x6c8] _defaultFg       : {RGB: 0, 0, 0; α: 0} [Type: til::color]
    [+0x6cc] _defaultBg       : {RGB: 0, 0, 0; α: 0} [Type: til::color]
    [+0x6d0] _defaultCursorShape : Legacy (0x0) [Type: CursorType]
    [+0x6d4] _screenReversed  : false [Type: bool]
    [+0x6d8] _blinkingState   [Type: Microsoft::Console::Render::BlinkingState]
    [+0x6f0] _snapOnInput     : false [Type: bool]
    [+0x6f1] _altGrAliasing   : false [Type: bool]
    [+0x6f2] _suppressApplicationTitle : false [Type: bool]
    [+0x6f3] _bracketedPasteMode : false [Type: bool]
    [+0x6f4] _trimBlockSelection : false [Type: bool]
    [+0x6f5] _intenseIsBright : false [Type: bool]
    [+0x6f8] _taskbarState    : 0x0 [Type: unsigned __int64]
    [+0x700] _taskbarProgress : 0x0 [Type: unsigned __int64]
    [+0x708] _hyperlinkPatternId : 0x0 [Type: unsigned __int64]
    [+0x710] _workingDirectory : "" [Type: std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >]
    [+0x730] _fontInfo        [Type: FontInfo]
    [+0x770] _selection       : nullopt [Type: std::optional<Microsoft::Terminal::Core::Terminal::SelectionAnchors>]
    [+0x77e] _blockSelection  : false [Type: bool]
    [+0x780] _wordDelimiters  : "" [Type: std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >]
    [+0x7a0] _multiClickSelectionMode : Cell (0) [Type: Microsoft::Terminal::Core::Terminal::SelectionExpansionMode]
    [+0x7a8] _readWriteLock   [Type: std::shared_mutex]
    [+0x7b0] _buffer          [Type: std::unique_ptr<TextBuffer,std::default_delete<TextBuffer> >]
    [+0x7b8] _mutableViewport : {LT(0, 0) RB(-1, -1) [0 x 0]} [Type: Microsoft::Console::Types::Viewport]
    [+0x7c0] _scrollbackLines : 0 [Type: short]
    [+0x7c4] _scrollOffset    : 0 [Type: int]
    [+0x7c8] _patternIntervalTree [Type: interval_tree::IntervalTree<til::point,unsigned __int64>]
    [+0x800] _lastKeyEventCodes : nullopt [Type: std::optional<Microsoft::Terminal::Core::Terminal::KeyEventCodes>]
    [+0x808] _sgrStack        [Type: Microsoft::Console::VirtualTerminal::SgrStack]

The quick fix will enable us to resolve the specific issue with touch keyboards and `tabtip.exe` relatively quickly. We should be able to turn it around and deploy it in the few days.

2. We're going to prepare a more comprehensive fix. This will take us potentially a week or two and might not be as easily serviceable to 1.10 or 1.11 and instead come in 1.12.

• In addition to guarding the nullptr dereferencing locations into the public UIA methods, we want to...
• Ensure we have unit or feature tests that run in our PR automation that will exercise the entire Terminal UIA provider surface from the early empty initialization state before everything comes up. I believe we have them for the "ready" state but not this new empty initialization state.
• Ensure that we have TraceLogging applied to every public UIA method so we can have people collect comprehensive logs of how they're being called in the trace files like I asked you to collect. We think we already have the provider setup, but that we fumbled at loading it into the Terminal.wprp file. So we'll get that fixed.
• Consider creating and applying a TraceLogging provider to the Terminal lock call sites so we can easily see the "Waiting", "Taken", and "Released" states interleaved with the other trace logging output. It should be on its own provider so we can easily exclude it (as it's probably going to be a perf hog and could affect timings for other tracing scenarios). Also it should be in the Terminal.wprp file... either in the existing session or in a new TerminalLocks session or something. (The specifics of this are to put it on terminalrenderdata.cpp!Terminal::LockConsole(), terminalrenderdata.cpp!Terminal::UnlockConsole(), terminal.cpp!Terminal::LockForReading(), and terminal.cpp!Terminal.LockForWriting().)

And then when we're done, we'll hold ourselves to doing a "Root Cause Analysis" to reflect on what went wrong, how we can correct it beyond this, how we can discover things like this faster, and how we can prevent issues like this in the future. Some of what we've slated for the comprehensive fix is already a part of satisfying the "how we can prevent this" and "how we can discover this faster", but we would like to be accountable for this issue.

[wkbrd]

Great to hear the progress you are all making. Once a patch is available I could try it in my Windows Server 2022 environment. Thanks!

[wkbrd]

FYI, it appears that a workaround for my scenario on Windows Server 2022 is to kill the TabTip.exe process using Task Manager before attempting to use Windows Terminal.

[miniksa]

Alright folks, I'll be preparing servicing builds for this fix early next week (mostly because preparing and launching something on Friday feels like a bad idea.) Thanks again for your help and it should be in your hands soon!

[Falco20019]

@miniksa Any timeline for when this will land in release (or a hint on how I can use the servicing build)?

[miniksa]

@miniksa Any timeline for when this will land in release (or a hint on how I can use the servicing build)?

It's rolling out now! It was sent to some inside rings on Friday and it's rolling out to the remaining rings today to be ready for the Windows launch date tomorrow. I'll have patch notes and the releases up by end of day today. And hopefully the winget manifests too.

Sorry, I'm a little behind schedule juggling things. @DHowett is out on vacation so I'm pretending to be him and I'm not as good a juggler!

[ghost]

🎉This issue was addressed in #11312, which has now been successfully released as Windows Terminal v1.10.2714.0.:tada:

Handy links:

• Release Notes
• Store Download

[ghost]

🎉This issue was addressed in #11312, which has now been successfully released as Windows Terminal Preview v1.11.2731.0.:tada:

Handy links:

• Release Notes
• Store Download

[ghost]

🎉This issue was addressed in #11312, which has now been successfully released as Windows Terminal Preview v1.12.2922.0.:tada:

Handy links:

• Release Notes
• Store Download