Hang/deadlock in wait_for_notifiers

[nirinchev]

Description

The Osu team reported seeing hangs when running their unit tests. I've reproduced it locally and this is the stacktrace that I'm seeing:

 	ntdll.dll!00007ff8f69d0014()	Unknown
 	ntdll.dll!00007ff8f69950c1()	Unknown
 	KernelBase.dll!00007ff8f41eba89()	Unknown
>	realm-wrappers.dll!__crtSleepConditionVariableSRW(_RTL_CONDITION_VARIABLE * pCond, _RTL_SRWLOCK * pLock, unsigned long dwMs, unsigned long flags) Line 468	C++
 	realm-wrappers.dll!Concurrency::details::stl_condition_variable_win7::wait_for(Concurrency::details::stl_critical_section_interface * lock, unsigned int timeout) Line 181	C++
 	realm-wrappers.dll!Concurrency::details::stl_condition_variable_win7::wait(Concurrency::details::stl_critical_section_interface * lock) Line 175	C++
 	realm-wrappers.dll!_Cnd_wait(_Cnd_internal_imp_t * cond, _Mtx_internal_imp_t * mtx) Line 62	C++
 	realm-wrappers.dll!realm::_impl::RealmCoordinator::wait_for_notifiers<bool <lambda>(void)>(realm::_impl::NotifierPackage::package_and_wait::__l2::bool <lambda>(void) && wait_predicate) Line 268	C++
 	realm-wrappers.dll!realm::_impl::NotifierPackage::package_and_wait(realm::util::Optional<unsigned __int64> target_version) Line 459	C++
 	realm-wrappers.dll!realm::_impl::RealmCoordinator::advance_to_latest(realm::Realm & realm) Line 1127	C++
 	realm-wrappers.dll!realm::Realm::do_refresh() Line 834	C++
 	realm-wrappers.dll!realm::Realm::refresh() Line 801	C++
 	realm-wrappers.dll!shared_realm_refresh::__l2::<lambda>() Line 304	C++
 	realm-wrappers.dll!realm::handle_errors<bool <lambda>(void)>(realm::NativeException::Marshallable & ex, shared_realm_refresh::__l2::bool <lambda>(void) && func) Line 127	C++
 	realm-wrappers.dll!shared_realm_refresh(std::shared_ptr<realm::Realm> & realm, realm::NativeException::Marshallable & ex) Line 305	C++
 	[Managed to Native Transition]	
 	Realm.dll!Realms.SharedRealmHandle.Refresh() Line 273	C#

Steps to reproduce

1. Clone or rename your .NET repo clone to *someplace*/dotnet.
2. git clone https://github.com/nirinchev/osu.git adjacent to the dotnet repo - i.e. *someplace*/osu.
3. Build wrappers for Windows/Debug/x64: powershell wrappers\build.ps1 Windows -Platforms x64 -Configuration Debug
4. git checkout ni/hookup-realm-project - this branch replaces the Realm dependency from Nuget with a local reference to the Realm project, wrappers, Core, and OS and adds the native dependencies that were built in step 3.
5. Open cmd and set process affinity to 2-4 cores.
6. dotnet test --filter Editor in the osu root directory - this will spawn 5 testhost processes that will try to run the tests (ignore the errors related to Xamarin.Android).
7. Wait 30-40 seconds for the processes to hang.
8. Now you can attach a debugger to any of the testhost processes - the deadlocked thread is named GameThread.Update.

[peppy]

As a heads-up, we look to be hitting similar deadlock cases, this time on linux (but only some distributions). A native stack trace is posted here: ppy/osu#11461 (comment).

As this issue is tracking the original windows-specific case, let me know if you'd prefer we open a new tracking issue for this.

[nirinchev]

I think the hangs may be related to a change in the Linux kernel reported here: realm/realm-core#4666. We do have a fix and we'll probably do another beta release this week or early next one.

[nirinchev]

@peppy I can see the osu! PR has been merged so assuming this is now resolved, so I'll be closing this issue - if you run into other problems, do let us know :)