dbghelp!MiniDumpWriteDump exits silently when called from client

[derekbruening]

From [email protected] on October 22, 2013 16:50:30

A client that finds dbghelp!MiniDumpWriteDump like this:

hLib = LoadLibraryA("C:\\Program Files (x86)\\Windows Kits\\8.0\\Debuggers\\x86\\dbghelp.dll");
DR_ASSERT(0 != hLib);
pMiniDumpWriteDump = (PMDWD)GetProcAddress(hLib, "MiniDumpWriteDump");

And invokes it, ends up exiting silently.

** TODO windbg 6.3 syms messed up

0:000> x minidump!pM*
70974570 minidump!pMiniDumpWriteDump = 0x63e1abed
0:000> x dbghelp!MiniDumpWriteDump
63e170d8 dbghelp!MiniDumpWriteDump =

privload_load_finalize: loaded dbghelp.dll @ 0x63dd0000-0x63efb000 from C:\Program Files (x86)\Windo
ws Kits\8.0\Debuggers\x86\dbghelp.dll
redirect_GetProcAddress: 0x63dd0000MiniDumpWriteDump
redirect_GetProcAddress: MiniDumpWriteDump => 0x63e1abed

0:000> ln 0x63e1abed
(63e1aa7b) dbghelp!GenRemoveMemoryRange+0x172 | (63e1abf1) dbghelp!GenAddPebMemory
0:000> U 0x63e1abed
dbghelp!GenRemoveMemoryRange+0x172:
63e1abed 8bff mov edi,edi
63e1abef 55 push ebp

ok, that's something weird w/ windbg 6.3 -- w/ recent windbg we have:

0:000> x dbghelp!MiniDumpWrite*
63e1abed dbghelp!MiniDumpWriteDump ()

** TODO it's a non-access violation exit, so DR doesn't report it as a client crash

0:000> kn

ChildEBP RetAddr

00 251ee7d0 63e39d02 KERNEL32_2b0000!UnhandledExceptionFilter+0x5f
01 251eeb00 63e213db dbghelp!__report_gsfailure+0xc8
02 251eebe8 cdcdcdcd dbghelp!Win32LiveSystemProvider::GetOsInfo+0x1b6
WARNING: Frame IP not in any known module. Following frames may be wrong.
03 251eec10 63e1a92b 0xcdcdcdcd
04 251eed94 63e1adbb dbghelp!MiniDumpProvideDump+0x12a
05 251eee10 7097125f dbghelp!MiniDumpWriteDump+0x1ce
06 251eeeb4 251b109c minidump!snap_cb3+0x8f [c:\src\dr\bugs\minidump\minidump.c @ 102]

0:000> .exr -1
ExceptionAddress: 002fff55 (KERNEL32_2b0000!UnhandledExceptionFilter+0x0000005f)
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 1
Parameter[0]: 00000000

Entry:
dbghelp!Win32LiveSystemProvider::GetOsInfo:
63e21225 8bff mov edi,edi
63e21227 55 push ebp
63e21228 8bec mov ebp,esp
63e2122a 81ece0000000 sub esp,0E0h
63e21230 a100d0ec63 mov eax,dword ptr [dbghelp!__security_cookie (63ecd000)]
63e21235 33c5 xor eax,ebp
63e21237 8945fc mov dword ptr [ebp-4],eax

Exit:
dbghelp!Win32LiveSystemProvider::GetOsInfo+0x1aa:
63e213cf 8b4dfc mov ecx,dword ptr [ebp-4]
63e213d2 5e pop esi
63e213d3 33cd xor ecx,ebp
63e213d5 5b pop ebx
63e213d6 e8b8870100 call dbghelp!__security_check_cookie (63e39b93)
63e213db c9 leave
63e213dc c21c00 ret 1Ch

0:000> Uf 63e39b93
dbghelp!__security_check_cookie:
63e39b93 3b0d00d0ec63 cmp ecx,dword ptr [dbghelp!__security_cookie (63ecd000)]
63e39b99 7503 jne dbghelp!__security_check_cookie+0xb (63e39b9e)

dbghelp!__security_check_cookie+0x8:
63e39b9b c20000 ret 0

dbghelp!__security_check_cookie+0xb:
63e39b9e e997000000 jmp dbghelp!__report_gsfailure (63e39c3a)

dbghelp!__report_gsfailure:
63e39c3a 8bff mov edi,edi
...
63e39cef 6a00 push 0
63e39cf1 ff15a4a3ee63 call dword ptr [dbghelp!_imp__SetUnhandledExceptionFilter (63eea3a4)]
63e39cf7 687c10dd63 push offset dbghelp!__xi_z+0x2c (63dd107c)
63e39cfc ff15a8a3ee63 call dword ptr [dbghelp!_imp__UnhandledExceptionFilter (63eea3a8)]
63e39d02 c785dcfcffff01000000 mov dword ptr [ebp-324h],1
63e39d0c 68090400c0 push 0C0000409h
63e39d11 ff1524a4ee63 call dword ptr [dbghelp!_imp__GetCurrentProcess (63eea424)]
63e39d17 50 push eax
63e39d18 ff15a0a3ee63 call dword ptr [dbghelp!_imp__TerminateProcess (63eea3a0)]
63e39d1e c9 leave
63e39d1f c3 ret

0:000> r
eax=00000000 ebx=24a7ed20 ecx=e96a2625 edx=00000000 esi=00000000 edi=24a7ecf8
0:000> dd dbghelp!__security_cookie
63ecd000 addcb782 5223487d 00000000 00000000

** TODO the culprit bug

Stepping through it, ebp-4 is clobbered after the xor'ed value is stored
there by our own memcpy:

0:000> bl
0 e 63e21225 0001 (0001) 0:* dbghelp!Win32LiveSystemProvider::GetOsInfo
1 e 19e9ebe4 w 4 0001 (0001) 0:*
2 e 63ecd000 w 4 0001 (0001) 0:* dbghelp!__security_cookie
3 e 63e213cf 0001 (0001) 0:* dbghelp!Win32LiveSystemProvider::GetOsInfo+0x1aa
0:000> g
Breakpoint 1 hit
0:000> kn

ChildEBP RetAddr

00 19e9e8b0 64237e9f ntdll!memcpy+0x33
01 19e9eae0 63e21391 dynamorio!redirect_RegQueryValueExA+0x18f [c:\src\dr\git\src\core\win32\drwinapi\advapi32_redir.c @ 290]
02 19e9ebe8 63e1a620 dbghelp!Win32LiveSystemProvider::GetOsInfo+0x16c
03 19e9ec10 63e1a92b dbghelp!GetSystemType+0xa2
04 19e9ed94 63e1adbb dbghelp!MiniDumpProvideDump+0x12a
05 19e9ee10 7097125f dbghelp!MiniDumpWriteDump+0x1ce
06 19e9eeb4 19e6109c minidump!snap_cb3+0x8f [c:\src\dr\bugs\minidump\minidump.c @ 102]
0:000> .frame 1
01 19e9eae0 63e21391 dynamorio!redirect_RegQueryValueExA+0x18f [c:\src\dr\git\src\core\win32\drwinapi\advapi32_redir.c @ 290]
0:000> dv
buf = 0x19ee4c44 "Multiprocessor Free"
prev_sofar = 0
sofar = 0x14
hKey = 0x00000094
lpValueName = 0x63de81f8 "CurrentType"
lpReserved = 0x00000000
lpType = 0x19e9eb14
lpData = 0x19e9ebbc "Multiprocessor Free"
lpcbData = 0x19e9eb18
wbuf = wchar_t [255] "CurrentType"
len = 0n11
max_len = 0x28
type = 1
res = 0n0
0:000> ?? *lpcbData
unsigned long 0x34
0:000> ?? lpData + *lpcbData
unsigned char * 0x19e9ebf0

dbghelp!Win32LiveSystemProvider::GetOsInfo+0x13d:
63e21362 8d8530ffffff lea eax,[ebp-0D0h]
63e21368 50 push eax
63e21369 8d45d4 lea eax,[ebp-2Ch]
63e2136c 50 push eax
63e2136d 8d852cffffff lea eax,[ebp-0D4h]
63e21373 50 push eax
63e21374 6a00 push 0
63e21376 68f881de63 push offset dbghelp!`string' (63de81f8)
63e2137b ffb534ffffff push dword ptr [ebp-0CCh]
63e21381 c78530ffffff28000000 mov dword ptr [ebp-0D0h],28h
63e2138b ff1508a3ee63 call dword ptr [dbghelp!_imp__RegQueryValueExA (63eea308)]

hmm, so the max len was upped from 0x28 to 0x34 by
redirect_RegQueryValueExA? ok, it fails to remove the rest of
KEY_VALUE_PARTIAL_INFORMATION in redirect_RegQueryValueExW.

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=1292

[derekbruening]

From [email protected] on October 23, 2013 11:44:03

I split out some other issues hit after fixing this bug into their own issues. This is a good stress test of some missing pieces in our private library isolation and redirection:

• issue ASSERT loader_shared.c:390 !DATASEC_PROTECTED(DATASEC_RARELY_PROT) with privlib loaded after DR init #1293 + issue handle user32 forwarder to ntdll having been patched #1294 + issue infinite loop loading imm32.dll in client due to load-by-path not finding an already-loaded lib of the same path #1295 + issue ASSERT callback.c:2841 os_using_app_state(dcontext) due to missing FreeLibrary redirect #1296 Along with some missing redirection that's really part of issue redirect more of ntdll for more transparent private libraries #235 which I will add separately here

[derekbruening]

From [email protected] on October 23, 2013 11:44:13

** DONE (issue #235) now MiniDumpWriteDump executes, but returns failure b/c we don't redirect LdrGetProcedureAddress
CLOSED: [2013-10-23 Wed 12:36]

• State "DONE" from "TODO" [2013-10-23 Wed 12:36]

GetLastError() is:
#define E_NOTIMPL HRESULT_TYPEDEF(0x80004001L)

works natively.

*** DONE even on app using user32, still get 0x80004001: tracking it down
CLOSED: [2013-10-23 Wed 12:30]
- State "DONE" from "TODO" [2013-10-23 Wed 12:30]

This guy returns 0x80004001, and MiniDumpWriteDump, who called
MiniDumpProvideDump, passes that result back:

0:000> U 63e1a971
dbghelp!MiniDumpProvideDump+0x170:
63e1a971 e83a400000 call dbghelp!GenGetProcessInfo (63e1e9b0)

Continuing in to callees:

dbghelp!GenGetProcessInfo+0x43:
63e1e9f3 ff9084000000 call dword ptr [eax+84h] ds:002b:63de88f4={dbghelp!NtWin32LiveSystemProvider::StartProcessEnum (63e23310)}
0:000> p
eax=80004001 ebx=240ced00 ecx=00000000 edx=00000001 esi=00000000 edi=00000000

dbghelp!NtWin32LiveSystemProvider::StartProcessEnum+0x13:
63e23323 e8adebffff call dbghelp!Win32LiveSystemProvider::StartProcessEnum (63e21ed5)
0:000> p
eax=80004001 ebx=2172ed00 ecx=00000000 edx=00000001 esi=2177d024 edi=ffffffff

Here is where the 0x80004001 is set:

0:000> U 63e21ed5
dbghelp!Win32LiveSystemProvider::StartProcessEnum:
63e21ed5 8bff mov edi,edi
63e21ed7 55 push ebp
63e21ed8 8bec mov ebp,esp
63e21eda 56 push esi
63e21edb 8bf1 mov esi,ecx
63e21edd 8b4e30 mov ecx,dword ptr [esi+30h]
63e21ee0 85c9 test ecx,ecx
63e21ee2 750a jne dbghelp!Win32LiveSystemProvider::StartProcessEnum+0x19 (63e21eee)
63e21ee4 b801400080 mov eax,80004001h
63e21ee9 e982000000 jmp dbghelp!Win32LiveSystemProvider::StartProcessEnum+0x9b (63e21f70)

ecx is passed through from GenGetProcessInfo, who gets it from its 1st
param w/ a deref + 8:
0:000> U 63e1e9c9 L20
dbghelp!GenGetProcessInfo+0x19:
63e1e9c9 8b5d08 mov ebx,dword ptr [ebp+8]
63e1e9cc 56 push esi
63e1e9cd 57 push edi
63e1e9ce 8985e4fbffff mov dword ptr [ebp-41Ch],eax
63e1e9d4 8d8520fcffff lea eax,[ebp-3E0h]
63e1e9da 50 push eax
63e1e9db 53 push ebx
63e1e9dc e8cbeaffff call dbghelp!GenAllocateProcessObject (63e1d4ac)
63e1e9e1 85c0 test eax,eax
63e1e9e3 0f856e070000 jne dbghelp!GenGetProcessInfo+0x7a7 (63e1f157)
63e1e9e9 ff7304 push dword ptr [ebx+4]
63e1e9ec 8b4b08 mov ecx,dword ptr [ebx+8]
63e1e9ef ff33 push dword ptr [ebx]
63e1e9f1 8b01 mov eax,dword ptr [ecx]
63e1e9f3 ff9084000000 call dword ptr [eax+84h]

ultimately coming from here:
0:000> U 63e1a966
dbghelp!MiniDumpProvideDump+0x165:
63e1a966 8d4520 lea eax,[ebp+20h]
63e1a969 50 push eax
63e1a96a 8d8564ffffff lea eax,[ebp-9Ch]
63e1a970 50 push eax
63e1a971 e83a400000 call dbghelp!GenGetProcessInfo (63e1e9b0)

is it dbghelp!GenAllocateProcessObject who sets it up?
no, ebp-9c+8 is written here:
0:000> U 63e1a8b5
dbghelp!MiniDumpProvideDump+0xb1:
63e1a8b2 8b4510 mov eax,dword ptr [ebp+10h]
63e1a8b5 89856cffffff mov dword ptr [ebp-94h],eax

ebp+10 is 3rd param, which is esp+28 in caller:
dbghelp!MiniDumpWriteDump+0x1ab:
63e1ad98 ff742430 push dword ptr [esp+30h]
63e1ad9c 8b5c2418 mov ebx,dword ptr [esp+18h]
63e1ada0 50 push eax
63e1ada1 ff7514 push dword ptr [ebp+14h]
63e1ada4 8b44242c mov eax,dword ptr [esp+2Ch]
63e1ada8 52 push edx
63e1ada9 57 push edi
63e1adaa 53 push ebx
63e1adab ff742428 push dword ptr [esp+28h]
63e1adaf ff750c push dword ptr [ebp+0Ch]
63e1adb2 ff742438 push dword ptr [esp+38h]
63e1adb6 e846faffff call dbghelp!MiniDumpProvideDump (63e1a801)

at that point, esp is -0x24 from after the sub of 0x44 at entry
=> that's esp+4 from esp after the sub

it's zeroed here:
0:000> U 63e1ac2f
dbghelp!MiniDumpWriteDump+0x42:
63e1ac2f 895c2410 mov dword ptr [esp+10h],ebx

then written here:
0:000> U 63e229f2 L15
dbghelp!MiniDumpCreateLiveSystemProvider+0x101:
63e22a1e 8b8564ffffff mov eax,dword ptr [ebp-9Ch]
63e22a24 8930 mov dword ptr [eax],esi

0:000> dd esi
00310808 63de8870 003107e0 00000002 00001db1
00310818 75bf0000 75c111e8 75c85bc3 75c85c6f
00310828 75c85d09 75c85df2 75c27991 75c27d2e
00310838 75c272f7 75c843af 75c0a2b5 75c1d5a7
00310848 75c045fa 75c2796c 00000001 00000000
00310858 00000000 00000001 00000000 00000000
0:000> U 63de8870
dbghelp!NtWin32LiveSystemProvider::`vftable':

So it's a C++ object instance, and the field at +0x30 stores:

0:000> U poi(esi+30)
kernel32!CreateToolhelp32Snapshot:
75c272f7 6a40 push 40h

Object is allocated in dbghelp!NewNtWin32LiveSystemProvider
Constructor: dbghelp!Win32LiveSystemProvider::Win32LiveSystemProvider

That field is filled in here:
dbghelp!Win32LiveSystemProvider::Initialize+0x78:
63e20c13 688c5cde63 push offset dbghelp!string' (63de5c8c) 63e20c18 ff7710 push dword ptr [edi+10h] 63e20c1b 89472c mov dword ptr [edi+2Ch],eax 63e20c1e ffd6 call esi 63e20c20 68dc80de63 push offset dbghelp!string' (63de80dc)
63e20c25 ff7710 push dword ptr [edi+10h]
63e20c28 894730 mov dword ptr [edi+30h],eax
0:000> da 63de5c8c
63de5c8c "CreateToolhelp32Snapshot"

0:000> U esi
dbghelp!kernel32_GetProcAddress_Thunk:
63e2521d 68c158e263 push offset dbghelp!GodotFailGetProcAddress (63e258c1)
63e25222 68501fed63 push offset dbghelp!_imp__GetProcAddress (63ed1f50)
63e25227 68948bde63 push offset dbghelp!string' (63de8b94) \<= "GetProcAddress" 63e2522c 685c5cde63 push offset dbghelp!string' (63de5c5c) <= "kernel32.dll"
63e25231 a16c97ee63 mov eax,dword ptr [dbghelp!Unicows_GetProcAddress (63ee976c)]
63e25236 e89c030000 call dbghelp!ResolveThunk (63e255d7)

0:000> dd edi+10
00290818 75bf0000

75bf0000 75d00000 kernel32 (pdb symbols) c:\src\symbols\wkernel32.pdb\139CA12C1AB645F6A7F2DD1A098696692\wkernel32.pdb

0:000> U poi(dbghelp!_imp__GetProcAddress)
KERNEL32_2370000!GetProcAddressStub:
02381222 8bff mov edi,edi
02381224 55 push ebp
02381225 8bec mov ebp,esp
02381227 5d pop ebp
02381228 eb05 jmp KERNEL32_2370000!GetProcAddress (0238122f)

*** DONE (issue #235) now, what goes wrong w/ that custom GetProcAddress? => have to intercept LdrGetProcedureAddress
CLOSED: [2013-10-23 Wed 12:30]
- State "DONE" from "TODO" [2013-10-23 Wed 12:30]

It is passing private kernel32 base.

Xref issue #450 where I encountered this dbghelp!ResolveThunk and concluded I
needed to hook priv kernel32's GetModuleFileNameW -- that was different
though I believe.

Here it ends up at priv kernel32 GetProcAddressStub.
So we need a hook there, or in priv kernel32 GetProcAddress?
Returns NULL today presumably b/c it invokes LdrGetProcedureAddress: so we
just need to redirect that?

• FIXME issue redirect more of ntdll for more transparent private libraries #235: redirect the Ldr* routines, incl LdrGetProcedureAddress.

0:000> U 002a1e33
KERNELBASE_290000!GetProcAddress+0x2c:
002a1e33 8d450c lea eax,[ebp+0Ch]
002a1e36 50 push eax
002a1e37 57 push edi
002a1e38 6a00 push 0
002a1e3a 6a00 push 0
002a1e3c ff7508 push dword ptr [ebp+8]
002a1e3f e8be5e0200 call KERNELBASE_290000!BasepMapModuleHandle (002c7d02)
002a1e44 50 push eax
002a1e45 ff15a8122900 call dword ptr [KERNELBASE_290000!_imp__LdrGetProcedureAddress (002912a8)]

It passes 0 for ordinal at least:
0:000> dds esp
1ed0eccc 02370000 KERNEL32_2370000!_imp__DebugBreak (KERNEL32_2370000+0x0)
1ed0ecd0 1ed0ece4
1ed0ecd4 00000000
1ed0ecd8 1ed0ecf8

[derekbruening]

From [email protected] on October 23, 2013 11:44:44

** DONE (issue #235) have to intercept LdrLoadDll to catch kernelbase calling its own LoadLibraryExW
CLOSED: [2013-10-23 Wed 14:44]

• State "DONE" from "TODO" [2013-10-23 Wed 14:44]

Still working on the client in issue #1292 , we now hit an ASSERT
(os_using_app_state) from DR's LdrLoadDll hook:

0:000> kn

ChildEBP RetAddr

00 1a9ed6a4 636adf4e ntdll!ZwRaiseHardError+0x12
01 1a9ed6f4 636766dd dynamorio!nt_messagebox+0xfe [c:\src\dr\git\src\core\win32\ntdll.c @ 3664]
02 1a9ed71c 634d9354 dynamorio!debugbox+0x4d [c:\src\dr\git\src\core\win32\os.c @ 4606]
03 1a9edf38 634d9853 dynamorio!notify+0x1e4 [c:\src\dr\git\src\core\utils.c @ 1946]
04 1a9edfcc 634d4544 dynamorio!report_dynamorio_problem+0x4a3 [c:\src\dr\git\src\core\utils.c @ 2205]
05 1a9ee104 63696d7e dynamorio!internal_error+0x124 [c:\src\dr\git\src\core\utils.c @ 183]
06 1a9ee14c 637e9f79 dynamorio!asynch_take_over+0x24e [c:\src\dr\git\src\core\win32\callback.c @ 2841]
07 1a9ee2e0 022a66b5 dynamorio!interception_code_array+0xf79
08 1a9ee510 022a6758 KERNELBASE_2270000!CreateSortVersionDllTableEntry+0x76
09 1a9ee524 022a6819 KERNELBASE_2270000!GetSortVersionHandle+0x25
0a 1a9ee540 022a6886 KERNELBASE_2270000!MakeSortHashNode+0x50
0b 1a9ee550 022a6905 KERNELBASE_2270000!GetInvariantSort+0x26
0c 1a9ee560 022a6a83 KERNELBASE_2270000!GetSortNode+0x32
0d 1a9ee570 02292ecb KERNELBASE_2270000!SortCompareString+0x27
0e 1a9ee59c 7333169f KERNELBASE_2270000!CompareStringW+0x38
0f 1a9ee5e0 73331b69 VERSION!VerpQueryValue+0x122
10 1a9ee5fc 63e20e78 VERSION!VerQueryValueW+0x18
11 1a9ee848 63e189b1 dbghelp!Win32LiveSystemProvider::QueryBuildString+0xe7
12 1a9eebb4 63e1a02a dbghelp!WriteMiscInfo+0x129
13 1a9eec04 63e1aa97 dbghelp!WriteDumpData+0xb4
14 1a9eed9c 63e1adbb dbghelp!MiniDumpProvideDump+0x296
15 1a9eee18 740f1255 dbghelp!MiniDumpWriteDump+0x1ce
16 1a9eeeb4 1a9b109c minidump!snap_cb3+0xa5 [c:\src\dr\bugs\minidump\minidump.c @ 107]

intercept_load_dll: kernel32.dll

0:000> U 022a66b5 -5
KERNELBASE_2270000!CreateSortVersionDllTableEntry+0x71:
022a66b0 e8efc3fdff call KERNELBASE_2270000!LoadLibraryExW (02282aa4)

LoadLibraryExW: file handle arg must be NULL, so it's just the flags, and
CreateSortVersionDllTableEntry passes 0 for those too. So we could just
implement as part of issue #1063.

Yet here it's kernelbase code calling its own routine, so import
redirection makes no difference and won't solve this:

0:000> U 002c66b5 -5
KERNELBASE_290000!CreateSortVersionDllTableEntry+0x71:
002c66b0 e8efc3fdff call KERNELBASE_290000!LoadLibraryExW (002a2aa4)

So we need a hook. Xref hooks needed for other cases: issue #450.

Although this is a simple replacement: so we just need a jmp to our
redirect routine and not a full, general, monitoring hook. And we never
have to remove it.

Or, can we just redirect LdrLoadDll? How much work is kernelbase doing?

[derekbruening]

From [email protected] on October 23, 2013 11:46:17

After:

• issue CRASH drwinapi_onload when privlib loads executable #1297 Now MiniDumpWriteDump works, at least for an app that loads user32.dll.

[derekbruening]

From [email protected] on October 23, 2013 11:59:28

The remaining issues, none of which are likely to be solvable easily, are:

• Issue 1299 : if the app does not load user32.dll, we end up crashing. We don't have a simple fix for this one and it's beyond the scope of this issue here.
• Should DR report this non-access violation exit somehow? It's not clear how DR would distinguish it from a deliberate, expected exit though, one that the client wants to happen.
• The client cannot get the proper app state, as calling dr_switch_to_app_state(drcontext) prior to the call to
  MiniDumpWriteDump results in a correct assert due to it being unsafe (as documented for dr_switch_to_app_state) to be in the app state while invoking priv lib routines.

[derekbruening]

From [email protected] on October 23, 2013 12:44:23

This issue was closed by revision r2336 .

Status: Fixed

[derekbruening]

From [email protected] on October 24, 2013 07:30:11

I split the app state vs priv state as its own issue #1301