sos commands eestack & dumpstack crashed with .net 7.0

[VincentBu]

Description
sos commands eestack & dumpstack crashed on Debian10-arm32 with 7.0.100-preview.3.22165.18

Error Message:

        /root/lldb/bin/../lib/liblldb.so.8(+0x16463e4)[0x737773e4]
	/root/lldb/bin/../lib/liblldb.so.8(+0x161e378)[0x7374f378]
	/root/lldb/bin/../lib/liblldb.so.8(+0x1755e0c)[0x73886e0c]
	/root/lldb/bin/../lib/liblldb.so.8(_ZN4lldb10SBDebugger21RunCommandInterpreterEbb+0x44)[0x7353c00c]
	lldb[0x1c01c]
	lldb[0x1d064]
	/lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x97)[0x71e27524]
	Stack dump:
	0.      Program arguments: lldb -c core_20220316_015216
        Segmentation fault (core dumped)

[josalem]

@mikem8361

[jakobbotsch]

I ran into similar problems while trying to analyze an arm32 dump for a System.Memory.Tests failure over in dotnet/runtime#68986 (comment). The specific failure is the System.Memory.Tests work item in the net7.0-Linux-Release-arm-CoreCLR job.

runfo get-helix-payload -j 634f1283-87d8-44bc-a750-d379c0d2a400 -w System.Memory.Tests -o c:\helix_payload\System.Memory.Tests

The stack trace on crash is (for dso):

Stack dump:
0.      Program arguments: /media/windev/dotnet/arm32-lldb-patched/bin/lldb --core /media/windev/dotnet/investigations/68986/System.Memory.Tests/workitems/System.Memory.Tests/coredump.61.dmp /media/windev/dotnet/investigations/68986/System.Memory.Tests/shared/Microsoft.NETCore.App/7.0.0/dotnet -o setclrpath /media/windev/dotnet/investigations/68986/System.Memory.Tests/shared/Microsoft.NETCore.App/7.0.0 -o setsymbolserver -directory /media/windev/dotnet/investigations/68986/System.Memory.Tests/shared/Microsoft.NETCore.App/7.0.0
 #0 0x0004542c PrintStackTraceSignalHandler(void*) (/media/windev/dotnet/arm32-lldb-patched/bin/lldb+0x4542c)
 #1 0x00042c2c llvm::sys::RunSignalHandlers() (/media/windev/dotnet/arm32-lldb-patched/bin/lldb+0x42c2c)
 #2 0x00045d0c SignalHandler(int) (/media/windev/dotnet/arm32-lldb-patched/bin/lldb+0x45d0c)
 #3 0xf1b4e01e (/root/.dotnet/shared/Microsoft.NETCore.App/6.0.3/libcoreclr.so+0x3ca01e)
 #4 0xf1eb75a0 __default_rt_sa_restorer (/lib/arm-linux-gnueabihf/libc.so.6+0x275a0)
 #5 0xba43ad82 MethodTable::GetClass_NoLogging() /__w/1/s/src/coreclr/vm/methodtable.inl:36:67
 #6 0xba43ad82 MethodTable::GetClassWithPossibleAV() /__w/1/s/src/coreclr/vm/methodtable.inl:69:12
 #7 0xba43ad82 MethodTable::ValidateWithPossibleAV() /__w/1/s/src/coreclr/vm/methodtable.cpp:493:34
 #8 0xba4924fe DacValidateMethodTable(MethodTable*, int&) /__w/1/s/src/coreclr/debug/daccess/request.cpp:151:17
 #9 0xba49ac88 ClrDataAccess::GetObjectData(unsigned long long, DacpObjectData*) /__w/1/s/src/coreclr/debug/daccess/request.cpp:2171:13
#10 0xf1c6c1ac (/root/.dotnet/sos/libsos.so+0x631ac)
#11 0xf1c6c9bc (/root/.dotnet/sos/libsos.so+0x639bc)
#12 0xf1c6caee (/root/.dotnet/sos/libsos.so+0x63aee)
#13 0xf1c4a5c6 (/root/.dotnet/sos/libsos.so+0x415c6)
#14 0xf1c4a736 DumpStackObjects (/root/.dotnet/sos/libsos.so+0x41736)
#15 0xf1e055f8 sosCommand::DoExecute(lldb::SBDebugger, char**, lldb::SBCommandReturnObject&) (/root/.dotnet/sos/libsosplugin.so+0xd5f8)
#16 0xf3b3bf34 CommandPluginInterfaceImplementation::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) (/media/windev/dotnet/arm32-lldb-patched/bin/../lib/liblldb.so.10+0x1959f34)
#17 0xf3f0aae8 lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) (/media/windev/dotnet/arm32-lldb-patched/bin/../lib/liblldb.so.10+0x1d28ae8)
#18 0xf3efb0d4 lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&, lldb_private::ExecutionContext*, bool, bool) (/media/windev/dotnet/arm32-lldb-patched/bin/../lib/liblldb.so.10+0x1d190d4)
#19 0xf3f00450 lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (/media/windev/dotnet/arm32-lldb-patched/bin/../lib/liblldb.so.10+0x1d1e450)
#20 0xf3e15494 lldb_private::IOHandlerEditline::Run() (/media/windev/dotnet/arm32-lldb-patched/bin/../lib/liblldb.so.10+0x1c33494)
#21 0xf3dee634 lldb_private::Debugger::ExecuteIOHandlers() (/media/windev/dotnet/arm32-lldb-patched/bin/../lib/liblldb.so.10+0x1c0c634)
#22 0xf3f019cc lldb_private::CommandInterpreter::RunCommandInterpreter(bool, bool, lldb_private::CommandInterpreterRunOptions&) (/media/windev/dotnet/arm32-lldb-patched/bin/../lib/liblldb.so.10+0x1d1f9cc)
#23 0xf3b80f54 lldb::SBDebugger::RunCommandInterpreter(bool, bool) (/media/windev/dotnet/arm32-lldb-patched/bin/../lib/liblldb.so.10+0x199ef54)
#24 0x0001e19c Driver::MainLoop() (/media/windev/dotnet/arm32-lldb-patched/bin/lldb+0x1e19c)
#25 0x0001fa18 main (/media/windev/dotnet/arm32-lldb-patched/bin/lldb+0x1fa18)
#26 0xf1ea75a4 __libc_start_main (/lib/arm-linux-gnueabihf/libc.so.6+0x175a4)
Segmentation fault (core dumped)

I get similar crashes (inside GetClassWithPossibleAV) when using dumpstack/eestack.

[mikem8361]

dumpstack/eestack can be very unreliable because they attempt to dump the raw stack a pointer at a time. It calls the DAC when it attempts to see if the pointer is a valid GC object which attempts to validate any the memory but that doesn't work well on Linux/MacOS because we can't use the C++ exception handling to catch h/w (invalid access) exceptions like we can on Windows.

So I recommend using "clrstack -f" which uses the managed and native (works only under lldb) unwinders to dump the stack.

[mikem8361]

When this is fixed, re-enable the SOS dumpstack/eestack tests in "StackTests.script" currently disabled for 7.0 runtimes, under dotnet-dump and arm32 architectures.

[mikem8361]

This should be fixed in 7.0.0 with PR dotnet/runtime#73426.

[mikem8361]

Fixed in .NET 8.0 runtime.