Core dump generation FAILED 0x80004005 and orphaned process

[Sean-Driscoll]

On a RapspberryPI 3B+ with Raspbian OS (ARM32) I'm running a self-contained ASP.NET Core 3.0 app and currently investigating debugging it. The command and error message are as follows

pi@raspberrypi:~ $ sudo /home/pi/.dotnet/tools/dotnet-dump collect -p <pid>
Writing minidump with heap to /home/pi/core_20191118_083149
Core dump generation FAILED 0x80004005

Each time the command is run it results in what seems to be an orphaned process, which I either kill with sudo kill -9 <newPid> or have to restart the host application. Unlike one of the other related issues I found the dump file is not created.

In trying to understand the current landscape of .NET Core debugging, I've tried using both lldb and dotnet-dump for analyzing dump files, and tried creating dump files using gcore, dotnet-dump, and createdump with no success thus far. I've also tried live-debugging in lldb with no success (tried versions 5.0, 6.0, and the current 7.0.1). I thought using the dotnet-dump tool would eliminate any versioning issues between the various dump generation/analysis methods and the host process but I'm stuck on the issue described above. In all other cases I've been able to generate dump files, load them, load the sos plugin, but running an actual SOS command always fails with a load issue on one of the lib core so files, regardless of what I setclrpath to. It seems like the wild, wild west for .NET Core debugging - little, sparse, and out-dated documentation. I decided to start here but any help is much appreciated. I have several other questions to try and navigate all the information I've read, what is the best way to surface those questions?

[mikem8361]

You should not need "sudo" on the dotnet-dump collect but regardless I'll have to investigate your failure. Could you add the "--diag" option to your dotnet-dump collect command and capture the output of the console where your ASP.NET Core app is running?

I tried this successfully on my pi3 arm32 raspbian v10 with the dotnet 3.0.100 SDK and version 3.0.52901 of dotnet-dump.

We have not found a lldb that works for ARM32 yet. lldb 9.0 looks promising but there are still some pending problems.

[Sean-Driscoll]

The app runs as a systemd daemon under a 'service' account. I'm not sure where the output, using the --diag option, would show up. I looked in the apps logs, logs from journalctl, and I also ran the application interactively in the console but did not see any messages from dotnet-dump.

I suppose I should have added this to the initial message but I just assumed sudo was required... not running as sudo throws the following

Unhandled exception: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (13): Permission denied /tmp/dotnet-diagnostic-19940-223819709-socket
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.GetTransport(Int32 processId) in //src/Microsoft.Diagnostics.Tools.RuntimeClient/DiagnosticsIpc/IpcClient.cs:line 54
at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.SendMessage(Int32 processId, IpcMessage message) in //src/Microsoft.Diagnostics.Tools.RuntimeClient/DiagnosticsIpc/IpcClient.cs:line 67
at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsHelpers.GenerateCoreDump(Int32 processId, String dumpName, DumpType dumpType, Boolean diagnostics) in //src/Microsoft.Diagnostics.Tools.RuntimeClient/DiagnosticsHelpers.cs:line 46
at Microsoft.Diagnostics.Tools.Dump.Dumper.Collect(IConsole console, Int32 processId, String output, Boolean diag, DumpTypeOption type) in //src/Tools/dotnet-dump/Dumper.cs:line 67
at System.CommandLine.Invocation.CommandHandler.GetResultCodeAsync(Object value, InvocationContext context)
at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass2_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.CommandLine.Invocation.InvocationExtensions.<>c.<b__16_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.CommandLine.Invocation.InvocationExtensions.<>c__DisplayClass8_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.CommandLine.Invocation.InvocationExtensions.<>c.<b__7_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.CommandLine.Invocation.InvocationExtensions.<>c.<b__6_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.CommandLine.Invocation.InvocationExtensions.<>c.<b__14_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass3_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.CommandLine.Invocation.InvocationExtensions.<>c.<b__17_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.CommandLine.Invocation.InvocationExtensions.<>c__DisplayClass5_0.<b__0>d.MoveNext()

[mikem8361]

Yes, you are right to use sudo. It make sense if your service is running under a different user than the one doing the dotnet-dump collect.

How did you install the .NET Core SDK? Did you download the zip or the tar file? Using the zip file, the files don't get set with the correct execute permissions mainly createdump. Can you ls -l createdump program in the shared runtime directory i.e. <installroot>/dotnet/shared/Microsoft.NETCore/3.0.0. And the execute permissions in your self-contained app directory for createdump.

Can you include the dotnet --info output?

[Sean-Driscoll]

Here are the steps I took to install .NET Core and dotnet-dump

$ wget -O dotnet.tar.gz https://download.visualstudio.microsoft.com/download/pr/8ddb8193-f88c-4c4b-82a3-39fcced27e91/b8e0b9bf4cf77dff09ff86cc1a73960b/dotnet-sdk-3.0.100-linux-arm.tar.gz
$ sudo mkdir -p /usr/share/dotnet
$ sudo tar -zxf dotnet.tar.gz -C /usr/share/dotnet
$ sudo ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet
$ dotnet tool install -g dotnet-dump

Requested info below. I had no idea there would be a createdump with the self-contained app. I'll run through testing that as well. Added ls -l results for the dotnet-dump command as well

pi@raspberrypi:~ $ ls -l /usr/share/dotnet/shared/Microsoft.NETCore.App/3.0.0/createdump
-rwxr-xr-x 1 1001 117 95288 Sep 12 19:01 /usr/share/dotnet/shared/Microsoft.NETCore.App/3.0.0/createdump

pi@raspberrypi:~ $ ls -l /opt/myApp/createdump
-rw-r----- 1 myAppSvcAct myAppSvcAct 95288 Nov  8 17:11 /opt/myApp/createdump

pi@raspberrypi:~ $ ls -l /home/pi/.dotnet/tools/dotnet-dump
-rwxr--r-- 1 pi pi 73400 Nov 16 15:08 /home/pi/.dotnet/tools/dotnet-dump

pi@raspberrypi:~ $ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.0.100
 Commit:    04339c3a26

Runtime Environment:
 OS Name:     raspbian
 OS Version:  10
 OS Platform: Linux
 RID:         linux-arm
 Base Path:   /usr/share/dotnet/sdk/3.0.100/

Host (useful for support):
  Version: 3.0.0
  Commit:  7d57652f33

.NET Core SDKs installed:
  3.0.100 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.0.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.0.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Some additional info. I can successfully create a dump with the createdump utility from the shared runtime directory but that led me to another issue, which is why I focused on working exclusively with dotnet-dump to both collect and analyze. After creating the dump file with createdump here is what happens when i try to analyze

pi@raspberrypi:~ $ /home/pi/.dotnet/tools/dotnet-dump analyze testCreateDump
Loading core dump: testCreateDump ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.
> clrthreads
Failed to load data access module, 0x80131c4f
You can run the debugger command 'setclrpath' to control the load of libmscordaccore.so.
If that succeeds, the SOS command should work on retry.
> setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/3.0.0/
Required command was not provided.
Unrecognized command or argument 'setclrpath'
Unrecognized command or argument '/usr/share/dotnet/shared/Microsoft.NETCore.App/3.0.0/'

Same thing happened while analyzing a dump file from the createdump utility that came with the self-contained app (had to make it executable but otherwise no issues creating dump file), regardless of what path I used for setclrpath. It doesn't seem like that command works.

[mikem8361]

pi@raspberrypi:~ $ ls -l /opt/myApp/createdump
-rw-r----- 1 myAppSvcAct myAppSvcAct 95288 Nov  8 17:11 /opt/myApp/createdump

This is why dotnet-dump collect isn't work. createdump doesn't have have the execute permission set (x).

I think the reason dotnet-dump analyze is failing is because the DAC (libmscordaccore.so) isn't in the same directory as libcoreclr.so which should be in your self-contained app directory. It looks you may have copied your app files from the publish directory of the project. Did you copy all of them? Are you running dotnet-dump analyze on the same machine as the dump was generated?

I probably should add the setclrpath command to dotnet-dump because that might have fixed your problem, but it really should find the DAC in the same directory as libcoreclr.so (your self-contained app directory).

[Sean-Driscoll]

Setting execute permissions on createdump in the self-contained folder solved the orphaned process issue. Unfortunately, it still ends up failing with the same reason. Previously it would fail pretty immediately but now it seems to do something for a while before failing. I can see a process spin up running the createdump command with the expected output path, but after it fails with the "Core dump generation FAILED 0x80004005" message the file is not generated.

I believe I copied everything except pdbs. I would need to double-check the azure pipeline I have setup for it. It looks like both those files you mentioned are with the self-contained app, below is a print out. And yes, I'm running dotnet-dump analyze on the same machine.

pi@raspberrypi:~ $ ls -l /opt/myApp/ | grep 'lib\|create'
-rwxr-x--x 1 myAppSvcAct myAppSvcAct 95288 Nov  8 17:11 createdump
-rw-r----- 2 myAppSvcAct myAppSvcAct 2158848 Nov  8 17:11 libclrjit.so
-rw-r----- 2 myAppSvcAct myAppSvcAct 7092616 Nov  8 17:11 libcoreclr.so
-rw-r----- 1 myAppSvcAct myAppSvcAct 604268 Nov  8 17:11 libcoreclrtraceptprovider.so
-rw-r----- 1 myAppSvcAct myAppSvcAct 729416 Nov  8 17:11 libdbgshim.so
-rw-r----- 1 myAppSvcAct myAppSvcAct 310196 Nov  8 17:11 libhostfxr.so
-rw-r----- 1 myAppSvcAct myAppSvcAct 277472 Nov  8 17:11 libhostpolicy.so
-rw-r----- 1 myAppSvcAct myAppSvcAct 2763744 Nov  8 17:11 libmscordaccore.so
-rw-r----- 1 myAppSvcAct myAppSvcAct 2191888 Nov  8 17:11 libmscordbi.so
-rw-r----- 1 myAppSvcAct myAppSvcAct 446536 Nov  8 17:11 libuv.so
-rw-r----- 1 myAppSvcAct myAppSvcAct 56696 Nov  8 17:11 mscorlib.dll

My vote is for adding setclrpath. If for no other reason then because the message printed out right before told me to run the setclrpath which then immediately failed...a little confusing.

[Sean-Driscoll]

I was able to figure out the cause of the second failure by temporarily logging in as the svc account and running the same command. Turned out to be a permission issue with the default output folder. The app has a config folder which the corresponding svc account is owner of. When I changed the output of the dotnet-dump collect to that folder everything worked correctly.

So to recap:

1. dotnet-dump collect on a self-contained app utilizes the createdump utility installed with the app and that utility needs to be executable

   • I had to run dotnet-dump as sudo because the app was run as a different user, which results in the createdump utility being executed as the user account running the app

   • Worth noting: this was causing the orphaned process issue

2. Default output of the dump file is the current users home directory which the account running the app didn't have access to so had to change the output folder

   • This unfortunately wasn't intuitive based on the error message and I was only able to find this issue by logging in as the svc account, but in a majority of cases I doubt users will be able to do that.

[Sean-Driscoll]

Figured out the dotnet-dump analyze issue as well... also a permissions thing :(

I just needed to run it as sudo, probably because my user doesn't have access to the necessary lib files in the self-contained app's directory.

With the above knowledge I was also able to use lldb to load a core dump and use the sos commands. Still haven't been able to use sos commands for attaching to a live process though. With regards to loading the core dump, I'm not sure what is accurate or available but it's a step forward. I had to run the following,

pi@raspberrypi:~ $ sudo lldb --core testCreateDump
(lldb) target create --core "testCreateDump"
Core file '/home/pi/testCreateDump' (arm) was loaded.
(lldb) plugin load /home/pi/.dotnet/sos/libsosplugin.so

[mikem8361]

I will attempt to document this behavior better. For dump collection, we expected that dotnet-dump collect would be the same user as the app being collected.

I will also attempt to display better messages in the 5.0 time frame. This will require changes on the runtime/createdump side. There isn't a lot I can do on the dotnet-dump collect side because all it gets is an HRESULT that is currently pretty much pass/fail.

[mikem8361]

I'm going to close this issue now. I think I have addressed all your concerns. I created issue https://github.com/dotnet/diagnostics/issues/733 about the weak error message.

[raducojocari]

I've been seeing a the exact same problem on MacOs. Adding an env var COMPlus_DbgEnableElfDumpOnMacOS:1 in the target process fixed it