[API Proposal]: Overriding the default behavior in case of unhandled exceptions and fatal errors.

[VSadov]

Re: The previous proposal and a discussion that led to this proposal - (#42275)

Background and motivation

The current default behavior in a case of unhandled exception is termination of a process.
The current default behavior in a case of a fatal error is print exception to console and invoke Watson/CrashDump.

While satisfactory to the majority of uses, the scheme is not flexible enough for some classes of scenarios.

Scenarios like Designers, REPLs or game scripting that host user provided code are not able to handle unhandled exceptions thrown by the user provided code. Unhandled exceptions on finalizer thread, threadpool threads or user created threads will take down the whole process. This is not desirable experience for these types of scenarios.

In addition, there are customers that have existing infrastructure for postmortem analysis of failures and inclusion of .NET components requires interfacing with or overriding the way the fatal errors are handled.

API Proposal

API for process-wide handling of unhandled exception

namespace System.Runtime.ExceptionServices
{
    public delegate bool UnhandledExceptionHandler(System.Exception exception);

    public static class ExceptionHandling
    {
        /// <summary>
        /// Sets a handler for unhandled exceptions.
        /// </summary>
        /// <exception cref="ArgumentNullException">If handler is null</exception>
        /// <exception cref="InvalidOperationException">If a handler is already set</exception>
        public static void SetUnhandledExceptionHandler(UnhandledExceptionHandler handler);
    }
}

The semantics of unhandled exception handler follows the model of imaginary handler like the following inserted in places where the exception will not lead to process termination regardless of what handler() returns.

try { UserCode(); } catch (Exception ex) when handler(ex){};

In particular:

• only exceptions that can be caught and ignored will cause the handler to be invoked. (i.e. stack overflow will not)
• an unhandled exception thrown in a handler will not invoke the handler, but will be treated as returning false.
• when an exception is handled via a handler in a user-started thread, the thread will still exit (but not escalate to process termination)
• when an exception is handled in a task-like scenario on an infrastructure thread (thread pool, finalizer queue...), the execution of tasks will continue.
  (Whether the infrastructure thread continues or restarted is unspecified, but the process should be able to proceed)
• a reverse pinvoke will not install the try/catch like above.
• main() will not install the try/catch like above

API Proposal for custom handling of fatal errors

Managed API to set up the handler.

namespace System.Runtime.ExceptionServices
{
    public static class ExceptionHandling
    {
        /// <summary>
        /// .NET runtime is going to call `fatalErrorHandler` set by this method before its own
        /// fatal error handling (creating .NET runtime-specific crash dump, etc.).
        /// </summary>
        /// <exception cref="ArgumentNullException">If fatalErrorHandler is null</exception>
        /// <exception cref="InvalidOperationException">If a handler is already set</exception>
        public static void SetFatalErrorHandler(delegate* unmanaged<int, void*, int> fatalErrorHandler);
    }
}

The shape of the FatalErrorHandler, if implemented in c++
(the default calling convention for the given platform is used)

// expected signature of the handler
FatalErrorHandlerResult FatalErrorHandler(int32_t hresult, struct FatalErrorInfo* data);

With FatalErrorHandlerResult and FatalErrorInfo defined in "FatalErrorHandling.h" under src/native/public:

enum FatalErrorHandlerResult : int32_t
{
    RunDefaultHandler = 0,
    SkipDefaultHandler = 1,
};

#if defined(_MSC_VER) && defined(_M_IX86)
#define DOTNET_CALLCONV __stdcall
#else
#define DOTNET_CALLCONV
#endif

struct FatalErrorInfo
{
    size_t size;    // size of the FatalErrorInfo instance
    void*  address; // code location correlated with the failure (i.e. location where FailFast was called)

    // exception/signal information, if available
    void* info;     // Cast to PEXCEPTION_RECORD on Windows or siginfo_t* on non-Windows.
    void* context;  // Cast to PCONTEXT on Windows or ucontext_t* on non-Windows.

    // An entry point for logging additional information about the crash.
    // As runtime finds information suitable for logging, it will invoke pfnLogAction and pass the information in logString.
    // The callback may be called multiple times.
    // Combined, the logString will contain the same parts as in the console output of the default crash handler.
    // The errorLog string will have UTF-8 encoding.
    void (DOTNET_CALLCONV *pfnGetFatalErrorLog)(
           FatalErrorInfo* errorData, 
           void (DOTNET_CALLCONV *pfnLogAction)(char8_t* logString, void *userContext), 
           void* userContext);

    // More information can be exposed for querying in the future by adding
    // entry points with similar pattern as in pfnGetFatalErrorLog
};

API Usage

Setting up a handler for unhandled exceptions:

using System.Runtime.ExceptionServices;

ExceptionHandling.SetUnhandledExceptionHandler(
    (ex) =>
    {
        if (DesignMode)
        {
            DisplayException(ex);
            // the exception is now "handled"
            return true;
        }
        return false;
    }
);

Setting up a handler for fatal errors:

Setting up the handler for the process (C# code in the actual app):

internal unsafe class Program
{
    [DllImport("myCustomCrashHandler.dll")]
    public static extern delegate* unmanaged<int, void*, int> GetFatalErrorHandler();

    static void Main(string[] args)
    {
        ExceptionHandling.SetFatalErrorHandler(GetFatalErrorHandler());

        RunMyProgram();
    }
}

The handler. (c++ in myCustomCrashHandler.dll)

#include "FatalErrorHandling.h"

static FatalErrorHandlerResult FatalErrorHandler(int32_t hresult, struct FatalErrorInfo* data)
{
    // this is a special handler that analyzes OOM crashes
    if (hresult != COR_E_OUTOFMEMORY)
    {
        return FatalErrorHandlerResult::RunDefaultHandler;
    }

    DoSomeCustomProcessingOfOOM(data);

    // retain the additional error data
    data->pfnGetFatalErrorLog(data, &LogErrorMessage, NULL);

    // no need for a huge crash dump after an OOM.
    return FatalErrorHandlerResult::SkipDefaultHandler;
}

static void LogErrorMessage(char8_t* logString, void *userContext)
{
    AppendToBlob(logString);
}

extern "C" DLL_EXPORT void* GetFatalErrorHandler()
{
    return &FatalErrorHandler;
}

Alternative Designs

Unmanaged hosting API that enables this behavior. (CoreCLR has undocumented and poorly tested configuration option for this today. #39587. This option is going to be replaced by this API.)

Extending AppDomain.CurrentDomain.UnhandledException API and make IsTerminating property writeable to allow "handling".
Upon scanning the existing use of this API it was found that IsTerminating is often used as a fact - whether an exception is terminal or not. Changing the behavior to mean "configurable" will be a breaking change to those uses.

Risks

This APIs can be abused to ignore unhandled exceptions or fatal errors in scenarios where it is not warranted.

[AaronRobinsonMSFT]

FatalErrorHandlerResult FatalErrorHandler(int32_t hresult, struct FatalErrorInfo* data);

Defines int32_t, but the C# function pointer in SetFatalErrorHandler() defined uint. I assume both should be signed.

[janvorli]

I wonder if having the messages etc. as 16 bit strings is the right way. Our coreclr_xxx hosting APIs use 8 bit (UTF-8) strings so that users on Unix don't have to convert them using some external library. 16 bit strings are very unusual on Unix and on Windows, it is trivial to do the conversion if needed thanks to the Windows APIs.

[VSadov]

We had to choose between char_t and char16_t. See: #42275 (comment)

The key observations were -

• unlike hosting, where strings originate from the native host and thus in the platform-specific form, these originate from the runtime and they are in UTF-16 form.
• the process might be in somewhat precarious state, so we'd like to minimize the work that is done "inline".
  For example the strings could be uninteresting to the given handler anyways, or could be stored as-is and converted later.

[janvorli]

In case the native handler on Unix would want to do anything sensible with these strings, even just writing them to a text log file, I think it would most likely need to convert them. In the runtime, we can easily convert them into stack allocated buffers without disturbing the process state. I still think it would be better if whatever string enters or leaves the runtime from the native side was 8 bit. Similar to the hosting APIs, it would add a little overhead for the handler on Windows, but much less than the overhead that would otherwise be needed on Unix.
I am not sure I understand the comment in the other issue on a need to have some PAL definitions if we went with 8 bit strings.

[AaronRobinsonMSFT]

I am not sure I understand the comment in the other issue on a need to have some PAL definitions if we went with 8 bit strings.

The approach in question was to have a char_t that represented wchar_t on Windows and char on non-Windows. If we wanted to make it always UTF-8, I could get behind that. That seems rather annoying, but I do get it. My desire is to avoid any sort of PAL types that users would need to think about. Casting doesn't concern me, but encoding types in signatures or data structures would be a red flag to me.

[VSadov]

How complex is UTF-16 to UTF-8 conversion? Is there IO or allocations?

[AaronRobinsonMSFT]

How complex is UTF-16 to UTF-8 conversion? Is there IO or allocations?

There will be allocations for sure. The IO is nil or very limited if done from managed code if I recall. The native side trans-coding is also new and I believe avoids IO. It isn't the worst idea if it is just for logging. The tricky bit is if the callback comes back into managed though - that is my biggest concern.

[jkotas]

How complex is UTF-16 to UTF-8 conversion? Is there IO or allocations?

It can be ~100 - ~1000 lines depending on how optimized implementation you want. The problem is that there is no broadly available standardized method to do this conversion on non-Windows platforms, so everybody using this API on non-Windows would have to copy the conversion routine from somewhere.

As a (usability) test for this API, you can write a sample handler that produces .json that captures the failure details (similar to .json that is produced by native AOT crash handler today). You will hit the UTF16 conversion problem pretty much immediately.

The tricky bit is if the callback comes back into managed though - that is my biggest concern.

Calling managed implementation is not an option for the crash handlers.

[VSadov]

There will be allocations for sure. The IO is nil or very limited if done from managed code if I recall. The native side trans-coding is also new and I believe avoids IO. It isn't the worst idea if it is just for logging. The tricky bit is if the callback comes back into managed though - that is my biggest concern.

If a string comes from native assert (like from GC), running managed code might not be an option.
I assumed the conversion can be done completely in native code.

[AaronRobinsonMSFT]

The problem is that there is no broadly available standardized method to do this conversion on non-Windows platforms

Right, they will likely need to use ICU or some other library.

The tricky bit is if the callback comes back into managed though - that is my biggest concern.

Calling managed implementation is not an option for the crash handlers.

Excellent

You will hit the UTF16 conversion problem pretty much immediately.

Fair. This implies UTF-8 should be the preference then.

[VSadov]

It can be ~100 - ~1000 lines depending on how optimized implementation you want. The problem is that there is no broadly available standardized method to do this conversion on non-Windows platforms, so everybody using this API on non-Windows would have to copy the conversion routine from somewhere.

That is not too bad.
Since this is a crash handling and strings are not expected to be huge, I'd optimize for reliability. (avoiding IO, allocations, syscalls, if possible - the process might have crashed, but who knows what locks are still held and such...)

[jkotas]

Right, they will likely need to use ICU or some other library.

ICU is big, complicated library. It is not something you want to depend on in a crash handler. I think that the most appropriate solution would be to include source for the simplest possible conversion from somewhere.

[vladimir-cheverdyuk-altium]

Is it possible to get content of char16_t* exceptionString; // for unhandled managed exceptions the result of "ex.ToString()" by request? I believe that when FatalErrorHandler is called stack is still available.

I suggest to do it on demand because it will create issues for COR_E_OUTOFMEMORY and also we had an issue that @jkotas helped us with COM when calling Exception.ToString considerably slowdown future COM calls. See #98788.

[VSadov]

If conversion to UTF-8 is going to be needed in majority of scenarios (especially on Unix), I think we should do UTF-8 rather than expect that the end user will convert.

I'll update the proposal to have char8_t.

[VSadov]

Is it possible to get content of char16_t* exceptionString; // for unhandled managed exceptions the result of "ex.ToString()" by request? I believe that when FatalErrorHandler is called stack is still available.

ex.ToString() typically happens at the time when runtime finds that the exception is unhandled. By the time when the native crash handler runs, managed code would not be able to run. To have any chance to report that string it should be captured and stored earlier, while that is still possible. I do not think the string can be produced on request while handling a crash.

On the other hand, a fatal error can happen only once in the life of an app, so perhaps performance of this API will not be as critical as in scenarios that can repeat.

[jkotas]

@vladimir-cheverdyuk-altium has a good point about having an option to suppress computation of the textual stacktrace or doing other non-trivial operations if the crash handler is not interested in the information. Such non-trivial operations can hit secondary crashes or otherwise interfere with the diagnosability of the original unhandled exception.

I had similar concern in the original discussion.

[am11]

they will likely need to use ICU or some other library.

Runtime's implementation (which is pretty much standalone) could also be an option for users.

e.g. this test code

#include <stdio.h>
#include <errno.h>
#include <minipal/utf8.h>

size_t wstrlen(CHAR16_T* str)
{
    size_t len = 0;
    while (str[len++]);
    return len;
}

void handleErrors()
{
    if (errno == MINIPAL_ERROR_INSUFFICIENT_BUFFER)
    {
        printf ("Allocation failed (%d)", errno);
        abort();
    }
    else if (errno == MINIPAL_ERROR_NO_UNICODE_TRANSLATION)
    {
        printf ("Illegal byte sequence encountered in the input. (%d)", errno);
        abort();
    }
}

int main(void)
{
    CHAR16_T wstr[] = u"ハローワールド! 👋 🌏";
    int wlen = wstrlen(wstr);

    size_t mblen = minipal_get_length_utf16_to_utf8(wstr, wlen, 0);
    handleErrors();

    char* mbstr = (char *)malloc((mblen + 1) * sizeof(char));
    size_t written = minipal_convert_utf16_to_utf8 (wstr, wlen, mbstr, mblen, 0);
    handleErrors();

    printf("Conversion completed. mblen: %zu, mbstr: %s\n", written, mbstr);

    return 0;
}

can be built with sources acquired from tag link:

$ curl --create-dirs --output-dir external/minipal -sSLO \
    "https://raw.githubusercontent.com/dotnet/runtime/v8.0.4/src/native/minipal/{utf8.c,utf8.h,utils.h}"
$ clang -Iexternal test.c external/minipal/utf8.c -o testutf8
$ ./testutf8
Conversion completed. mblen: 33, mbstr: ハローワールド! 👋 🌏

[VSadov]

@vladimir-cheverdyuk-altium has a good point about having an option to suppress computation of the textual stacktrace or doing other non-trivial operations if the crash handler is not interested in the information. Such non-trivial operations can hit secondary crashes or otherwise interfere with the diagnosability of the original unhandled exception.

At the time of reporting the crash it would be too late to decide whether FailFast should have captured the exception message for the unhandled exception or not.

We could add a bool parameter to SetFatalErrorHandler that would have effect of suppressing that ex.ToString process-wide.
The result of that would be that the default crash processing will not have the string either - i.e. will not print it to console.

The benefits of that would be slightly faster handling of the crash and more robust behavior in case ex.ToString causes secondary errors.

Cases like OOM or stack overflow are always tricky. Also ex.ToString can do anything - like throw this;.
In most cases unhandled exceptions are just that - unhandled exceptions (null dereferences, file did not exist, ..) and getting exception message would be safe and useful.

Basically - we can add a knob to SetFatalErrorHandler to suppress ex.ToString when installing the handler, but I wonder is there will be a lot of users for that.

Is it really that often that ex.ToString can do really bad things (like stack overflow) and simply placing it inside try/catch will not make the scenario safe enough, that you'd rather prefer to not see stack traces?

[vladimir-cheverdyuk-altium]

@VSadov

At the time of reporting the crash it would be too late to decide whether FailFast should have captured the exception message for the unhandled exception or not.
I'm not sure about FailFast but for regular exceptions I think exception stack should be available unless I'm missing something.

As it was stated in #98788, ex.ToString will make COM calls slow if it was in the call stack and it is why I'm concerned.

[VSadov]

Perhaps I do not understand the scenario...

When a managed exception is unhandled(*), the runtime will call Environment_FailFast and will pass the exception as one of the parameters.

runtime/src/coreclr/classlibnative/bcltype/system.cpp

Line 168 in 6cd329b

extern "C" void QCALLTYPE Environment_FailFast(QCall::StackCrawlMarkHandle mark, PCWSTR message, QCall::ObjectHandleOnStack exception, PCWSTR errorSource)

Environment_FailFast will make a call to exception.ToString(). (via GetExceptionMessage). At that point we still can run managed code, but not for long.
A few lines later Environment_FailFast will call EEPolicy::HandleFatalError, which will proceed to things like capturing a dump and exiting the process.

I understand the reliability concern about exception.ToString() doing something horrible, but not sure where the performance concern comes from. What will observe slowness in COM calls after Environment_FailFast?

(*) Unhandled here means - "really unhandled". If there was UnhandledExceptionHandler and handled it, the exception is not unhandled after that and will not FailFast/crash.

[jkotas]

I am not worried about performance. I am worried about providing the right information to the crash handler, without introducing unnecessary failure points between the failure point and the call of the crash handler.

On one end, you can have crash handlers that just want to capture crashdump for offline processing and do nothing else. Computing the textual stacktrace is source of unnecessary failure points for those.

On the other end, you can have crash handlers that want to capture many details like stacktrace with file and line numbers if possible.

The design should accommodate the whole spectrum. (We do not need to implement everything in the first round, but we should have an idea for how we would go about covering the whole spectrum.)

[vladimir-cheverdyuk-altium]

@VSadov
My idea was to pass exception handle (it can be exception object itself) to handler and later handler can call function to get more information from this handle.

What will observe slowness in COM calls after Environment_FailFast?

COM calls slows down after ex.ToString called if some module is on the call stack. Some code that walks stack converts module to read/write mode from read-only mode and as result search became linear instead of binary.

[AaronRobinsonMSFT]

I am ready to have a broader conversation. @jkotas?

[jkotas]

Same here.

[rolfbjarne]

public static void SetUnhandledExceptionHandler(UnhandledExceptionHandler handler);

The naming here seems confusing to me, this is named very similarly to the AppDomain.UnhandledException event, and in particular it doesn't seem clear that this won't be called for all unhandled exceptions, only unhandled exceptions that can actually be ignored.

It's not a big difference, but what about SetUnhandledExceptionFilter? That matches the imaginary code:

try { UserCode(); } catch (Exception ex) when handler(ex){};

in that handler is an exception filter.

IMHO it's also somewhat more obvious that this won't be called for all unhandled exceptions, only those that can actually be filtered/ignored.

A few other ideas, I don't like any of them, but maybe somebody else are inspired to come up with better names:

• SetFilterableUnhandledExceptionFilter
• SetCatchUnhandledExceptionHandler

Additionally, it would be nice to be able to invoke the callback.

Example use case for .NET for iOS: we have an API called InvokeOnMainThread (Action action), where developers can do this:

InvokeOnMainThread (() => { ShowMessageBox ("Hello World!"); });

which would currently, if ShowMessageBox threw an exception, end up bringing the process down. If we could implement this internally like this:

public void InvokeOnMainThread (Action action)
{
    NativeInvokeOnMainThread (() => {
        // This is effectively called from a reverse P/Invoke
        try {
            action ():
        } catch (Exception ex) when ExceptionHandling.FilterUnhandledException (ex) {
            // ignored
        }
    });
}

that would be nice.

Lastly:

/// <exception cref="ArgumentNullException">If handler is null</exception>
/// <exception cref="InvalidOperationException">If a handler is already set</exception>
public static void SetUnhandledExceptionHandler(UnhandledExceptionHandler handler);

There's no way to clear/change/reset the handler once it's set? Is that intentional?

[lambdageek]

Does this proposal address crashes in unmanaged code? The main motivation in #79706 is the ability for a user app (which is commonly a combination of managed code and native libraries) to install a third-party crash reporter behind the .NET runtime so that all fatal app terminations are captured and logged. This proposal seems to only deal with unhandled managed exceptions - or will SetFatalErrorHandler also install something for those cases? The proposal is fairly vague about what kinds of fatal errors will trigger the handler

[jkotas]

The naming here seems confusing to me, this is named very similarly to the AppDomain.UnhandledException event, and in particular it doesn't seem clear that this won't be called for all unhandled exceptions

Good point. I agree that the name variants like you have suggested can be used to make the behavior more clear.

Another option is to call it for all unhandled exceptions and change the filter to take an extra bool ignorable argument that will say whether the exception can be ignored.

Additionally, it would be nice to be able to invoke the callback.

It sound reasonable to me.

There's no way to clear/change/reset the handler once it's set? Is that intentional?

Yes, it is intentional. These callbacks are meant to be used at the applications level. The application is expected to be have one and only unhandled exception and fatal error handling policy.

Does this proposal address crashes in unmanaged code?

SetFatalErrorHandler is expected to address crashes in unmanaged code. Thinking about this more, there are two cases: fatal crashes in the .NET code (including unmanaged runtime code) and crashes in other code.

We clearly want this to intercept handle crashes in .NET code, but handling crashes in other code is less clear cut. It makes sense if the app wants to own the process, but it does not make sense if the .NET code is a library (e.g. think native AOT library) and the main app has its own crash handler. Are we going to limit this to handling crashes in .NET code or should we have a flag that says whether the handler should intercept all crashes - both in .NET code and other code?

[colejohnson66]

Why a dedicated delegate type and not Func<Exception, bool>?

[bartonjs]

Video

• Instead of creating a new delegate type, use Func<Exception, bool>
• We renamed fatalErrorHandler to handler for consistency.

namespace System.Runtime.ExceptionServices
{
    public static class ExceptionHandling
    {
        /// <summary>
        /// Sets a handler for unhandled exceptions.
        /// </summary>
        /// <exception cref="ArgumentNullException">If handler is null</exception>
        /// <exception cref="InvalidOperationException">If a handler is already set</exception>
        public static void SetUnhandledExceptionHandler(Func<Exception, bool> handler);

        /// <summary>
        /// .NET runtime is going to call `fatalErrorHandler` set by this method before its own
        /// fatal error handling (creating .NET runtime-specific crash dump, etc.).
        /// </summary>
        /// <exception cref="ArgumentNullException">If fatalErrorHandler is null</exception>
        /// <exception cref="InvalidOperationException">If a handler is already set</exception>
        public static unsafe void SetFatalErrorHandler(delegate* unmanaged<int, void*, int> handler);
    }
}

[rgroenewoudt]

Will this also be implemented for .NET Android? On mobile platforms logging and debugging of crashes is very complex, especially because retrieving device logs means complicated instructions to a customer.

[jkotas]

Will this also be implemented for .NET Android?

Yes, it is the intention to implement these APIs on all platforms. (On some platforms, the fatal error handler may have limitations imposed by the platform.)

[bruno-garcia]

Will this also be implemented for .NET Android? On mobile platforms logging and debugging of crashes is very complex, especially because retrieving device logs means complicated instructions to a customer.

You don't really need to deal with this manually though.

Sentry on Android will capture crashes caused by unhandled exceptions in C# as well as from Java/Kotlin or native (NDK, C/C++/etc). The terminology used in this issue is fatal error.
Similar on iOS, C# exceptions as well as Swift/Obj-C fatal errors, crashes, etc.

It attaches logs automatically, optionally attaches screenshots too.
If you upload debug files during build all the crash events in Sentry will contain line numbers (even in release builds of course). It supports Portable PDBs, ProGuard and native symbols (ELF, Mach-O, and more). It pulls symbols from nuget.org automatically so it'll include line numbers of .NET's runtime and OSS libraries too. If source_link is available, it pulls the surrounding lines of code on the frame so it's easier to understand what's going on in the code (see screenshot below).

Supports Native AOT also outside of mobile. With line numbers on stack traces from C# as well as native code.

some screenshots of Symbol Collector

I do some dogfooding of it on Symbol Collector: https://github.com/getsentry/symbol-collector/

Disclaimer: I worked on building the Sentry SDK for .NET