Backend selection for static lib

[vaind]

I'm trying to integrate sentry-native crash handling to sentry-dotnet SDK when compiled to NativeAOT. I'm integrating as a static library to avoid having the end consumer application to need to bundle an additional library. Bundling and compilation all work fine without any backend and transport (originally, I've just used sentry-native to get the list of loaded debug images to enable server-side symbolication; that all worked fine).

However, I'm trying to figure out which backend to use for desktop platforms (Windows, Linux, macOS) in this scenario, specifically for a static library. Since crashpad is an external process (i.e. an external binary which I've tried to avoid with the static lib), I was thinking breakpad or inproc. The thing is, I could not find any documentation what are the main differences, functionally, between the various backends, aside from platform support. It would be nice to have a doc that I could consult. I've already had the same issue when implementing sentry-native to the Unity and Unreal SDKs and because All these use-cases/platforms are a bit different, I'm having trouble making the decision each time, other than "pick the default" (then new questions arise, like: is the default the default because it's the best one or because it has been the default years ago and is just kept that way to not break users, etc)

[supervacuus]

The thing is, I could not find any documentation what are the main differences, functionally, between the various backends, aside from platform support. It would be nice to have a doc that I could consult.

You are right. We don't have a document to help users decide on a specific backend. After 1.5 years of being exposed to questions surrounding the backend, I can short-cut it to always using Crashpad, except when you shouldn't.

The reasons for "shouldn't" by order of occurrence are:

1. can't distribute/deploy an external handler artifact/process (for which there are uncountable, but often artificial reasons)
2. want to use a crash handler hook on macOS
3. clients running on a version of the OS that Crashpad doesn't support

If you touch one of those, you should always default to breakpad first. It immediately eliminates 1.) and 2.) above while still providing decent coverage of platforms, and it, like crashpad, sends minidumps, which provide more context than what the inproc backend offers.

You'll need to use inproc only if you have tested breakpad and crashpad and are 100% sure they don't fit your use case.

I've already had the same issue when implementing sentry-native to the Unity and Unreal SDKs and because All these use-cases/platforms are a bit different, I'm having trouble making the decision each time, other than "pick the default" (then new questions arise, like: is the default the default because it's the best one or because it has been the default years ago and is just kept that way to not break users, etc)

Case in point: the Unity and Unreal SDKs nowadays use practically all Native backends because the decision space is ample and not only related to platform support (which is rarely a reason to choose one over the other). The choice most often boils down to particular differences in the deployment or distribution scenario, which are more related to the environment in which it is used rather than the backend itself.

What kinds of information would you base your choice on?

[vaind]

which provide more context than what the inproc backend offers.

OK, but what exactly does that mean? I'm really curious because if the others are more complicated to integrate for me, why should I use them instead of inproc?

What kinds of information would you base your choice on?

Hmm, a few things come to mind:

• what are the functional differences between the backends, e.g.
  • are there crash types that are not captured by one or the other?
  • when do they send errors? (at the time of the error, during next startup)
  • are there callbacks that could be registered for the error handler? When are these called?
• what are the supported modes of integration?
  • static library
  • separate executable
  • separate shared library for the backend
• Maybe for others: what 3rd party libraries does it uses? This can be relevant for SBoM, although it's not really something I'm basing the decision upon.

Basically, it's all about weight pros and cons. As is, i.e. with the information that is available, I don't see why I would ever choose anything by inproc, which is easiest to integrate, it doesn't add an additional library/dependency.

[supervacuus]

which provide more context than what the inproc backend offers.

OK, but what exactly does that mean? I'm really curious because if the others are more complicated to integrate for me, why should I use them instead of inproc?

because you can offer your users more useful data when crashes happen:

• the stack trace of each thread running at the time of the crash (this is for most users the most significant feature missing in inproc)
• more supported error types
  • especially for crashpad on Windows you'll get abort-handling, fast-fail crashes, handling of heap corruptions, ...
  • crashpad on Linux has support for weirder ELF configurations that lead to non-contiguously mapped executables
  • for crashpad and breakpad support for mach exceptions on macOS instead of using a posix signal-handler + the buggy backtrace() implementation of Apple on top
• having the crashpad_handler as a separate process allows us to run the snapshot and upload outside of the crashing process which makes it the only backend that will send crashes without having to restart the application
• more consistent data-quality across platforms (there is much more parity over what kind of data you'll get vs inproc where this can't have been a goal)

The list could be longer, but there is also way more nuance than what this short list hints at, because some higher level platforms use standard paths (for crashing, exiting applications, etc.) that wouldn't even be exercised in other environments. Case in point: abort() will not end in SEH on Windows, so currently only the crashpad backend will be reporting this as a crash.

What kinds of information would you base your choice on?

• what are the functional differences between the backends, e.g.
  • are there crash types that are not captured by one or the other?
  • when do they send errors? (at the time of the error, during next startup)

i covered these above

• are there callbacks that could be registered for the error handler? When are these called?

yes, all backends provide a before_send hook and an on_crash hook see https://docs.sentry.io/platforms/native/configuration/filtering/ for details. The docs also talk about the limitations across platforms and backends.

• what are the supported modes of integration?
  • static library

whether you build a static library or not is independent of the backend you choose

• separate executable

only crashpad requires a separate executable

• separate shared library for the backend

we only build a single shared library that contains the Native SDK and the backend (without exposing the backend in the library symbols)

• Maybe for others: what 3rd party libraries does it uses? This can be relevant for SBoM, although it's not really something I'm basing the decision upon.

the only 3rd party dependencies are libcurl and zlib (of course for each of the platforms there are additional system dependencies, but nothing a user would have to maintain in any of the supported OS versions).

Basically, it's all about weight pros and cons. As is, i.e. with the information that is available, I don't see why I would ever choose anything by inproc, which is easiest to integrate, it doesn't add an additional library/dependency.

I do understand what the basic idea is. My point is that what users actually need is extremely specific to their use-case most of the time and much broader and deeper than we are currently able to cover.

I also wonder whether it would be helpful to most, because it requires a lot of detail knowledge with regard to the error handling mechanisms on each platform (from the interactions i had with users I am almost sure it would overwhelm most). So some limitation in scope is required for this to be helpful (so users don't just end up with "well, this is complicated... let's go with crashpad"). But maybe I am wrong here and open discussion.

[vaind]

Thank you. This is exactly the information that I believe should be documented, be it on docs.sentry.io for sentry-native or at least here in the repository. And yes, there will always be questions about this and that, but unless those are extremely specific, they could be added to the doc. As is, it is all just guesswork for integrators. I'm not saying the doc should cover every nuance there is, we could just try to add the basic info, i.e. what has been listed in this issue so far, and start there.

Also, not everyone has to read it, that's what the default settings are for, but if someone is interested, the information should be available rather than having to ask maintainers IMO. That also helps future contributors to catch up.