UnsatisfiedLinkError in SentryNdk

[tobi512]

Hi there,
we recently upgraded Sentry from v1.7.23 to v2.0.2 and so far everything looked good. Now we published the first version via our beta channel in Google Play and received a strange crash related to Sentry's new native crash handling feature.

Stacktrace:

java.lang.UnsatisfiedLinkError: 
  at java.lang.Runtime.loadLibrary0 (Runtime.java:1016)
  at java.lang.System.loadLibrary (System.java:1657)
  at io.sentry.android.ndk.SentryNdk.<clinit> (SentryNdk.java:12)
  at java.lang.Class.classForName (Class.java)
  at java.lang.Class.forName (Class.java:453)
  at java.lang.Class.forName (Class.java:378)
  at io.sentry.android.core.NdkIntegration.register (NdkIntegration.java:26)
  at io.sentry.core.Hub.<init> (Hub.java:35)
  at io.sentry.core.Sentry.init (Sentry.java:143)
  at io.sentry.core.Sentry.init (Sentry.java:91)
  at io.sentry.android.core.SentryAndroid.init (SentryAndroid.java:59)
  at io.sentry.android.core.SentryAndroid.init (SentryAndroid.java:44)

Affected device:
Samsung Galaxy J7 Prime (on7xelte), 2816MB RAM, Android 8.1

Right now, we only have one user suffering from the crash according to the Play Console, but from our experience, the number of users will increase as soon as we publish the update to everyone. We've been testing the app on a couple of devices and it seems to run fine so far (however most of the weird crashes happen out in the wild as you probably know).

Is this a known issue and do you have any idea on how to fix it?
Disabling the NDK crash feature by using setEnableNdk(false) might be an option, but the feature is quite nice, so we'd like to keep that enabled.

Regards,
Tobias

[marandaneto]

hey @tobi512 thanks for raising this issue.

actually it's the first UnsatisfiedLinkError report and not sure what this is about.
it's a known problem though, eg https://medium.com/keepsafe-engineering/the-perils-of-loading-native-libraries-on-android-befa49dce2db

We do have a try/catch block when loading the native libs, but it seems to not be enough.
I'll try to investigate a bit deeper on what could cause that.

A few questions to understand a bit more:

Do you have Proguard/R8 enabled?
Do you strip native libs? (separated APK for each CPU architecture)
Do you have multidex enabled?
Do you use App bundles?
Supported CPU architectures of the device? this can be seen in the crash report.

if you decide to disable it, for now, you can save the native bits by doing this:
https://docs.sentry.io/platforms/android/#requirements
so you don't even bundle the native libs.

Please answer the questions above and I'll try to figure this out, thanks.

[tobi512]

Hi @marandaneto,
thanks for looking into this! I'll provide the answers to your questions below:

Do you have Proguard/R8 enabled? Yes, we use R8
Do you strip native libs? (separated APK for each CPU architecture) Not explicitly, but probably implicitly by using App Bundles?
Do you have multidex enabled? We use minSdkVersion 23, so it's enabled by default (https://developer.android.com/studio/build/multidex#mdex-on-l "Therefore, if your minSdkVersion is 21 or higher multidex is enabled by default, and you do not need the multidex support library.")
Do you use App bundles? Yes, for publishing in the Play Store
Supported CPU architectures of the device? ABIs are armeabi-v7a and armeabi according to the Play Console

Regards,
Tobias

[marandaneto]

hey @tobi512 thanks for your patience.
I could not reproduce it at all, I've tried a few devices and emulators.
Would it help if I bring #344 into 2.0.3? that would not make App's to crash at least till we get the ability to reproduce this error.
Also, have you seen more issues like that one? Thanks.

[tobi512]

Hi @marandaneto,
thanks for you reply.

Bringing #344 into 2.0.3 would definitely help in my opinion since it's always a good idea to prevent crashes. We will beging rolling out the version to production in the near future, till now we only see 21 crashes from exactly one user/device (so it seems like the app is unusable there). I'll let you know how the crashes develop when we have a reasonable amount of users.

The only thing left for now would be to get a Samsung Galaxy J7 Prime somewhere and test on exactly that device (already checked on AWS device farm, but they only have a "normal" Galaxy J7).

[marandaneto]

@tobi512 yep, I've tested it using a Galaxy J7, but could not reproduce it, I also could not find a J7 Prime image.
fwiw #344 is already in our alpha version, in case you'd like to run tests on that too, but since you also don't have a J7 Prime, I believe we are both stuck on the same problem.
Please do let us know how it develops, if we see more cases on different devices/users, we'll prioritize this, thanks again.
cc @Swatinem

[tobi512]

Hi @marandaneto,
quick update from my side:

As I was expecting the number of crashes increased after taking the update live and we now see 7 new users affected with the following devices:

Samsung Galaxy A3(2017) (a3y17lte), 2048MB RAM, Android 8.0
Samsung Galaxy S7 (herolte), 3584MB RAM, Android 8.0
Samsung Galaxy J5(2016) (j5xnlte), 2048MB RAM, Android 7.1
Samsung Galaxy S8 (dreamlte), 3840MB RAM, Android 9
Redmi Redmi Note 8 Pro (begonia), 5632MB RAM, Android 9
ZTE BLADE A510 (P635A50), 1024MB RAM, Android 6.0
Huawei P8 (HWGRA), 2816MB RAM, Android 6.0

[marandaneto]

hey @tobi512 thanks and sorry for this :(

we do have a J5, we are gonna test it and let you know asap, stay tuned.

[marandaneto]

@tobi512 would you mind updating with its Android API? some of them may have upgraded, thanks.

[tobi512]

@marandaneto Sure, I'll add the Android versions to the devices

[marandaneto]

@tobi512 thanks, I've released a patch
https://github.com/getsentry/sentry-android/releases/tag/2.0.3

It's not going to fix it as I still don't know the problem, but it's not gonna crash it anymore.
I'll keep trying to reproduce it and let you know, thanks and sorry about that.

[marandaneto]

cc @Swatinem ideas? halp

[Swatinem]

If I remember correctly, I had something like this when playing around with the simulator shell.

You can always check with ldd which external libraries your executable/library depends on.
If one of those can’t be found, you can override LD_LIBRARY_PATH and add the directory that your libraries are coming from.

I think you can pull the library you want to load, and run ldd locally. Though you should only use LD_LIBRARY_PATH during development to debug your problem, not really in production.

Can you try that and give feedback?

[marandaneto]

@Swatinem but this is a runtime error, not a compile error, not sure if I follow it.

[Swatinem]

yes, ldd shows you the libraries that the dynamic linker has to resolve at runtime.

[Swatinem]

I spent quite some time researching this topic.
So there was a bug in order Android versions which forced you to manually load all transitive dependencies before the main library, as explained here: https://android.googlesource.com/platform/bionic/+/master/android-changes-for-ndk-developers.md#changes-to-library-dependency-resolution

The page is quite informative when it comes to all these linking things.

If you can reproduce this, maybe try enabling logging for the dynamic loader:
https://android.googlesource.com/platform/bionic/+/master/android-changes-for-ndk-developers.md#enable-logging-of-dlopen_dlsym-and-library-loading-errors-for-apps-available-in-android-o

Another question: does your app try to load native libs directly from the apk, as explained here: https://android.googlesource.com/platform/bionic/+/master/android-changes-for-ndk-developers.md#opening-shared-libraries-directly-from-an-apk ? As the guid says, libraries loaded this way need to be page-aligned and uncompressed.

Just for reference, here are some logs from my testing within the simulator:

2020-04-22 16:48:25.523 5376-5376/io.sentry.sample D/linker: dlopen(name="/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so", flags=0x2, extinfo=[flags=0x200, reserved_addr=0x7ffcdd5c1b7c, reserved_size=0x73bff26f86ba, relro_fd=-173918981, library_fd=173378327, library_fd_offset=0x73c077ae2fc0, library_namespace=classloader-namespace@0x73c077ce0650], caller="/apex/com.android.runtime/lib64/libnativeloader.so", caller_ns=runtime@0x73c077ce0010, targetSdkVersion=29) ...
2020-04-22 16:48:25.524 5376-5376/io.sentry.sample D/linker: find_libraries(ns=classloader-namespace): task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so, is_dt_needed=0
2020-04-22 16:48:25.524 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so, flags=0x2, realpath=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so)
2020-04-22 16:48:25.525 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so): Adding DT_NEEDED task: libdl.so
2020-04-22 16:48:25.525 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so): Adding DT_NEEDED task: liblog.so
2020-04-22 16:48:25.525 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so): Adding DT_NEEDED task: libm.so
2020-04-22 16:48:25.525 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so): Adding DT_NEEDED task: libc.so
2020-04-22 16:48:25.525 5376-5376/io.sentry.sample D/linker: find_libraries(ns=classloader-namespace): task=libdl.so, is_dt_needed=1
2020-04-22 16:48:25.525 5376-5376/io.sentry.sample D/linker: find_library_internal(ns=classloader-namespace, task=libdl.so): Already loaded (by soname): /apex/com.android.runtime/lib64/bionic/libdl.so
[…]
2020-04-22 16:48:25.528 5376-5376/io.sentry.sample D/linker: ... dlopen calling constructors: realpath="/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so", soname="libsentry.so", handle=0x58d74f8083ec3ffb
2020-04-22 16:48:25.528 5376-5376/io.sentry.sample D/linker: ... dlopen successful: realpath="/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so", soname="libsentry.so", handle=0x58d74f8083ec3ffb
[…]
2020-04-22 16:48:25.529 5376-5376/io.sentry.sample D/linker: dlopen(name="/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry-android.so", flags=0x2, extinfo=[flags=0x200, reserved_addr=0x7ffcdd5c1b7c, reserved_size=0x73bff26f86ba, relro_fd=-173918981, library_fd=173378327, library_fd_offset=0x73c077ae2fc0, library_namespace=classloader-namespace@0x73c077ce0650], caller="/apex/com.android.runtime/lib64/libnativeloader.so", caller_ns=runtime@0x73c077ce0010, targetSdkVersion=29) ...
2020-04-22 16:48:25.530 5376-5376/io.sentry.sample D/linker: find_libraries(ns=classloader-namespace): task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry-android.so, is_dt_needed=0
2020-04-22 16:48:25.530 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry-android.so, flags=0x2, realpath=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry-android.so)
2020-04-22 16:48:25.531 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry-android.so): Adding DT_NEEDED task: libsentry.so
2020-04-22 16:48:25.531 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry-android.so): Adding DT_NEEDED task: libm.so
2020-04-22 16:48:25.532 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry-android.so): Adding DT_NEEDED task: libdl.so
2020-04-22 16:48:25.532 5376-5376/io.sentry.sample D/linker: load_library(ns=classloader-namespace, task=/data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry-android.so): Adding DT_NEEDED task: libc.so
2020-04-22 16:48:25.532 5376-5376/io.sentry.sample D/linker: find_libraries(ns=classloader-namespace): task=libsentry.so, is_dt_needed=1
2020-04-22 16:48:25.532 5376-5376/io.sentry.sample D/linker: find_library_internal(ns=classloader-namespace, task=libsentry.so): Already loaded (by soname): /data/app/io.sentry.sample-K2Ib0Jj6sdFofprGmFW_Dg==/lib/x86_64/libsentry.so

[marandaneto]

Another question: does your app try to load native libs directly from the apk, as explained here: https://android.googlesource.com/platform/bionic/+/master/android-changes-for-ndk-developers.md#opening-shared-libraries-directly-from-an-apk ? As the guid says, libraries loaded this way need to be page-aligned and uncompressed.

cc @tobi512

[tobi512]

Hi guys,
sorry for the delay and thanks @marandaneto for cc'ing.

A quick update first: we don't see any new crashes after publishing the update with your fix, so that seems fine now.

Regarding your cited question: in our own code we don't directly use any System.loadLibrary() calls. However, I think that some of our dependencies might do that (e.g. we use React Native which does a lot of stuff with native libs as far as I know) so not sure if we can change anything on that end. Could you clarify again what exactly this means for us?

[Swatinem]

A quick update first: we don't see any new crashes after publishing the update with your fix, so that seems fine now.

ah nice! Well then call this fixed?

[marandaneto]

@tobi512 being react-native would change the target of our tests, we didn't know that, but thanks for all the info, we are gonna monitor it.

@Swatinem this is because of https://github.com/getsentry/sentry-android/releases/tag/2.0.3
not sure if UnsatisfiedLinkError is still happening or not, we'll know that the day after today, when sentry-android 2.1 goes GA and people start updating it, let's close it and if so, people open it again, thank you all :)