Android Java app crash in MediaPipe PacketGetter.getBool(packet)

[brianm-sra]

I mentioned this in the mediapipe Google Group, and Ming Yong asked me to file it as a Github issue.

Quoting e-mail from the Google Group below.

Let me know if you need more details.

"I modified mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt
to include:
...

input_stream: "input_video_cpu"
output_stream: "output_video"

node: {
    calculator: "ImageFrameToGpuBufferCalculator"
    input_stream: "input_video_cpu"
    output_stream: "input_video"
}

...

and modified the BUILD file for the Android AAR to include the calculator:

load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")

mediapipe_aar(
    name = "mp_hand_tracking_aar",
    calculators = ["//mediapipe/graphs/hand_tracking:mobile_calculators",
        "//mediapipe/gpu:image_frame_to_gpu_buffer_calculator"],
)

and used bazel to rebuild the graph binary and aar, and copied them to my
Android app project's
Application/libs/mp_hand_tracking_aar.aar
Application/src/main/assets/handtrackinggpu.binarypb

When I re-ran my test app (which uses android.hardware.camera2, captures a single frame, stores as JPG,
reads as Bitmap with config ARGB_8888, passes this to MediaPipe methods via

packet = AndroidPacketCreator.createRgbaImageFrame(bitmap);
graph.startRunningGraph();
graph.addPacketToInputStream(INPUT_VIDEO_STREAM_NAME, packet, System.currentTimeMillis());

... then my FrameProcessor's PacketCallback's process method is getting successfully called by the framework
(Packet is not null, and has positive numeric values for native handle and timestamp)

... but then now a native crash in MediaPipe is happening when I try to call

Boolean handPresence = PacketGetter.getBool(packet);

inside of the PacketCallback's process(Packet) method

Any ideas on what could be causing the following crash ?

These are some excerpts from the logcat error I am seeing today:

F libc : Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 31376 (Thread-7), pid 30865 (id.camera2basic)
I crash_dump64: obtaining output fd from tombstoned, type: kDebuggerdTombstone
I crash_dump64: performing dump of process 30865 (target tid = 31376)
F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
...
F DEBUG : backtrace:
F DEBUG : #00 pc 0000000000022364 /system/lib64/libc.so (abort+116)
F DEBUG : #1 pc 000000000089dc00 /data/app/com.example.android.camera2basic-laLagDKtmKyuXLx6ub9BCA==/base.apk (offset 0x67000)
F DEBUG : #2 pc 000000000089d08c /data/app/com.example.android.camera2basic-laLagDKtmKyuXLx6ub9BCA==/base.apk (offset 0x67000) (google::LogMessage::SendToLog()+912)
F DEBUG : #3 pc 000000000089d604 /data/app/com.example.android.camera2basic-laLagDKtmKyuXLx6ub9BCA==/base.apk (offset 0x67000) (google::LogMessage::Flush()+228)
F DEBUG : #4 pc 00000000008a0cf8 /data/app/com.example.android.camera2basic-laLagDKtmKyuXLx6ub9BCA==/base.apk (offset 0x67000) (google::LogMessageFatal::~LogMessageFatal()+16)
F DEBUG : #5 pc 0000000000383100 /data/app/com.example.android.camera2basic-laLagDKtmKyuXLx6ub9BCA==/base.apk (offset 0x67000) (bool const& mediapipe::Packet::Get() const+232)
F DEBUG : #6 pc 000000000037f52c /data/app/com.example.android.camera2basic-laLagDKtmKyuXLx6ub9BCA==/base.apk (offset 0x67000) (Java_com_google_mediapipe_framework_PacketGetter_nativeGetBool+48)
F DEBUG : #7 pc 0000000000560fe0 /system/lib64/libart.so (art_quick_generic_jni_trampoline+144)
F DEBUG : #8 pc 000000000055824c /system/lib64/libart.so (art_quick_invoke_static_stub+604)
F DEBUG : #9 pc 00000000000cfce8 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+232)
F DEBUG : #10 pc 00000000002805cc /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
F DEBUG : #11 pc 000000000027a5d4 /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+968)
F DEBUG : #12 pc 0000000000528554 /system/lib64/libart.so (MterpInvokeStatic+204)
F DEBUG : #13 pc 000000000054a814 /system/lib64/libart.so (ExecuteMterpImpl+14612)
F DEBUG : #14 pc 0000000000277cb8 /dev/ashmem/dalvik-classes.dex extracted in memory from /data/app/com.example.android.camera2basic-laLagDKtmKyuXLx6ub9BCA==/base.apk_30865_30865 (deleted) (com.google.mediapipe.framework.PacketGetter.getBool+8)
F DEBUG : #15 pc 00000000002542d8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.1451459442+488)
F DEBUG : #16 pc 0000000000259dcc /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+216)
F DEBUG : #17 pc 000000000027a5b8 /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+940)
F DEBUG : #18 pc 0000000000528554 /system/lib64/libart.so (MterpInvokeStatic+204)
F DEBUG : #19 pc 000000000054a814 /system/lib64/libart.so (ExecuteMterpImpl+14612)
F DEBUG : #20 pc 0000000000019836 /dev/ashmem/dalvik-classes2.dex extracted in memory from /data/app/com.example.android.camera2basic-laLagDKtmKyuXLx6ub9BCA==/base.apk!classes2.dex_30865_30865 (deleted) (com.example.android.camera2basic.HandTracker.processPacket+146)
..."

[brianm-sra]

I suspect the issue occurs if PacketGetter.getBool(packet) is called from packet callback for
OUTPUT_LANDMARKS_STREAM_NAME

PacketGetter.getBool(packet) works fine and app does not crash if it is only called in the
packet callback for OUTPUT_HAND_PRESENCE_STREAM_NAME

[AlexYiningLiu]

I suspect the issue occurs if PacketGetter.getBool(packet) is called from packet callback for
OUTPUT_LANDMARKS_STREAM_NAME

PacketGetter.getBool(packet) works fine and app does not crash if it is only called in the
packet callback for OUTPUT_HAND_PRESENCE_STREAM_NAME

Sorry this isn't an answer to your problem, but can you please tell me about the process you're using to grab the raw camera frames you mentioned in: "When I re-ran my test app (which uses android.hardware.camera2, captures a single frame, stores as JPG,
reads as Bitmap with config ARGB_8888, passes this to MediaPipe methods via" ?
Did you simply add a packetCallback for the output_video stream? I tried it this way and I believe I am receiving the packets, but whenever I use

Bitmap bitmap_raw_img = AndroidPacketGetter.getBitmapFromRgba(packet);

the app crashes with the error

A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 20047 (Thread-8), pid 19914 (andstrackingapp)

[sohailziahh]

what does this crash even mean? its so gibberish! Is this caused when CPU/GPU utilization is very high?

[sgowroji]

Above crash seen when there are any changes in graph which is not fit. May be you can test the graph in a desktop model before trying to build the AAR with it. Please reopen the issue if you still see the same error. Thanks!

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No