Coredump during freeSampleStreamingSession while running Viewer

[disa6302]

While running the WebRTC Viewer sample, the application coredumps after about 45-50 minutes everytime. The coredump seems to be caused while freeing Streaming Session. The logs are attached.

viewer-coredump.txt

Stack Trace on Mac:

(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff7afbd2c2 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff7afba866 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff7afba46e libsystem_kernel.dylib`__recvfrom + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff7afbc786 libsystem_kernel.dylib`__recvmsg + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff7afbc786 libsystem_kernel.dylib`__recvmsg + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff7afbaf2e libsystem_kernel.dylib`__semwait_signal + 10, stop reason = signal SIGSTOP
(lldb) thread backtrace
* thread #1, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff7afbd2c2 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff7b078bf1 libsystem_pthread.dylib`pthread_kill + 284
    frame #2: 0x00007fff7af276a6 libsystem_c.dylib`abort + 127
    frame #3: 0x00007fff7b036077 libsystem_malloc.dylib`malloc_vreport + 545
    frame #4: 0x00007fff7b035e38 libsystem_malloc.dylib`malloc_report + 151
    frame #5: 0x0000000107665f15 libkvsWebrtcClient.dylib`defaultMemFree + 21
    frame #6: 0x000000010762fe98 kvsWebrtcClientViewer`freeSampleStreamingSession + 328
    frame #7: 0x00000001076309a2 kvsWebrtcClientViewer`freeSampleConfiguration + 146
    frame #8: 0x0000000107631640 kvsWebrtcClientViewer`main + 1936
    frame #9: 0x00007fff7ae823d5 libdyld.dylib`start + 1

**Stack Trace on Ubuntu:**

[KVS Viewer] Cleaning up....
2020-03-26 01:21:16 INFO    freeSignalingClient(): Freeing Signaling Client
[Thread 0x7fffeffff700 (LWP 9755) exited]
2020-03-26 01:21:16 DEBUG   freeSampleStreamingSession(): Freeing streaming session with peer id: ConsumerViewer
[Thread 0x7fffeeffd700 (LWP 9781) exited]
[Thread 0x7fffef7fe700 (LWP 9758) exited]
[Thread 0x7fffee7fc700 (LWP 9760) exited]
2020-03-26 01:21:18 DEBUG   freeSampleStreamingSession(): Freeing streaming session with peer id: ConsumerViewer
double free or corruption (!prev)

Thread 1 "kvsWebrtcClient" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff7398801 in __GI_abort () at abort.c:79
#2  0x00007ffff73e1897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff750eb9a "%s\n")
    at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007ffff73e890a in malloc_printerr (str=str@entry=0x7ffff7510890 "double free or corruption (!prev)") at malloc.c:5350
#4  0x00007ffff73efe84 in _int_free (have_lock=0, p=0x5555557a51f0, av=0x7ffff7743c40 <main_arena>) at malloc.c:4281
#5  __GI___libc_free (mem=0x5555557a5200) at malloc.c:3124
#6  0x00007ffff7bc130c in defaultMemFree ()
   from amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so
#7  0x0000555555557fad in freeSampleStreamingSession ()
#8  0x0000555555558acd in freeSampleConfiguration ()
#9  0x00005555555595f1 in main ()

[MushMal]

@disa6302 Has the viewer app been terminated by the client? If so, how? Ctrl+C? I see the initiation of the termination in the logs but I am not sure what caused it to terminate

[disa6302]

The viewer app is not being terminated by the client. The viewer just ends up in termination after 45 - 50 minutes. That is why I am unable to understand what is causing the termination. And this is 100% reproducible as well on my end.

[MushMal]

Seems that there are two issues,

1. For some reason we are getting an error that causes the sample to terminate.
2. There is a teardown crash that's happening.

[disa6302]

I just found out STATUS_ICE_FAILED_TO_RECOVER_FROM_DISCONNECTION error in the logs (error code: 0x5a000025). Is this non recoverable?

Besides this, I do not see any other error code.

[MushMal]

Hmm, I don't see any errors in the attached log. The STATUS_ICE_FAILED_TO_RECOVER_FROM_DISCONNECTION should not cause a fatal error I believe

[MushMal]

OK. the crash is fixed and I also added some logging at the end. Now, we can see what's causing the application to exit. One of the main issues with the samples is that the media thread emulator thread can terminate/exit. Let's see what it is

[disa6302]

Closing this since the core dump issue is fixed.