BAD_ACCESS while running createSignalingClientSync()

[unnikrishnan-anil]

I'm on the master, commit f98abc1 from 6 days ago and I am using the static version of the library in my project. While running

createSignalingClientSync(&_sampleConfiguration->clientInfo, &_sampleConfiguration->channelInfo,
                                      &_sampleConfiguration->signalingClientCallbacks, _sampleConfiguration->pCredentialProvider,
                                      &_sampleConfiguration->signalingClientHandle);

I get the following BAD ACCESS stack trace:

• thread 1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  frame #0: 0x00007fff68ecb6bb libsystem_platform.dylib_platform_strnlen + 43 frame 1: 0x00000001004193e5 kinesis_test_cppcreateValidateChannelInfo + 117
  frame 2: 0x000000010041e80b kinesis_test_cppcreateSignalingSync + 331 frame 3: 0x0000000100419d02 kinesis_test_cppcreateSignalingClientSync + 322
  • frame 4: 0x0000000100008447 kinesis_test_cppstreamer::kinesisInit(this=0x0000000100cbfde0) at streamer.cpp:81:17 frame 5: 0x00000001000081bb kinesis_test_cppstreamer::init(this=0x0000000100cbfde0, options=0x00007ffeefbff4a8) at streamer.cpp:32:16
    frame 6: 0x0000000100001f09 kinesis_test_cppmain at streamer_test.cpp:24:8 frame 7: 0x00007fff68cd4cc9 libdyld.dylibstart + 1

Sometimes it also gives me:

• thread 1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x50)
  frame 0: 0x00007fff68ecb6bb libsystem_platform.dylib_platform_strnlen + 43 frame 1: 0x0000000100418b85 kinesis_test_cppcreateValidateChannelInfo + 117
  frame 2: 0x000000010041dfab kinesis_test_cppcreateSignalingSync + 331 frame 3: 0x00000001004194a2 kinesis_test_cppcreateSignalingClientSync + 322
  • frame 4: 0x0000000100007dc7

Same method -> createValidateChannelInfo() but different BAD_ACCESS error.

[MushMal]

@CatsNipYummy can you provide a little more information..

• Are you able to build and run the sample master application?
• Can you provide the sample of the code that sets the channel info, callbacks, credential provider and client info structures that are being passed into the create function?
• Are you observing the crash with a dynamic linkage?

The crash stack points to an invalid string access with STRNLEN - this simply means that some of the structure strings might not be null terminated.

[unnikrishnan-anil]

@MushMal

• I am able to run the sample master application without an issue. The issue is coming when I built a static version of the library inside my codebase.

• As far as the sample code goes, this is what it looks like:

STATUS retStatus = _kinesisCommon->createKinesisConfiguration(KINESIS_CHANNEL_NAME,
                                                                  SIGNALING_CHANNEL_ROLE_TYPE_MASTER,
                                                                  TRUE,
                                                                  TRUE,
                                                                  &_pSampleConfiguration);
    if(retStatus != STATUS_SUCCESS) {
        Log::error("[KVS Master] createKinesisConfiguration(): operation returned status code: 0x%08x \n", retStatus);
        return false;
    }
    
    Log::message("[KVS Master] Created signaling channel %s\n", KINESIS_CHANNEL_NAME);
    
    // Set the audio and video handlers
    _pSampleConfiguration->audioSource = nullptr;//sendAudioPackets;
    _pSampleConfiguration->videoSource =  nullptr;//sendVideoPackets;
    _pSampleConfiguration->receiveAudioVideoSource =  kinesisReceiveAudioFrame;
    _pSampleConfiguration->onDataChannel = onDataChannel;
    _pSampleConfiguration->mediaType = KINESIS_STREAMING_AUDIO_VIDEO;
    
    Log::message("[KVS Master] Finished setting audio and video handlers\n");
    
    // Initialize KVS WebRTC. This must be done before anything else, and must only be done once.
    retStatus = initKvsWebRtc();
    if(retStatus != STATUS_SUCCESS) {
        Log::error("[KVS Master] initKvsWebRtc(): operation returned status code: 0x%08x \n", retStatus);
        kinesisCleanup();
        return false;
    }
    
    Log::message("[KVS Master] KVS WebRTC initialization completed successfully\n");
    
    _pSampleConfiguration->signalingClientCallbacks.messageReceivedFn = Common::masterMessageReceived;
    
    strcpy(_pSampleConfiguration->clientInfo.clientId, KINESIS_MASTER_CLIENT_ID);
    
    retStatus = createSignalingClientSync(&_pSampleConfiguration->clientInfo, &_pSampleConfiguration->channelInfo,
                                          &_pSampleConfiguration->signalingClientCallbacks, _pSampleConfiguration->pCredentialProvider,
                                          &_pSampleConfiguration->signalingClientHandle);

Here's the channel name I'm using
#define KINESIS_CHANNEL_NAME (PCHAR) "TSKinesisSignaling" similar to the sample code.

I went into createValidateChannelInfo() inside ChannelInfo.c and I see the pOrigChannelInfo->pChannelName is coming through alright. Is is normal to have pChannelArn as nil at that point?

createKinesisConfiguration() returns success. Currently I'm passing my AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY directly inside createKinesisConfiguration().

The only difference I have is that I currently don't have callbacks for audioSource and videoSource. I'm using the example from that logic inside my codebase. I'm wondering if that can cause this crash. I have an update loop that provides me with H264 frames that I pass directly to the writeFrame() method. But the crash happens even before all this happens.

[MushMal]

Looks correct to me.

I would recommend running this under a debugger and get a debug version of the crash stack with local variables so we can analyze.

ARN can be NULL. It is passed in the scenarios where you have a pre-created channel and would need to skip describe state. In this case, the describe channel call will fill the internal copy of the structure.

Access Key/Secret Key as well as other parameters can be discarded after the call to create channel sync API as they will be copied/packed inside the internal structures.

Not sure what you mean about the audio and video sources - you can simply feed in the frame in the transceiver.

[unnikrishnan-anil]

Here's my entire call stack at this point:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x30)
    frame #0: 0x00007fff68ecb6bb libsystem_platform.dylib`_platform_strnlen + 43
    frame #1: 0x0000000100418b85 hugo_ostreamer-ostreamer_run_cpp`createValidateChannelInfo + 117
    frame #2: 0x000000010041dfab hugo_ostreamer-ostreamer_run_cpp`createSignalingSync + 331
    frame #3: 0x00000001004194a2 hugo_ostreamer-ostreamer_run_cpp`createSignalingClientSync + 322
  * frame #4: 0x0000000100007dc7 hugo_ostreamer-ostreamer_run_cpp`hugo::ostreamer::Ostreamer::kinesisInit(this=0x0000000100cbfd80) at ostreamer.cpp:81:17
    frame #5: 0x0000000100007b3b hugo_ostreamer-ostreamer_run_cpp`hugo::ostreamer::Ostreamer::init(this=0x0000000100cbfd80, options=0x00007ffeefbff4f8) at ostreamer.cpp:32:16
    frame #6: 0x0000000100001889 hugo_ostreamer-ostreamer_run_cpp`main at ostreamer_run.cpp:24:8
    frame #7: 0x00007fff68cd4cc9 libdyld.dylib`start + 1
    frame #8: 0x00007fff68cd4cc9 libdyld.dylib`start + 1
  thread #2
    frame #0: 0x00007fff68e18882 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x0000000100b11f81 libsystem_pthread.dylib`_pthread_cond_wait + 698
    frame #2: 0x00007fff65fa8592 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
    frame #3: 0x000000010007f248 hugo_ostreamer-ostreamer_run_cpp`hugo::core::SafeQueue<hugo::core::Module::Quatuor>::dequeue(this=0x0000000100cbfda0) at safequeue.h:50:19
    frame #4: 0x000000010007d446 hugo_ostreamer-ostreamer_run_cpp`hugo::core::Module::innerUpdate(this=0x0000000100cbfd80) at module.cpp:351:25
    frame #5: 0x000000010007d30a hugo_ostreamer-ostreamer_run_cpp`hugo::core::Module::InnerModule::process(this=0x0000000100cc00a0) at module.cpp:43:22
    frame #6: 0x0000000100086b6e hugo_ostreamer-ostreamer_run_cpp`hugo::core::Thread::thread_handler(data=0x0000000100cc00a0) at thread.cpp:69:17
    frame #7: 0x0000000100b11c65 libsystem_pthread.dylib`_pthread_start + 148
    frame #8: 0x0000000100b0d4af libsystem_pthread.dylib`thread_start + 15
  thread #3, name = 'SCTP iterator'
    frame #0: 0x00007fff68e18882 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x0000000100b11f81 libsystem_pthread.dylib`_pthread_cond_wait + 698
    frame #2: 0x000000010038e0db hugo_ostreamer-ostreamer_run_cpp`sctp_iterator_thread + 75
    frame #3: 0x0000000100b11c65 libsystem_pthread.dylib`_pthread_start + 148
    frame #4: 0x0000000100b0d4af libsystem_pthread.dylib`thread_start + 15
  thread #4, name = 'SCTP addr mon'
    frame #0: 0x00007fff68e193a6 libsystem_kernel.dylib`__recvfrom + 10
    frame #1: 0x00000001003eed93 hugo_ostreamer-ostreamer_run_cpp`recv_function_route + 115
    frame #2: 0x0000000100b11c65 libsystem_pthread.dylib`_pthread_start + 148
    frame #3: 0x0000000100b0d4af libsystem_pthread.dylib`thread_start + 15
  thread #5, name = 'SCTP/UDP/IP4 rcv'
    frame #0: 0x00007fff68e19682 libsystem_kernel.dylib`__recvmsg + 10
    frame #1: 0x00000001003ef693 hugo_ostreamer-ostreamer_run_cpp`recv_function_udp + 339
    frame #2: 0x0000000100b11c65 libsystem_pthread.dylib`_pthread_start + 148
    frame #3: 0x0000000100b0d4af libsystem_pthread.dylib`thread_start + 15
  thread #6, name = 'SCTP/UDP/IP6 rcv'
    frame #0: 0x00007fff68e19682 libsystem_kernel.dylib`__recvmsg + 10
    frame #1: 0x00000001003f0210 hugo_ostreamer-ostreamer_run_cpp`recv_function_udp6 + 400
    frame #2: 0x0000000100b11c65 libsystem_pthread.dylib`_pthread_start + 148
    frame #3: 0x0000000100b0d4af libsystem_pthread.dylib`thread_start + 15
  thread #7, name = 'SCTP timer'
    frame #0: 0x00007fff68e18756 libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff68d9beea libsystem_c.dylib`nanosleep + 196
    frame #2: 0x000000010038e653 hugo_ostreamer-ostreamer_run_cpp`user_sctp_timer_iterate + 83
    frame #3: 0x0000000100b11c65 libsystem_pthread.dylib`_pthread_start + 148
    frame #4: 0x0000000100b0d4af libsystem_pthread.dylib`thread_start + 15

Channel Info:

(ChannelInfo) channelInfo = {
version = 1
pChannelName = 0x00000001004b9af6 "TSKinesisSignaling"
pChannelArn = 0x0000000000000031 ""
pRegion = 0x00000001004b93c2 "us-west-2"
pControlPlaneUrl = 0x0000000100cbff80 "/Users/anil/Code/Repo/bitbucket/data/cert.pem"
pCertPath = 0x0000000100cc0270 "/Users/anil/Code/Repo/bitbucket/data/cert.pem"
pUserAgentPostfix = 0x0000000000000040 ""
pCustomUserAgent = 0x0000000100cc0050 ""
pUserAgent = 0x6766632e676e6964 ""
pKmsKeyId = 0x0000000000000000
channelType = SIGNALING_CHANNEL_TYPE_SINGLE_MASTER
channelRoleType = SIGNALING_CHANNEL_ROLE_TYPE_MASTER
reserved = 47
cachingPeriod = 0
retry = YES
reconnect = YES
messageTtl = 0
tagCount = 0
pTags = 0x0000000000000000
cachingPolicy = SIGNALING_API_CALL_CACHE_TYPE_DESCRIBE_GETENDPOINT
asyncIceServerConfig = YES
}

[unnikrishnan-anil]

I dont know if this line is causing the issue:
strcpy(_pOstreamerConfiguration->clientInfo.clientId, KINESIS_MASTER_CLIENT_ID);
The output doesn't look right to me:

Printing description of this->_pOstreamerConfiguration->clientInfo: (SignalingClientInfo) clientInfo { version 0 clientId {'P''r''o''d''u''c''e''r''M''a' 's' 't' 'e' 'r' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\x02' '\0' '\0' '\0' '\0' '\0' '\0' '\0' 'P' '*' '\x1a' '\0' '\x01' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' 'p' '*' '\x1a' '\0' '\x01' '\0' '\0' '\0' '\xd0' '\x01' '\xcc' '\0' '\x01' '\0' '\0' '\0' '\xa0' '\x01' '\xcc' '\0' '\x01' '\0' '\0' '\0' '\x17' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\x02' '\0' '1' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '-' '\0' '\0' '\0' '\0' '\0' '\0' '\0' 'p' '\x02' '\xcc' '\0' '\x01' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '/' 'U' 's' 'e' 'r' 's' '/' 'a' 'n' 'i' 'l' '/' 'C' 'o' 'd' 'e' '/' 'R' 'e' 'p' 'o' '/' 'b' 'i' 't' 'b' 'u' 'c' 'k' 'e' 't' '/' 'd' 'a' 't' 'a' '/' 'c' 'e' 'r' 't' '.' 'p' 'e' 'm' '\0' '\0' '\0' '\0' '\0' '\0' '\0' '~' '\0' '\0' '\0' '\xe0' '\x02' '\xcc' '\0' ... } loggingLevel 20 }

Note - edited for readability. Why is there a path to the cert?

[MushMal]

So far, all looks good to me. The last piece you've indeed looks corrupted to me. These should all be zeroed out by the calloc so you should have seen 256 bytes of 0 as the field size is 256. And, the logging level is weird. The crash is a NULL deref but I can't tell from the stack what it is. Could you run the same with a debug build or with the symbols to pinpoint the line that's crashing or messing things up?

[unnikrishnan-anil]

I'll have to make some changes to get a debug build. But pretty sure that's the issue. Each run gives me a lot of scrambled values after 'ProducerMaster'. A simple printf() looks okay. But there's more stuff after the \0. I'll get a debug build up and post more findings.

[unnikrishnan-anil]

@MushMal
Debug build errors:

CHK((arnLen = (UINT32) STRNLEN(pOrigChannelInfo->pChannelArn, MAX_ARN_LEN + 1)) <= MAX_ARN_LEN, STATUS_SIGNALING_INVALID_CHANNEL_ARN_LENGTH);

Error:

Printing description of pOrigChannelInfo->pChannelArn:
(PCHAR) pChannelArn = 0x0000000000000031 ""
Printing description of *(pOrigChannelInfo->pChannelArn):
(CHAR) *pChannelArn = <read memory from 0x31 failed (0 of 1 bytes read)>

Another time I got it here.

CHK((cplLen = (UINT32) STRNLEN(pOrigChannelInfo->pControlPlaneUrl, MAX_URI_CHAR_LEN + 1)) <= MAX_URI_CHAR_LEN,
            STATUS_SIGNALING_INVALID_CPL_LENGTH);
po pOrigChannelInfo->pControlPlaneUrl;

Error:
Printing description of *(pOrigChannelInfo->pControlPlaneUrl):
(CHAR) *pControlPlaneUrl = <read memory from 0x616e626d75685474 failed (0 of 1 bytes read)>

There is some memory being mangled. Both from source/signaling/ChannelInfo.c

[MushMal]

OK, seems that your channel info is not being properly zeroed or the public header file is different.

In the samples, pChannelArn should be NULL. If it's random pointer then random memory address will be dereferenced which will cause a access violation.

Please make sure you compile with the same header files as the library is build.

Please zero the fields first before setting them. In the sample, we allocate the channel info as part of the auxiliary structure to keep track of other things which is calloc-ed which zeroes all the allocation.

[unnikrishnan-anil]

I had a bug in malloc code that was causing weird behavior. Closing issue.