Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crashes on too large messages #24

Closed
xiaojun789 opened this issue Nov 8, 2013 · 17 comments
Closed

crashes on too large messages #24

xiaojun789 opened this issue Nov 8, 2013 · 17 comments
Assignees
@edenhill

Comments

@xiaojun789
Copy link

I want to use librdkafa in my project, but when I do test, it cause my app crash.
my situation is, in main thread create neccessary object, and multithread will call rd_kafka_produce to send the message to kafka server.

when the application runnig sometimes, it will crashed. the call stack like following:

Core was generated by `nelo2-thrift                                          '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000005ec47d in rd_kafka_broker_produce_toppar (rkb=0x7f8124000ef0) at rdkafka_broker.c:1944
1944            rkbuf->rkbuf_ts_timeout =
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6_3.3.x86_64 libgcc-4.4.6-4.el6.x86_64 libgcrypt-1.4.5-9.el6_2.2.x86_64 libgpg-error-1.7-4.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0  0x00000000005ec47d in rd_kafka_broker_produce_toppar (rkb=0x7f8124000ef0) at rdkafka_broker.c:1944
#1  rd_kafka_broker_producer_serve (rkb=0x7f8124000ef0) at rdkafka_broker.c:2105
#2  0x00000000005ed1d8 in rd_kafka_broker_thread_main (arg=0x7f8124000ef0) at rdkafka_broker.c:3002
#3  0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003e86ae76dd in clone () from /lib64/libc.so.6
@edenhill
Copy link
Contributor

edenhill commented Nov 8, 2013

Hi,

could you provide the full output from gdb:

thread apply all bt

You can trim out any parts of your program you dont want to expose, but please try to retain librdkafka call chains.

Thanks

@xiaojun789
Copy link
Author

HI,
that is all the call stack I get from gdb, following is full gdb output message. FYI
Thanks

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from ***...done.
[New Thread 29626]
[New Thread 29629]
[New Thread 29642]
[New Thread 29628]
[New Thread 29634]
[New Thread 29638]
[New Thread 29618]
[New Thread 29641]
[New Thread 29623]
[New Thread 29625]
[New Thread 29605]
[New Thread 29627]
[New Thread 29615]
[New Thread 29590]
[New Thread 29602]
[New Thread 29610]
[New Thread 29606]
[New Thread 29614]
[New Thread 29603]
[New Thread 29589]
[New Thread 29604]
[New Thread 29633]
[New Thread 29612]
[New Thread 29616]
[New Thread 29632]
[New Thread 29631]
[New Thread 29639]
[New Thread 29621]
[New Thread 29601]
[New Thread 29593]
[New Thread 29613]
[New Thread 29600]
[New Thread 29588]
[New Thread 29620]
[New Thread 29617]
[New Thread 29619]
[New Thread 29635]
[New Thread 29637]
[New Thread 29611]
[New Thread 29608]
[New Thread 29640]
[New Thread 29636]
[New Thread 29599]
[New Thread 29595]
[New Thread 29598]
[New Thread 29630]
[New Thread 29622]
[New Thread 29594]
[New Thread 29591]
[New Thread 29624]
[New Thread 29597]
[New Thread 29609]
[New Thread 29607]
[New Thread 29596]
[New Thread 29592]
Missing separate debuginfo for /data3/nelo2/building/collector/bin/../../lib/libboost_program_options.so.1.49.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/ea/798ac78e0ee12ee921d440b2eeb93d8bdf146a
Missing separate debuginfo for /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c4/d27a30896887cd369e21792dbfd46218d02de6
Missing separate debuginfo for /data3/nelo2/building/collector/bin/../../lib/libboost_filesystem.so.1.49.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c7/48f0a99e08c86aa5e93f72473b47548784540f
Missing separate debuginfo for /data3/nelo2/building/collector/bin/../../lib/libboost_system.so.1.49.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/09/2169a954a5c65eb4cc371f083548c7494c9b87
Missing separate debuginfo for /data3/nelo2/building/collector/bin/../../lib/libboost_date_time.so.1.49.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/12/c132ed1b7a26393ad74c521fbd842ea37bb064
Missing separate debuginfo for /data3/nelo2/building/collector/bin/../../lib/libboost_regex.so.1.49.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/86/0e09170b1f9010b10a75fce0d700825281ded0
Missing separate debuginfo for /data3/nelo2/building/collector/bin/../../lib/libboost_log_setup.so.1.49.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/17/b2a94efaa90e03901555b639780ed318fad0a0
Missing separate debuginfo for /data3/nelo2/building/collector/bin/../../lib/libboost_log.so.1.49.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/05/2da0205ffc2883ba57f23e742b092f3dc2372d
Missing separate debuginfo for /home1/irteam/apps/jdk/jre/lib/amd64/server/libjvm.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/25/38297eb170c7e2d8832308ed002e32c3befadb
Missing separate debuginfo for 
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/74/22987f6cfa699f465b0327139aac3fdac4a838
Reading symbols from /data3/nelo2/building/collector/bin/../../lib/libboost_program_options.so.1.49.0...(no debugging symbols found)...done.
Loaded symbols for /data3/nelo2/building/collector/bin/../../lib/libboost_program_options.so.1.49.0
Reading symbols from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0...(no debugging symbols found)...done.
Loaded symbols for /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
Reading symbols from /data3/nelo2/building/collector/bin/../../lib/libboost_filesystem.so.1.49.0...(no debugging symbols found)...done.
Loaded symbols for /data3/nelo2/building/collector/bin/../../lib/libboost_filesystem.so.1.49.0
Reading symbols from /data3/nelo2/building/collector/bin/../../lib/libboost_system.so.1.49.0...(no debugging symbols found)...done.
Loaded symbols for /data3/nelo2/building/collector/bin/../../lib/libboost_system.so.1.49.0
Reading symbols from /data3/nelo2/building/collector/bin/../../lib/libboost_date_time.so.1.49.0...(no debugging symbols found)...done.
Loaded symbols for /data3/nelo2/building/collector/bin/../../lib/libboost_date_time.so.1.49.0
Reading symbols from /data3/nelo2/building/collector/bin/../../lib/libboost_regex.so.1.49.0...(no debugging symbols found)...done.
Loaded symbols for /data3/nelo2/building/collector/bin/../../lib/libboost_regex.so.1.49.0
Reading symbols from /data3/nelo2/building/collector/bin/../../lib/libboost_log_setup.so.1.49.0...(no debugging symbols found)...done.
Loaded symbols for /data3/nelo2/building/collector/bin/../../lib/libboost_log_setup.so.1.49.0
Reading symbols from /data3/nelo2/building/collector/bin/../../lib/libboost_log.so.1.49.0...(no debugging symbols found)...done.
Loaded symbols for /data3/nelo2/building/collector/bin/../../lib/libboost_log.so.1.49.0
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /home1/irteam/apps/nlog/lib/mysql/libmysqlclient.so.18...done.
Loaded symbols for /home1/irteam/apps/nlog/lib/mysql/libmysqlclient.so.18
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /home1/irteam/apps/nlog/lib/cubrid/libcascci.so.8...done.
Loaded symbols for /home1/irteam/apps/nlog/lib/cubrid/libcascci.so.8
Reading symbols from /data3/nelo2/building/collector/bin/../../lib/libhdfs.so.0...(no debugging symbols found)...done.
Loaded symbols for /data3/nelo2/building/collector/bin/../../lib/libhdfs.so.0
Reading symbols from /home1/irteam/apps/jdk/jre/lib/amd64/server/libjvm.so...Missing separate debuginfo for /home1/irteam/apps/jdk/jre/lib/amd64/server/libjvm.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/25/38297eb170c7e2d8832308ed002e32c3befadb.debug
(no debugging symbols found)...done.
Loaded symbols for /home1/irteam/apps/jdk/jre/lib/amd64/server/libjvm.so
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcrypt.so.11
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgpg-error.so.0
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `nelo2-thrift                                          '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000005ec47d in rd_kafka_broker_produce_toppar (rkb=0x7f81280013f0) at rdkafka_broker.c:1944
1944            rkbuf->rkbuf_ts_timeout =
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6_3.3.x86_64 libgcc-4.4.6-4.el6.x86_64 libgcrypt-1.4.5-9.el6_2.2.x86_64 libgpg-error-1.7-4.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) bt
#0  0x00000000005ec47d in rd_kafka_broker_produce_toppar (rkb=0x7f81280013f0) at rdkafka_broker.c:1944
#1  rd_kafka_broker_producer_serve (rkb=0x7f81280013f0) at rdkafka_broker.c:2105
#2  0x00000000005ed1d8 in rd_kafka_broker_thread_main (arg=0x7f81280013f0) at rdkafka_broker.c:3002
#3  0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003e86ae76dd in clone () from /lib64/libc.so.6
(gdb)

@edenhill
Copy link
Contributor

edenhill commented Nov 8, 2013

Could you write this in gdb after crash, instead of 'bt':

thread apply all bt full

@ghost ghost assigned edenhill Nov 8, 2013
@xiaojun789
Copy link
Author

···
Thread 55 (Thread 0x7f81635fe700 (LWP 29592)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0xa97990, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c0015b0) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c0015b0) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c0015b0) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 54 (Thread 0x7f8160dfa700 (LWP 29596)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0xa97990, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c003030) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c003030) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c003030) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 53 (Thread 0x7f81317fb700 (LWP 29607)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c007ad0) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c007ad0) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c007ad0) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 52 (Thread 0x7f810ffff700 (LWP 29609)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c008770) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c008770) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c008770) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x7f8147fff700 (LWP 29597)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0xa97990, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c003700) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c003700) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c003700) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x7f80da1fc700 (LWP 29624)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00ed80) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00ed80) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00ed80) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7f8163fff700 (LWP 29591)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0xa97990, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c000c70) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c000c70) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c000c70) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7f81475fe700 (LWP 29598)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0xa97990, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c003dd0) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c003dd0) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c003dd0) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7f81617fb700 (LWP 29595)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0xa97990, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c002960) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c002960) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c002960) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7f8146bfd700 (LWP 29599)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0xa97990, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c0044a0) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c0044a0) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c0044a0) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7f80b61fc700 (LWP 29636)):
#0 0x0000003e86a82854 in memchr () from /lib64/libc.so.6
#1 0x0000003512c9c759 in std::basic_string<char, std::char_traits, std::allocator >::find_first_not_of(char const*, unsigned long, unsigned long) const () from /usr/lib64/libstdc++.so.6
#2 0x00000000004cc2e5 in find_first_not_of (strKey="Message") at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/bits/basic_string.h:1882
#3 NeloEvent::checkCustomKey (strKey="Message") at src/common/core/neloEventCheck.cpp:17
#4 0x00000000004cc6da in checkCustomKeys (ptrEvt=...) at src/common/core/neloEventCheck.cpp:71
#5 NeloEvent::verifyNeloEvent (ptrEvt=...) at src/common/core/neloEventCheck.cpp:117
---Type to continue, or q to quit---
#6 0x00000000004e0973 in NeloSource::pushToQueue (this=0x7f8164000bf0, ptrEvt=...) at src/common/core/neloSource.cpp:179
#7 0x0000000000535ec0 in ThriftNeloEventServerHandler::_append (this=0x7f815c011090, evt=...) at src/collector/sources/thrift/thriftEventSource.cpp:69
#8 0x000000000053630b in ThriftNeloEventServerHandler::ackedAppend (this=0x7f815c011090, evt=...) at src/collector/sources/thrift/thriftEventSource.cpp:28
#9 0x00000000004319fb in nelo::thrift::ThriftNeloEventServerProcessor::process_ackedAppend (this=0x7f815c0110d0, seqid=0, iprot=, oprot=0x7f8080002840, callContext=)
at src/third-party/thrift/ThriftNeloEventServer.cpp:811
#10 0x0000000000430594 in nelo::thrift::ThriftNeloEventServerProcessor::process_fn (this=, iprot=0x7f80800027f0, oprot=0x7f8080002840, fname="ackedAppend", seqid=0, callContext=)
at src/third-party/thrift/ThriftNeloEventServer.cpp:710
#11 0x00000000004307f3 in nelo::thrift::ThriftNeloEventServerProcessor::process (this=0x7f815c0110d0, piprot=, poprot=, callContext=0x7f80b61f9038)
at src/third-party/thrift/ThriftNeloEventServer.cpp:692
#12 0x000000000053a099 in ThriftHandler::doIt (this=0x7f815c011310, pBuffer=, uiLength=, respBuff=..., cliAddr=) at ./src/common/service/thrift/thriftHandler.h:187
#13 0x000000000053cefc in FrameSessionboost::shared_ptr::_readRequestBody (this=0x7f8074002df0) at ./src/common/service/thrift/frameSession.h:273
#14 0x000000000053d9ad in FrameSessionboost::shared_ptr::_socketRecv (this=0x7f8074002df0, fd=73) at ./src/common/service/thrift/frameSession.h:176
#15 0x000000000053dfb5 in FrameSessionboost::shared_ptr::doEvent (this=0x7f8074002df0, fd=, which=) at ./src/common/service/thrift/frameSession.h:108
#16 0x00000000004fb37b in EventUnit::_IOHandler (fd=73, which=2, arg=) at src/common/service/eventPlusUnit.cpp:216
#17 0x00000000005900e0 in event_process_active_single_queue (base=0x7f815c0165a0, flags=0) at event.c:1325
#18 event_process_active (base=0x7f815c0165a0, flags=0) at event.c:1392
#19 event_base_loop (base=0x7f815c0165a0, flags=0) at event.c:1589
#20 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c013190) at src/common/service/eventPlusWorker.cpp:39
#21 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#22 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#23 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7f80af5fe700 (LWP 29640)):
#0 is_utf8 (strUtf8=) at src/common/utils/utilUtf8.cpp:170
#1 is_utf8 (strUtf8=) at src/common/utils/utilUtf8.cpp:183
#2 nana::is_utf8 (strUtf8=) at src/common/utils/utilUtf8.cpp:380
#3 0x00000000004cc6fd in NeloEvent::verifyNeloEvent (ptrEvt=...) at src/common/core/neloEventCheck.cpp:123
#4 0x00000000004e0973 in NeloSource::pushToQueue (this=0x7f8164000bf0, ptrEvt=...) at src/common/core/neloSource.cpp:179
#5 0x0000000000535ec0 in ThriftNeloEventServerHandler::_append (this=0x7f815c011000, evt=...) at src/collector/sources/thrift/thriftEventSource.cpp:69
#6 0x000000000053630b in ThriftNeloEventServerHandler::ackedAppend (this=0x7f815c011000, evt=...) at src/collector/sources/thrift/thriftEventSource.cpp:28
#7 0x00000000004319fb in nelo::thrift::ThriftNeloEventServerProcessor::process_ackedAppend (this=0x7f815c011ca0, seqid=0, iprot=, oprot=0x7f8070005840, callContext=)
at src/third-party/thrift/ThriftNeloEventServer.cpp:811
#8 0x0000000000430594 in nelo::thrift::ThriftNeloEventServerProcessor::process_fn (this=, iprot=0x7f80700057f0, oprot=0x7f8070005840, fname="ackedAppend", seqid=0, callContext=)
at src/third-party/thrift/ThriftNeloEventServer.cpp:710
#9 0x00000000004307f3 in nelo::thrift::ThriftNeloEventServerProcessor::process (this=0x7f815c011ca0, piprot=, poprot=, callContext=0x7f80af5fb038)
at src/third-party/thrift/ThriftNeloEventServer.cpp:692
#10 0x000000000053a099 in ThriftHandler::doIt (this=0x7f815c011e90, pBuffer=, uiLength=, respBuff=..., cliAddr=) at ./src/common/service/thrift/thriftHandler.h:187
#11 0x000000000053cefc in FrameSessionboost::shared_ptr::_readRequestBody (this=0x7f8074003cb0) at ./src/common/service/thrift/frameSession.h:273
#12 0x000000000053d9ad in FrameSessionboost::shared_ptr::_socketRecv (this=0x7f8074003cb0, fd=77) at ./src/common/service/thrift/frameSession.h:176
#13 0x000000000053dfb5 in FrameSessionboost::shared_ptr::doEvent (this=0x7f8074003cb0, fd=, which=) at ./src/common/service/thrift/frameSession.h:108
#14 0x00000000004fb37b in EventUnit::_IOHandler (fd=77, which=2, arg=) at src/common/service/eventPlusUnit.cpp:216
#15 0x00000000005900e0 in event_process_active_single_queue (base=0x7f815c0183a0, flags=0) at event.c:1325
#16 event_process_active (base=0x7f815c0183a0, flags=0) at event.c:1392
#17 event_base_loop (base=0x7f815c0183a0, flags=0) at event.c:1589
#18 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c013da0) at src/common/service/eventPlusWorker.cpp:39
#19 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#20 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#21 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7f8130dfa700 (LWP 29608)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c008120) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c008120) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c008120) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7f810ebfd700 (LWP 29611)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c0094f0) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c0094f0) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c0094f0) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

---Type to continue, or q to quit---
Thread 38 (Thread 0x7f80b57fb700 (LWP 29637)):
#0 0x0000003e86ae7cd3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000059e252 in epoll_dispatch (base=0x7f815c016d20, tv=) at epoll.c:407
#2 0x000000000058ffeb in event_base_loop (base=0x7f815c016d20, flags=0) at event.c:1575
#3 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c013450) at src/common/service/eventPlusWorker.cpp:39
#4 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#5 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7f80b75fe700 (LWP 29635)):
#0 0x0000003e86ae7cd3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000059e252 in epoll_dispatch (base=0x7f815c015e20, tv=) at epoll.c:407
#2 0x000000000058ffeb in event_base_loop (base=0x7f815c015e20, flags=0) at event.c:1575
#3 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c012e40) at src/common/service/eventPlusWorker.cpp:39
#4 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#5 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7f80f57fb700 (LWP 29619)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00cb70) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00cb70) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00cb70) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7f80f6bfd700 (LWP 29617)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00bdd0) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00bdd0) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00bdd0) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f80f4dfa700 (LWP 29620)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00d240) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00d240) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00d240) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f816a647700 (LWP 29588)):
#0 0x0000003e86a3399d in sigtimedwait () from /lib64/libc.so.6
#1 0x000000000046ae22 in _childSignalCycle (this=0x7f816a645c60) at src/common/process/processChild.cpp:653
#2 ChildProcess::childProcessCycle (this=0x7f816a645c60) at src/common/process/processChild.cpp:714
#3 0x00000000004707f5 in MasterProcess::_forkChildProcess (this=0xa94c80) at src/common/process/processMaster.cpp:393
#4 0x000000000047f22c in MasterProcess::_timerCommand (this=0xa94c80) at src/common/process/processMasterSignal.cpp:170
#5 0x000000000047f4f5 in MasterProcess::onProcessSignal (this=0xa94c80, iCmd=14) at src/common/process/processMasterSignal.cpp:242
#6 0x000000000049ae3b in nana::SignalThread::_threadWorker (this=0xa94e78) at src/common/utils/utilThread.cpp:386
#7 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#8 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f81461fc700 (LWP 29600)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0xa97990, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c004b70) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c004b70) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c004b70) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6
---Type to continue, or q to quit---

Thread 31 (Thread 0x7f810d7fb700 (LWP 29613)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00a290) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00a290) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00a290) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7f8162bfd700 (LWP 29593)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0xa97990, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c001c00) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c001c00) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c001c00) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f81457fb700 (LWP 29601)):
#0 0x0000003e86aab97d in nanosleep () from /lib64/libc.so.6
#1 0x0000003e86aab7f0 in sleep () from /lib64/libc.so.6
#2 0x00000000005e5314 in rd_kafka_thread_main (arg=0x7f815c005450) at rdkafka.c:743
#3 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f80dbfff700 (LWP 29621)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00d910) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00d910) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00d910) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f80affff700 (LWP 29639)):
#0 0x0000003e86ae7cd3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000059e252 in epoll_dispatch (base=0x7f815c017c20, tv=) at epoll.c:407
#2 0x000000000058ffeb in event_base_loop (base=0x7f815c017c20, flags=0) at event.c:1575
#3 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c0139d0) at src/common/service/eventPlusWorker.cpp:39
#4 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#5 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f80be1fc700 (LWP 29631)):
#0 0x0000003e86ae7cd3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000059e252 in epoll_dispatch (base=0x7f815c014040, tv=) at epoll.c:407
#2 0x000000000058ffeb in event_base_loop (base=0x7f815c014040, flags=0) at event.c:1575
#3 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c0123c0) at src/common/service/eventPlusWorker.cpp:39
#4 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#5 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f80bd7fb700 (LWP 29632)):
#0 0x0000003e86ae7cd3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000059e252 in epoll_dispatch (base=0x7f815c0147a0, tv=) at epoll.c:407
#2 0x000000000058ffeb in event_base_loop (base=0x7f815c0147a0, flags=0) at event.c:1575
#3 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c012660) at src/common/service/eventPlusWorker.cpp:39
#4 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#5 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f80f75fe700 (LWP 29616)):
#0 assign (value=) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/bits/char_traits.h:247
#1 _M_set_length_and_sharable (value=) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/bits/basic_string.h:207
#2 push_back (value=) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/bits/basic_string.h:920
---Type to continue, or q to quit---
#3 operator+= (value=) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/bits/basic_string.h:815
#4 Json::valueToQuotedString (value=) at src/third-party/jsoncpp/jsoncpp.cpp:3551
#5 0x00000000004428b1 in Json::FastWriter::writeValue (this=0x7f80f75fd010, value=...) at src/third-party/jsoncpp/jsoncpp.cpp:3611
#6 0x00000000004427dc in Json::FastWriter::writeValue (this=0x7f80f75fd010, value=...) at src/third-party/jsoncpp/jsoncpp.cpp:3643
#7 0x0000000000442a8f in Json::FastWriter::write (this=0x7f80f75fd010, root=...) at src/third-party/jsoncpp/jsoncpp.cpp:3587
#8 0x0000000000532692 in KafkaSink::_getJsonBody (this=, ptrEvt=) at src/collector/sinks/kafka/kafkaSink.cpp:271
#9 0x0000000000533eb2 in KafkaSink::transfer (this=0x7f81640017a0, evt=std::vector of length 1, capacity 1 = {...}, dealFail=true) at src/collector/sinks/kafka/kafkaSink.cpp:141
#10 0x00000000004d73c3 in NeloSender::_NeloSender::running (this=0x7f80900076f0) at src/common/core/neloSender.cpp:95
#11 0x000000000049f745 in nana::SafeThreadPool::Customer::running (this=0x7f815c00b700) at src/common/utils/utilThread.h:426
#12 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00b700) at src/common/utils/utilThread.cpp:97
#13 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#14 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#15 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f810e1fc700 (LWP 29612)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c009bc0) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c009bc0) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c009bc0) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f80bcdfa700 (LWP 29633)):
#0 0x0000003e86ae7cd3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000059e252 in epoll_dispatch (base=0x7f815c014f20, tv=) at epoll.c:407
#2 0x000000000058ffeb in event_base_loop (base=0x7f815c014f20, flags=0) at event.c:1575
#3 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c012900) at src/common/service/eventPlusWorker.cpp:39
#4 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#5 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f81335fe700 (LWP 29604)):
#0 0x0000003e86ade023 in poll () from /lib64/libc.so.6
#1 0x00000000005eb9e8 in rd_kafka_broker_io_serve (rkb=0x7f815c006560) at rdkafka_broker.c:2005
#2 0x00000000005ed125 in rd_kafka_broker_ua_idle (arg=0x7f815c006560) at rdkafka_broker.c:2029
#3 rd_kafka_broker_thread_main (arg=0x7f815c006560) at rdkafka_broker.c:3000
#4 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f8169c46700 (LWP 29589)):
#0 0x0000003e86e0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f817333d802 in boost::condition_variable::timed_wait(boost::unique_lockboost::mutex&, boost::posix_time::ptime const&) () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#2 0x00007f8173339256 in boost::this_thread::sleep(boost::posix_time::ptime const&) () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#3 0x00000000004a5708 in sleep (uiMilliSec=) at /data3/nelo2/building//include/boost/thread/detail/thread.hpp:347
#4 nana::utilSleep (uiMilliSec=) at src/common/utils/utilTools.cpp:236
#5 0x00000000004ad656 in ComplexStatis::running (this=0xa95900) at src/common/statistics/complexStatis.cpp:81
#6 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0xa95900) at src/common/utils/utilThread.cpp:97
#7 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#8 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f8133fff700 (LWP 29603)):
#0 0x0000003e86ade023 in poll () from /lib64/libc.so.6
#1 0x00000000005eb9e8 in rd_kafka_broker_io_serve (rkb=0x7f815c005f00) at rdkafka_broker.c:2005
#2 0x00000000005ed125 in rd_kafka_broker_ua_idle (arg=0x7f815c005f00) at rdkafka_broker.c:2029
#3 rd_kafka_broker_thread_main (arg=0x7f815c005f00) at rdkafka_broker.c:3000
#4 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f810cdfa700 (LWP 29614)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00a960) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00a960) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00a960) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
---Type to continue, or q to quit---
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f81321fc700 (LWP 29606)):
#0 0x0000003e86e0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f817333d802 in boost::condition_variable::timed_wait(boost::unique_lockboost::mutex&, boost::posix_time::ptime const&) () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#2 0x00007f8173339256 in boost::this_thread::sleep(boost::posix_time::ptime const&) () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#3 0x00000000004a5708 in sleep (uiMilliSec=) at /data3/nelo2/building//include/boost/thread/detail/thread.hpp:347
#4 nana::utilSleep (uiMilliSec=) at src/common/utils/utilTools.cpp:236
#5 0x00000000004dc679 in nana::TimerThread::running (this=0x7f8164001910) at ./src/common/common/../utils/utilThread.h:606
#6 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f8164001910) at src/common/utils/utilThread.cpp:97
#7 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#8 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f810f5fe700 (LWP 29610)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c008e20) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c008e20) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c008e20) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f8144dfa700 (LWP 29602)):
#0 0x0000003e86ade023 in poll () from /lib64/libc.so.6
#1 0x00000000005eb9e8 in rd_kafka_broker_io_serve (rkb=0x7f815c0058a0) at rdkafka_broker.c:2005
#2 0x00000000005ed125 in rd_kafka_broker_ua_idle (arg=0x7f815c0058a0) at rdkafka_broker.c:2029
#3 rd_kafka_broker_thread_main (arg=0x7f815c0058a0) at rdkafka_broker.c:3000
#4 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f8169245700 (LWP 29590)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f816a645e98, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049adee in nana::SignalThread::_threadWorker (this=0x7f816a645e58) at src/common/utils/utilThread.cpp:367
#3 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#4 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f80f7fff700 (LWP 29615)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00b030) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00b030) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00b030) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f80bffff700 (LWP 29627)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00f450) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00f450) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00f450) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f8132bfd700 (LWP 29605)):
#0 0x0000003e86ade023 in poll () from /lib64/libc.so.6
#1 0x00000000005eb9e8 in rd_kafka_broker_io_serve (rkb=0x7f815c006bc0) at rdkafka_broker.c:2005
#2 0x00000000005ed125 in rd_kafka_broker_ua_idle (arg=0x7f815c006bc0) at rdkafka_broker.c:2029
#3 rd_kafka_broker_thread_main (arg=0x7f815c006bc0) at rdkafka_broker.c:3000
#4 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f80d97fb700 (LWP 29625)):
---Type to continue, or q to quit---
#0 0x0000003e86ade023 in poll () from /lib64/libc.so.6
#1 0x00000000005eb9e8 in rd_kafka_broker_io_serve (rkb=0x7f8128000d90) at rdkafka_broker.c:2005
#2 0x00000000005ec283 in rd_kafka_broker_producer_serve (rkb=0x7f8128000d90) at rdkafka_broker.c:2128
#3 0x00000000005ed1d8 in rd_kafka_broker_thread_main (arg=0x7f8128000d90) at rdkafka_broker.c:3002
#4 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f80dabfd700 (LWP 29623)):
#0 Json::Value::Value (this=0x7f80dabfbfc0, other=...) at src/third-party/jsoncpp/jsoncpp.cpp:1951
#1 0x000000000043eb11 in Json::Value::operator= (this=0x7f80cc303680, other=) at src/third-party/jsoncpp/jsoncpp.cpp:2036
#2 0x000000000053265f in KafkaSink::_getJsonBody (this=, ptrEvt=) at src/collector/sinks/kafka/kafkaSink.cpp:265
#3 0x0000000000533eb2 in KafkaSink::transfer (this=0x7f81640017a0, evt=std::vector of length 20, capacity 20 = {...}, dealFail=true) at src/collector/sinks/kafka/kafkaSink.cpp:141
#4 0x00000000004d73c3 in NeloSender::_NeloSender::running (this=0x7f8070005380) at src/common/core/neloSender.cpp:95
#5 0x000000000049f745 in nana::SafeThreadPool::Customer::running (this=0x7f815c00e6b0) at src/common/utils/utilThread.h:426
#6 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00e6b0) at src/common/utils/utilThread.cpp:97
#7 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#8 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f80aebfd700 (LWP 29641)):
#0 0x0000003e86ae7cd3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000059e252 in epoll_dispatch (base=0x7f815c018b20, tv=) at epoll.c:407
#2 0x000000000058ffeb in event_base_loop (base=0x7f815c018b20, flags=0) at event.c:1575
#3 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c012030) at src/common/service/eventPlusWorker.cpp:39
#4 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#5 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f80f61fc700 (LWP 29618)):
#0 0x0000003e86e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000049f4e7 in boost::condition_variable_any::waitboost::mutex::scoped_lock (this=0x7f815c007a10, m=...) at /data3/nelo2/building//include/boost/thread/pthread/condition_variable.hpp:137
#2 0x000000000049f809 in dequeue (this=0x7f815c00c4a0) at src/common/utils/utilThread.h:249
#3 nana::SafeThreadPool::Customer::running (this=0x7f815c00c4a0) at src/common/utils/utilThread.h:421
#4 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f815c00c4a0) at src/common/utils/utilThread.cpp:97
#5 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#6 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f80b4dfa700 (LWP 29638)):
#0 0x0000003e86ae7cd3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000059e252 in epoll_dispatch (base=0x7f815c0174a0, tv=) at epoll.c:407
#2 0x000000000058ffeb in event_base_loop (base=0x7f815c0174a0, flags=0) at event.c:1575
#3 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c013710) at src/common/service/eventPlusWorker.cpp:39
#4 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#5 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f80b7fff700 (LWP 29634)):
#0 0x0000003e86ae7cd3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000059e252 in epoll_dispatch (base=0x7f815c0156a0, tv=) at epoll.c:407
#2 0x000000000058ffeb in event_base_loop (base=0x7f815c0156a0, flags=0) at event.c:1575
#3 0x00000000004fcf94 in IOService::_workerThread (this=0x7f815c012ba0) at src/common/service/eventPlusWorker.cpp:39
#4 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#5 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f80bf5fe700 (LWP 29628)):
#0 0x0000003e86ade023 in poll () from /lib64/libc.so.6
#1 0x00000000005eb9e8 in rd_kafka_broker_io_serve (rkb=0x7f8128001a50) at rdkafka_broker.c:2005
#2 0x00000000005ec283 in rd_kafka_broker_producer_serve (rkb=0x7f8128001a50) at rdkafka_broker.c:2128
#3 0x00000000005ed1d8 in rd_kafka_broker_thread_main (arg=0x7f8128001a50) at rdkafka_broker.c:3002
#4 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f80ae1fc700 (LWP 29642)):
#0 0x0000003e86e0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f817333d802 in boost::condition_variable::timed_wait(boost::unique_lockboost::mutex&, boost::posix_time::ptime const&) () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#2 0x00007f8173339256 in boost::this_thread::sleep(boost::posix_time::ptime const&) () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#3 0x00000000004a5708 in sleep (uiMilliSec=) at /data3/nelo2/building//include/boost/thread/detail/thread.hpp:347
---Type to continue, or q to quit---
#4 nana::utilSleep (uiMilliSec=) at src/common/utils/utilTools.cpp:236
#5 0x00000000004e478c in nana::TimerThread::running (this=0x7f8164000c30) at ./src/common/common/../utils/utilThread.h:606
#6 0x00000000004992b9 in nana::SafeThread::_threadWorker (this=0x7f8164000c30) at src/common/utils/utilThread.cpp:97
#7 0x00007f8173338cf3 in thread_proxy () from /data3/nelo2/building/collector/bin/../../lib/libboost_thread.so.1.49.0
#8 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f80b6bfd700 (LWP 29629)):
#0 0x0000003e86ade023 in poll () from /lib64/libc.so.6
#1 0x00000000005eb9e8 in rd_kafka_broker_io_serve (rkb=0x7f81280020b0) at rdkafka_broker.c:2005
#2 0x00000000005ec283 in rd_kafka_broker_producer_serve (rkb=0x7f81280020b0) at rdkafka_broker.c:2128
#3 0x00000000005ed1d8 in rd_kafka_broker_thread_main (arg=0x7f81280020b0) at rdkafka_broker.c:3002
#4 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f80d8dfa700 (LWP 29626)):
#0 0x00000000005ec47d in rd_kafka_broker_produce_toppar (rkb=0x7f81280013f0) at rdkafka_broker.c:1944
#1 rd_kafka_broker_producer_serve (rkb=0x7f81280013f0) at rdkafka_broker.c:2105
#2 0x00000000005ed1d8 in rd_kafka_broker_thread_main (arg=0x7f81280013f0) at rdkafka_broker.c:3002
#3 0x0000003e86e07851 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003e86ae76dd in clone () from /lib64/libc.so.6

···

@edenhill
Copy link
Contributor

edenhill commented Nov 8, 2013

Could you check one more thing for me, when it crashes in rd_kafka_broker_produce_toppar(), type:
p *rkbuf
p msgcnt
p *rktp

and paste that into this bug issue report.

How large are the messages you are producing? (they need to be smaller than message.max.bytes, which you can increase as you see fit but remember to do it on the broker aswell)

@xiaojun789
Copy link
Author

···
(gdb) p *rkbuf
$1 = {rkbuf_link = {tqe_next = 0x0, tqe_prev = 0x0}, rkbuf_corrid = 0, rkbuf_ts_retry = 0, rkbuf_flags = 0, rkbuf_msg = {msg_name = 0x0, msg_namelen = 0, msg_iov = 0x7f80a805cf48, msg_iovlen = 5, msg_control = 0x0, msg_controllen = 0,
msg_flags = 0}, rkbuf_iov = 0x7f80a805cf48, rkbuf_iovcnt = 1024, rkbuf_of = 0, rkbuf_len = 0, rkbuf_size = 5566, rkbuf_buf = 0x7f80a8060f48 "", rkbuf_buf2 = 0x0, rkbuf_reqhdr = {Size = 0, ApiKey = 0, ApiVersion = 0, CorrId = 0},
rkbuf_reshdr = {Size = 0, CorrId = 0}, rkbuf_expected_size = 0, rkbuf_cb = 0, rkbuf_refcnt = 1, rkbuf_opaque = 0x0, rkbuf_retries = 0, rkbuf_ts_enq = 0, rkbuf_ts_sent = 0, rkbuf_ts_timeout = 0, rkbuf_msgq = {rkmq_msgs = {
tqh_first = 0x0, tqh_last = 0x7f80a805cf28}, rkmq_msg_cnt = 0, rkmq_msg_bytes = 0}}
(gdb) p msgcnt
$2 =
(gdb) p *rktp
$3 = {rktp_rklink = {tqe_next = 0x0, tqe_prev = 0x0}, rktp_rkblink = {tqe_next = 0x0, tqe_prev = 0x7f8128001498}, rktp_rktlink = {tqe_next = 0x0, tqe_prev = 0x0}, rktp_rkt = 0x7f815c0053a0, rktp_partition = 1,
rktp_leader = 0x7f81280013f0, rktp_refcnt = 19, rktp_lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>,
__align = 0}, rktp_msgq = {rkmq_msgs = {tqh_first = 0x7f80cc1fad00, tqh_last = 0x7f80cc2e45d0}, rkmq_msg_cnt = 98, rkmq_msg_bytes = 73198}, rktp_xmit_msgq = {rkmq_msgs = {tqh_first = 0x7f80f8309490, tqh_last = 0x7f80c0068c90},
rkmq_msg_cnt = 555, rkmq_msg_bytes = 6836899}, rktp_ts_last_xmit = 24971403536470, rktp_fetchq = {rkq_lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0,
__next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, rkq_cond = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0},
__size = '\000' <repeats 47 times>, __align = 0}, rkq_q = {tqh_first = 0x0, tqh_last = 0x7f81280029b8}, rkq_qlen = 0}, rktp_fetch_state = RD_KAFKA_TOPPAR_FETCH_NONE, rktp_ts_offset_req_next = 0, rktp_query_offset = 0,
rktp_next_offset = 0, rktp_app_offset = 0, rktp_commited_offset = 0, rktp_eof_offset = 0, rktp_flags = 0, rktp_c = {tx_msgs = 293149, tx_bytes = 928849461}}
···

@edenhill
Copy link
Contributor

edenhill commented Nov 8, 2013

How large are the messages you are producing? (they need to be smaller than message.max.bytes, which you can increase as you see fit but remember to do it on the broker and consumers aswell)

@xiaojun789
Copy link
Author

for log size,
the sdk default value for "message.max.bytes" is 40000000, also I set the kafka server "message.max.bytes" is 4000000 . and from the log of kafka server, there is infomation about log size is out of limitation .

@edenhill
Copy link
Contributor

edenhill commented Nov 8, 2013

I think this happens because you are trying to produce a message larger than "message.max.bytes", which is not handled correctly (I will fix that).

Could you type this in gdb:
p rk->rk_conf.max_msg_size
p *rktp->rktp_xmit_msgq.rkmq_msgs.tqh_first

@xiaojun789
Copy link
Author

(gdb) p *rktp->rktp_xmit_msgq.rkmq_msgs.tqh_first
$1 = {rkm_link = {tqe_next = 0x7f80ec27d890, tqe_prev = 0x7f8128002938}, rkm_flags = 2, rkm_len = 6239888, rkm_payload = 0x7f80f83094d8, rkm_opaque = 0x0, rkm_partition = -1, rkm_key = 0x7f80f8055f30, rkm_ts_timeout = 24971703369298}
(gdb) p rk->rk_conf.max_msg_size
No symbol "rk" in current context.
(gdb)

@edenhill
Copy link
Contributor

edenhill commented Nov 8, 2013

Sorry, try:

p rkb->rkb_rk->rk_conf.max_msg_size

The message is 6.2Mb, and if you havent changed "messages.max.bytes" which defaults to 4Mb, then the message is too large to be produced and triggers a bug. I will push a fix for this causing the rd_kafka_produce() call to fail with errno set to EMSGSIZE.

Ive reproduced this locally:

examples/rdkafka_performance -P -t onepart -s 4000001 -b localhost -c 1 -d msg

What you want to do is increase "messages.max.bytes" to fit your messages:

rd_kafka_conf_set(conf, "messages.max.bytes", "8000000", errstr, sizeof(errstr));

@xiaojun789
Copy link
Author

(gdb) p rkb->rkb_rk->rk_conf.max_msg_size
$2 = 4000000

@edenhill
Copy link
Contributor

edenhill commented Nov 8, 2013

Yep, what I thought. Thanks.

@xiaojun789
Copy link
Author

greate, waiting for your patch. Thanks~!

edenhill added a commit that referenced this issue Nov 8, 2013
@edenhill
Let produce API honour "message.max.bytes" (issue #24)
83478fb 
Producing a message larger than "message.max.bytes" will now make
rd_kafka_produce() return -1 and set errno to EMSGSIZE.
@edenhill
Copy link
Contributor

edenhill commented Nov 8, 2013

Could you update and verify that this issue has been fixed in your setup aswell?

@xiaojun789
Copy link
Author

OK, I will check it.

edenhill added a commit that referenced this issue Nov 8, 2013
@edenhill
Producer msg count was increased if msg was too large (for issue #24)
0beaebc
edenhill added a commit that referenced this issue Nov 8, 2013
@edenhill
0003-msgmaxsize: test case for message sizes (issue #24)
7632484
@xiaojun789
Copy link
Author

I already test in my side, it works.

@DEvil0000 DEvil0000 mentioned this issue Oct 12, 2015
Closed
@matthrae matthrae mentioned this issue Nov 8, 2018
Closed
@MockingJayWong MockingJayWong mentioned this issue Jul 9, 2019
Closed
7 tasks
Closed
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@edenhill edenhill

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@edenhill @xiaojun789