static linking of librdkafka.a needs dynamic pthreads linking

[winbatch]

Seeing this on Red Hat Linux 6. It's possible there's something in my code leading up to it, but the stack trace seems to show only librdkafka code:
#0 0x0000003456832885 in raise () from /lib64/libc.so.6

Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6_2.9.x86_64 libgcc-4.4.6-3.el6.x86_64 libstdc++-4.4.6-3.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) where
#0 0x0000003456832885 in raise () from /lib64/libc.so.6
#1 0x0000003456834065 in abort () from /lib64/libc.so.6
#2 0x000000345686f977 in __libc_message () from /lib64/libc.so.6
#3 0x0000003456875296 in malloc_printerr () from /lib64/libc.so.6
#4 0x0000003456865a1d in fclose@@GLIBC_2.2.5 () from /lib64/libc.so.6
#5 0x00000034584047fa in _nss_files_gethostbyname2_r () from /lib64/libnss_files.so.2
#6 0x0000003456900a39 in gethostbyname2_r@@GLIBC_2.2.5 () from /lib64/libc.so.6
#7 0x00000034568ce9f6 in gaih_inet () from /lib64/libc.so.6
#8 0x00000034568d0170 in getaddrinfo () from /lib64/libc.so.6
#9 0x00007f288c761d8c in rd_getaddrinfo (nodesvc=, defsvc=0x7f287c0008e0 "5757", flags=32, family=,

socktype=<value optimized out>, protocol=<value optimized out>, errstr=0x7f2877ffeda8) at rdaddr.c:161

#10 0x00007f288c757d46 in rd_kafka_broker_resolve (rkb=0x1d15fe0) at rdkafka_broker.c:432
#11 0x00007f288c75e03b in rd_kafka_broker_connect (arg=0x1d15fe0) at rdkafka_broker.c:1292
#12 rd_kafka_broker_thread_main (arg=0x1d15fe0) at rdkafka_broker.c:3041
#13 0x00000034570077f1 in start_thread () from /lib64/libpthread.so.0
#14 0x00000034568e5ccd in clone () from /lib64/libc.so.6

Valgrind shows:

==24258== Thread 3:^M
==24258== Invalid read of size 8^M
==24258== at 0x345687112E: _IO_file_underflow@@GLIBC_2.2.5 (in /lib64/libc-2.12.so)^M
==24258== by 0x3456872C2D: _IO_default_uflow (in /lib64/libc-2.12.so)^M
==24258== by 0x3456867419: _IO_getline_info (in /lib64/libc-2.12.so)^M
==24258== by 0x3456870140: fgets_unlocked (in /lib64/libc-2.12.so)^M
==24258== by 0x34584046CE: _nss_files_gethostbyname2_r (in /lib64/libnss_files-2.12.so)^M
==24258== by 0x3456900A38: gethostbyname2_r@@GLIBC_2.2.5 (in /lib64/libc-2.12.so)^M
==24258== by 0x34568CE9F5: gaih_inet (in /lib64/libc-2.12.so)^M
==24258== by 0x34568D016F: getaddrinfo (in /lib64/libc-2.12.so)^M
==24258== by 0x529DD8B: rd_getaddrinfo (rdaddr.c:161)^M
==24258== by 0x5293D45: rd_kafka_broker_resolve (rdkafka_broker.c:432)^M
==24258== by 0x529A03A: rd_kafka_broker_thread_main (rdkafka_broker.c:1292)^M
==24258== by 0x34570077F0: start_thread (in /lib64/libpthread-2.12.so)^M
==24258== Address 0x4ef3d40 is 144 bytes inside a block of size 568 free'd^M
==24258== at 0x4A06300: free (vg_replace_malloc.c:446)^M
==24258== by 0x3456865A1C: fclose@@GLIBC_2.2.5 (in /lib64/libc-2.12.so)^M
==24258== by 0x34584047F9: _nss_files_gethostbyname2_r (in /lib64/libnss_files-2.12.so)^M
==24258== by 0x3456900A38: gethostbyname2_r@@GLIBC_2.2.5 (in /lib64/libc-2.12.so)^M
==24258== by 0x34568CE9F5: gaih_inet (in /lib64/libc-2.12.so)^M
==24258== by 0x34568D016F: getaddrinfo (in /lib64/libc-2.12.so)^M
==24258== by 0x529DD8B: rd_getaddrinfo (rdaddr.c:161)^M
==24258== by 0x5293D45: rd_kafka_broker_resolve (rdkafka_broker.c:432)^M
==24258== by 0x529A03A: rd_kafka_broker_thread_main (rdkafka_broker.c:1292)^M
==24258== by 0x34570077F0: start_thread (in /lib64/libpthread-2.12.so)^M
==24258== by 0x34568E5CCC: clone (in /lib64/libc-2.12.so)^M

With debug turned on, this is what I see:
1388099207.266 RDKAFKA-7-BROKER: LARA#producer-0: kafkadevcluster1-1.maskedout.com:5757/bootstrap: Added new broker with NodeId -1
1388099207.266 RDKAFKA-7-BRKMAIN: LARA#producer-0: kafkadevcluster1-1.maskedout.com:5757/bootstrap: Enter main broker thread
1388099207.266 RDKAFKA-7-CONNECT: LARA#producer-0: kafkadevcluster1-1.maskedout.com:5757/bootstrap: broker in state DOWN connecting
1388099207.266 RDKAFKA-7-BROKER: LARA#producer-0: kafkadevcluster1-2.maskedout.com:5757/bootstrap: Added new broker with NodeId -1
1388099207.266 RDKAFKA-7-BRKMAIN: LARA#producer-0: kafkadevcluster1-2.maskedout.com:5757/bootstrap: Enter main broker thread
1388099207.266 RDKAFKA-7-CONNECT: LARA#producer-0: kafkadevcluster1-2.maskedout.com:5757/bootstrap: broker in state DOWN connecting
1388099207.267 RDKAFKA-7-BROKER: LARA#producer-0: kafkadevcluster1-3.maskedout.com:5757/bootstrap: Added new broker with NodeId -1
AFTER TRYING BROKERS:kafkadevcluster1-1.maskedout.com:5757,kafkadevcluster1-2.maskedout.com:5757,kafkadevcluster1-3.maskedout.com:5757
1388099207.267 RDKAFKA-7-BRKMAIN: LARA#producer-0: kafkadevcluster1-3.maskedout.com:5757/bootstrap: Enter main broker thread
1388099207.267 RDKAFKA-7-CONNECT: LARA#producer-0: kafkadevcluster1-3.maskedout.com:5757/bootstrap: broker in state DOWN connecting
1388099207.267 RDKAFKA-7-TOPIC: LARA#producer-0: New local topic: Lara
*** glibc detected *** ./replicator: double free or corruption (out): 0x00007fd1f0001200 ***

[edenhill]

Hi,

no, I havent seen this one before.

Some questions:

• Does it happen on each execution of your program?
• Does it rely on a certain number or set of brokers?
• Is it reproducible with the rdkafka_example program?
• Are you linking your program statically? I found this: https://sourceware.org/bugzilla/show_bug.cgi?id=10652

[winbatch]

Hi Magnus,

To answer your questions:
Yes - happens every time
No - doesn't matter the number
No - doesn't happen on the example program, or even my own sample program
Yes - I think this is almost certainly the problem. I have a binary
which does a dlopen on a shared library that I wrote. That shared library
is statically linked to librdkafka.a. Based on the link you sent, I
wasn't sure if a final solution was found? From your reading, does it
mean I have to change how you build librdkafka.a to link in pthread?

On Fri, Dec 27, 2013 at 2:35 AM, Magnus Edenhill
[email protected]:

Hi,

no, I havent seen this one before.

Some questions:

• Does it happen on each execution of your program?
• Does it rely on a certain number or set of brokers?
• Is it reproducible with the rdkafka_example program?
• Are you linking your program statically? I found this:
  https://sourceware.org/bugzilla/show_bug.cgi?id=10652

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-31250326
.

[edenhill]

From what I understand you may link your program statically with librdkafka, but you need to link it dynamically with pthreads.
Can you try that?

[winbatch]

Success! (Although it feels weird to make my single threaded binary link
with pthread just 'in case' one of the .so's I dlopen happens to use
threads, but I think I can deal with that :) )

On Fri, Dec 27, 2013 at 6:14 AM, Magnus Edenhill
[email protected]:

From what I understand you may link your program statically with
librdkafka, but you need to link it dynamically with pthreads.
Can you try that?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-31256767
.

[edenhill]

Thing is that librdkafka is threaded (one thread per rd_kafka_t + one thread for each broker), and each broker thread will try to resolve its broker address using getaddrinfo(), and thats where it fails :)

[fongjeng-kum]

Success! (Although it feels weird to make my single threaded binary link with pthread just 'in case' one of the .so's I dlopen happens to use threads, but I think I can deal with that :) )

On Fri, Dec 27, 2013 at 6:14 AM, Magnus Edenhill [email protected]:

From what I understand you may link your program statically with
librdkafka, but you need to link it dynamically with pthreads.
Can you try that?
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-31256767
.

Hi, can I know the command to statically link the librdkafka that you have success on?

Thanks.

[winbatch]

This was 8 years ago so no, I no longer have the command ;)