Could threadSanitizer be used with static linked lib

[0chunhui]

An old project which is linked with some xx.a file, can I add thead sanitizer to this project?
When I did so, a lot of undefined referenced __tsan_func_entry printed.
What should I do?

[dvyukov]

Should work. Did you link with -fsanitize=thread as well? If yes, please provide exact sequence of commands you execute and/or reproducer.

[0chunhui]

Yes, I linked with -fsanitize=thread as well. The Makefile is very large, I added -fsanitize=thread to the tail of CC=xxx CPP= xxxx and LD = xxx. Should the .a file be recompiled with -fsanitize=thread?

[chefmax]

Do you link static binary or dynamic? If static, try add '-static-libtsan' to your link flags.

[dvyukov]

Yes, I linked with -fsanitize=thread as well.

Then it should work. Try to create a minimal repro. It will be much easier to figure out what happens then.

[0chunhui]

Do you link static binary or dynamic? If static, try add '-static-libtsan' to your link flags.

Still don't work.

some errors are:
/usr/linclude/c++/4.8.2/bits/stl_heap.h:327 undefined reference to '__tsan_func_read8'
/usr/linclude/c++/4.8.2/bits/stl_heap.h:327 undefined reference to '__tsan_func_write8'
/usr/linclude/c++/4.8.2/bits/stl_heap.h:327 undefined reference to '__tsan_func_exit'

[0chunhui]

A small repo worked when I use the following command:
g++ main.cpp -c -fsanitize=thread -pie -fPIE -std=c++11
g++ main.o -o m -fsanitize=thread -pie -fPIE

I cannot figure out why it didn't work for the big project.

[dvyukov]

But you did not link any uninstrumented static libraries to the binary. Though, I did not expect it to make any difference.

Re linker "undefined reference" errors: that happens because tsan runtime is not linked. Does it happen when you link executable or shared library? Can you capture linker command line and post here?

[0chunhui]

Thank you. I rechecked the linker command, and found out that I missed -fsanitize=thread when I modifyed it last time. I added it to the Makefile, the link process failed with error:
/usr/bin/ld relocation R_X86_64_32 against '--pthread_key_create@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
xxx.a: could not read symbols: wrong value

[dvyukov]

xxx.a is your prebuilt shared library, right?
This error is expected -- we are building a pie executable, but the object file is not compiled with -fPIE/PIC.
Newer tsan supports binaries built without -pie. I am not sure if this support has reached latest versions of gcc, but fresh clang supports it.

[0chunhui]

Yes, I think my compiler doesn't support, and my boss won't let me update it.
So, it is the only way that I recompile XXX.a with -fPIC?

XXX.a is generated by cmake, which is wrote by someone else, I have not idea how to add -fPIC to it..
Do you have any idea?

[dvyukov]

No, I am not a cmake expert, sorry. Maybe it also supports something like CFLAGS?

[0chunhui]

Thank you all the same. After Compile with -fPIC, the linker doesn't report that error.
However, another same error reported, libtcmalloc.a....... now I am searching where is the source code...
Why all the xxx.a file needed to be recompiled with -fPIC? It is not very easy for me to used it in an old project.

[dvyukov]

Sanitizers are not compatible with tcmalloc (and other malloc replacement libraries) as they replace malloc themselves. So just remove libtcmalloc.a from linker command when building with sanitizers.

[0chunhui]

Hi, I succeed in linking the program finally. One thing I don't understand is that, some of my XXX.a lib need to be recompiled with -fPIC, while some of them don't need, Why?

[dvyukov]

Great!
I see at least 2 reasons for that:

1. Trivial functions can be position-independent even when not compiled with -fPIE. For example:

int foo(int x) { return x + 1; }

will not involve any relocations and will position-independent (though, I did not try).
2. Maybe there is something you miss in the build process. E.g. these .a files are not actually linked into the tsan binary. Makefiles can be complex.

[0chunhui]

From the link command showed when linking, the .a files are surely linked into the binary. Those files are all about logs, I don't have the source code.
Thank you very much. I learned a lot from you, today. Thank you.

[dvyukov]

You are welcome.

[0chunhui]

Last night when I was on the bed, I thought of a question. If one dynamic lib xxxx.so which uses tcmalloc, is used in my project, is it necessary for me to recompile it without tcmalloc?

[dvyukov]

What do you mean "uses tcmalloc"? tcmalloc is usually LD_PRELOADed, or just linked into the final executable. There is usually no need to depend/use tcmalloc explicitly, it's just a malloc replacement. Dynamic libraries use malloc and depend on libc.so.

[0chunhui]

In a Makefile of one of my old project, the linking commands include line like this:
/usr/local/lib/libtcmalloc.a /usr/local/lib/libunwind.a
This situation is probably the second one that you mentioned. You mean there is other ways to uses tcmalloc?
In this file, the only way to do is to remove those .a files?

[dvyukov]

Yes, explicit linking of libtcmalloc.a needs to be removed.

[0chunhui]

Well, OK..
Another question, If I add -fPIC into the compile flags in release version, would it lead to any side effect?

[0chunhui]

When I run the binary file, it takes 300+% cpu, 97T VIRT, 2.6g RES .... All the processes have no response...

[dvyukov]

Another question, If I add -fPIC into the compile flags in release version, would it lead to any side effect?

-fPIC can slightly affect performance.
But if you also link with -pie, it will improve security due to ASLR.
I would suggest to replace -fPIC with -fPIE which is enough for main executable and leads to faster code.

When I run the binary file, it takes 300+% cpu, 97T VIRT, 2.6g RES .... All the processes have no response...

Is it a binary built with tsan?
Can you capture thread stacks with gdb? What is it doing?

[0chunhui]

Yes, it's a binary built with tsan.
The stack is like this:
(gdb) bt
#0 0x00007fb53a98aa09 in syscall () from /lib64/libc.so.6
#1 0x00007fb53dc0a280 in ?? () from /lib64/libtsan.so.0
#2 0x00007fb53dbe5c32 in ?? () from /lib64/libtsan.so.0
#3 0x00007fb53dbff5a2 in ?? () from /lib64/libtsan.so.0
#4 0x00007fb53dbf06e3 int sleep () from /lib64/libtsan.so.0
....

(gdb) c
Continuing

(gdb) bt
#0 0x.......... in syscall (0 from /lib64/litsan.so.0
#1 0x ...... in ?? () from /lib64/litsan.so.0
#2 0x ........ in ?? () from /lib64/litsan.so.0
#3 0x ....... in ?? () from /lib64/litsan.so.0
#4 0x ....... in usleep () from /lib64/litsan.so.0
#5 0x .......... in mutex_spinlock2（int*） () from ../../lib/libio64.so

max CPU 680+%.....

[dvyukov]

So libio64 waits on a mutex. Who is holding the mutex?
You can do 'thread apply all bt' in gdb to get bracktrace of all threads.
It's a pity that gcc's libtsan does not have symbols.
What version of gcc do you use? There were some bugs. I can suggest to try the latest clang version.

[0chunhui]

It seems that some threads need to write to a message queue concurrently, and while a thread is waiting for the mutex, it would call usleep from /lib64/libtsan.so.0. Each Time I press 'Ctrl C' and 'thread apply all bt' to see the stack, the stack is more or less the same. The virtual memory watched from top command is always 97.001t, 500%~600%, or even more CPU is occupied.

My gcc version is 4.8.5.
I'm afraid that my boss won't let me to update gcc arbitrarily.
Maybe I would try to convince him that we do not use tsan in this project, maybe next time, after we update gcc.

[dvyukov]

Then maybe it works, but very slowly. Would it be possible to replace the mutex with pthread_mutex_t under tsan?

[0chunhui]

The mutex_spinlock is in a dynamic library provided by the platform team in our company. It was written by some old guys in 2002. It is provided in the form of .so, no one in my team has its source code, and they say that the source code is top secret....(faint). So maybe it's hard to replace it.