SIGSEGV when the client loads the auxiliary library and the auxiliary library invokes printf.

[wierton]

Describe the bug

Necessary Conditions:

1. client.so wraps the function func_for_hook in the target application.
2. Upon invocation of func_to_hook by the target application, client.so executes pre_func_for_hook.
3. Inside pre_func_to_hook, client.so uses dr_load_aux_library to load the function libtest_main from libtest.so.
4. client.so executes libtest_main.
5. Within libtest_main, the function simply calls printf and returns its return

Then, drrun crashes.

To Reproduce
My client.c is:

#define _GNU_SOURCE
#include "dr_api.h"
#include "drmgr.h"
#include "drsyms.h"
#include "drwrap.h"

#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>

static void pre_func_for_hook(void *wrapcxt, OUT void **user_data) {
  void *mod = dr_load_aux_library("output/libtest.so", NULL, NULL);

  int (*libtest_main)() =
      (void *)dr_lookup_aux_library_routine(mod, "libtest_main");

  libtest_main();
}

static void event_module_load(
    void *drcontext, const module_data_t *mod, bool loaded) {
  size_t offset = -1;
  drsym_error_t error = drsym_lookup_symbol(
      mod->full_path, "func_for_hook", &offset, DRSYM_DEMANGLE);
  if (offset != -1 && error == DRSYM_SUCCESS &&
      drwrap_wrap(mod->start + offset, pre_func_for_hook, NULL)) {
    dr_printf("Successfully wrapped 'func_for_hook' function.\n");
  }
}

static void event_exit(void) { drmgr_exit(); }

DR_EXPORT void dr_client_main(client_id_t id, int argc, const char *argv[]) {
  drwrap_init();
  drsym_init(0);
  drmgr_init();

  drmgr_register_module_load_event(event_module_load);
  dr_register_exit_event(event_exit);
}

My app's source is:

#include <stdio.h>
#include <unistd.h>

int func_for_hook() { return 0; }

int main() {
  printf("[app]: starting...\n");
  func_for_hook();
  return 0;
}

My libtest is:

#include <stdio.h>

// Uncommenting these two lines will expose crash earlier
// _Thread_local int bigarray[1024 + 128 + 16 + 8 + 1];
// _Thread_local int local = 4578;

int libtest_main() {
  printf("Solely invoking printf is ok\n");
  return printf("Obtaining return value is not ok\n");
}

My makefile is:

DYNAMORIO_DIR ?= /usr/lib/DynamoRIO

CC := gcc
LIB_SCRH_DIRS != gcc -print-search-dirs | grep libraries | sed 's/libraries: =/-L/' | sed 's/:/ -L/g'
CFLAGS := -DLINUX -DX86_64
CFLAGS += -fPIC -shared
CFLAGS += -I$(DYNAMORIO_DIR)/include
CFLAGS += -I$(DYNAMORIO_DIR)/ext/include
CFLAGS += -I$(DYNAMORIO_DIR)/drmemory/drmf/include
CFLAGS += -L$(DYNAMORIO_DIR)/lib64/release
CFLAGS += -L$(DYNAMORIO_DIR)/ext/lib64/release
CFLAGS += $(LIB_SCRH_DIRS)
CFLAGS += -ldynamorio
CFLAGS += -ldrwrap -ldrmgr -ldrsyms -ldrreg -ldrx

ODIR     := output
SO_FILE  := $(ODIR)/client.so
LIB_TEST := $(ODIR)/libtest.so

all: $(SO_FILE)

$(SO_FILE): client.c $(LIB_TEST) Makefile
	@mkdir -p $(@D)
	@$(CC) $< $(CFLAGS) -o $@

$(LIB_TEST): libtest.c
	@mkdir -p $(@D)
	@gcc $< -fPIC -shared -o $@

test: $(SO_FILE) $(LIB_TEST)
	@gcc -L$(abspath $(ODIR)) -ltest app.c -o $(ODIR)/app
	@$(DYNAMORIO_DIR)/bin64/drrun -c $< -- $(ODIR)/app

clean:
	@rm -rf $(ODIR)

Let's assume that the DYNAMORIO_DIR variable in the Makefile has been correctly set to the relevant path.

To reproduce the crash, place all the files in the same directory and execute make test.

To simplify the reproduction process, I have packaged these files into an attachment reproduce.zip. Please find the attached file for your convenience.

Expected behavior

The libtest_main is expected to normally return.

Screenshots or Pasted Text
Output from my pc:

$ make test
Successfully wrapped 'func_for_hook' function.
[app]: starting...
Solely invoking printf is ok
<Application /home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/app (50899).  Tool internal crash at PC 0x00007f4ee3e0d1d3.  Please report this at your tool's issue tracker.  Program aborted.
Received SIGSEGV at pc 0x00007f4ee3e0d1d3 in thread 50899
Base: 0x00007f4ee4204000
Registers:eax=0x0000000000000000 ebx=0x00000000fbad0084 ecx=0x0000000000000000 edx=0x00007f4ca02abc40
        esi=0x0000000000000025 edi=0x00007f4ca02ab7c0 esp=0x00007f4ca02ab6c8 ebp=0x00007f4ee3f96780
        r8 =0x0000000000000000 r9 =0x00007f4ca10e4768 r10=0x00007f4ea03e4020 r11=0x0000000000000246
        r12=0x00007f4ea02cd3d8 r13=0x0000000000000000 r14=0x0000000000000000 r15=0x00007f4ca02abd80
        eflags=0x0000000000010246
version 10.0.0, custom build
-no_dynamic_options -client_lib '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -client_lib64 '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -e
0x00007f4ee3f96780 0x00007f4ca10e5a40>
make: *** [Makefile:32: test] Error 255

Versions

• I am running version 10.0.0 (64-bit) on Ubuntu 22.04 (x86-64 architecture).

Additional context
Add any other context about the problem here.

[derekbruening]

• The standard first step is to run debug build (-debug) (see https://dynamorio.org/page_debugging.html). Does it report asserts or useful messages?
• Does it work if the aux library is loaded at init time instead of in the pre-wrap function?
• The next step would be to attach a debugger at the SIGSEGV and get a symbolized callstack

[wierton]

• The standard first step is to run debug build (-debug) (see https://dynamorio.org/page_debugging.html). Does it report asserts or useful messages?
• Does it work if the aux library is loaded at init time instead of in the pre-wrap function?
• The next step would be to attach a debugger at the SIGSEGV and get a symbolized callstack

Hi~, I just tried your mentioned three steps.

(It should be noted that defining a large thread-local array inside the auxiliary library can also cause a crash, even without using printf.)

With -debug option enabled, I didn't find any assertion failure, the full output is shown below:

<Starting application /home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/app (61734)>
<Initial options = -no_dynamic_options -client_lib '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -client_lib64 '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so' 0x00007f648bf561c0
add-symbol-file '/usr/lib/DynamoRIO/lib64/debug/libdynamorio.so' 0x00007f64cffa9000
add-symbol-file '/usr/lib/DynamoRIO/ext/lib64/debug/libdrwrap.so' 0x00007f648bf9b7d0
add-symbol-file '/usr/lib/DynamoRIO/ext/lib64/debug/libdrmgr.so' 0x00007f648bfaecb0
add-symbol-file '/usr/lib/DynamoRIO/ext/lib64/debug/libdrsyms.so' 0x00007f648bfc9000
add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007f64cfaf7700
add-symbol-file '/lib64/ld-linux-x86-64.so.2' 0x00007f64cfedb090
>
<(1+x) Handling our fault in a TRY at 0x00007f64d02130d9>
Successfully wrapped 'func_for_hook' function.
<spurious rep/repne prefix @0x00007f64cff37030 (f3 0f 1e fa): >
<curiosity: rex.w on OPSZ_6_irex10_short4!>
[app]: starting...
<Paste into GDB to debug DynamoRIO clients:
add-symbol-file './output/libtest.so' 0x00007f648c14d080
>
<Application /home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/app (61734).  Tool internal crash at PC 0x00007f64cfb5f9ff.  Please report this at your tool's issue tracker.  Program aborted.
Received SIGSEGV at pc 0x00007f64cfb5f9ff in thread 61734
Base: 0x00007f64cff5d000
Registers:eax=0x0000000000000000 ebx=0x000000000000001d ecx=0x00007f628bfef700 edx=0x000000000000001d
        esi=0x00007f628cd62e90 edi=0x0000000000000001 esp=0x00007f628bffec08 ebp=0x00007f628cd62e90
        r8 =0x0000000000000000 r9 =0x0000000000000001 r10=0x0000000000001000 r11=0x0000000000000246
        r12=0x000000000000001d r13=0x00007f64cfce9780 r14=0x00007f64cfce5600 r15=0x00007f64cfce4a00
        eflags=0x0000000000010246
version 10.0.0, custom build
-no_dynamic_options -client_lib '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -client_lib64 '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -e
0x00007f628cd62e90 0x20676e696b6f766e>
make: *** [Makefile:32: test] Error 255

I also tried loading the library in event_module_load, the crash still happens.

And, for gdb, the full interactive output is pasted below:

$ gdb -q --args /usr/lib/DynamoRIO/bin64/drrun -c output/client.so -- output/app
Reading symbols from /usr/lib/DynamoRIO/bin64/drrun...
Reading symbols from /usr/lib/DynamoRIO/bin64/drrun.debug...
(gdb) r
Starting program: /usr/lib/DynamoRIO/bin64/drrun -c output/client.so -- output/app
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
process 63391 is executing new program: /usr/lib/DynamoRIO/lib64/release/libdynamorio.so
warning: File "/usr/lib/DynamoRIO/lib64/release/libdynamorio.so-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /usr/lib/DynamoRIO/lib64/release/libdynamorio.so-gdb.py
line to your configuration file "/home/wierton/.config/gdb/gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/wierton/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"

Program received signal SIGILL, Illegal instruction.
0x00007ffff7f18a49 in syscall_ready ()
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7eef3d3 in safe_read_asm_mid ()
(gdb) bt
#0  0x00007ffff7eef3d3 in safe_read_asm_mid ()
#1  0x00007ffff7eeeb8f in safe_read_fast (base=0xffffffffff600000, size=size@entry=64, out_buf=out_buf@entry=0x7fffffffc310,
    bytes_read=bytes_read@entry=0x0) at /home/runner/work/dynamorio/dynamorio/core/arch/x86_code.c:431
#2  0x00007ffff7ef7d47 in safe_read_ex (bytes_read=0x0, out_buf=<optimized out>, size=64, base=<optimized out>)
    at /home/runner/work/dynamorio/dynamorio/core/unix/os.c:4847
#3  0x00007ffff7f1a13d in is_elf_so_header_common (memory=true, size=<optimized out>, base=<optimized out>)
    at /home/runner/work/dynamorio/dynamorio/core/drlibc/drlibc_module_elf.c:79
#4  is_elf_so_header (base=<optimized out>, size=<optimized out>)
    at /home/runner/work/dynamorio/dynamorio/core/drlibc/drlibc_module_elf.c:150
#5  0x00007ffff7f12029 in module_is_header (base=<optimized out>, size=<optimized out>)
    at /home/runner/work/dynamorio/dynamorio/core/unix/module_elf.c:579
#6  0x00007ffff7ef9874 in os_walk_address_space (iter=iter@entry=0x7fffffffc3d0, add_modules=add_modules@entry=true)
    at /home/runner/work/dynamorio/dynamorio/core/unix/os.c:9937
#7  0x00007ffff7ef9b9d in find_executable_vm_areas () at /home/runner/work/dynamorio/dynamorio/core/unix/os.c:10099
#8  0x00007ffff7e6b800 in vm_areas_init () at /home/runner/work/dynamorio/dynamorio/core/vmareas.c:1675
#9  0x00007ffff7e30175 in dynamorio_app_init_part_two_finalize () at /home/runner/work/dynamorio/dynamorio/core/dynamo.c:683
#10 0x00007ffff7e30212 in dynamorio_app_init_part_two_finalize () at /home/runner/work/dynamorio/dynamorio/core/dynamo.c:509
#11 0x00007ffff7f0eab2 in privload_early_inject (sp=0x7fffffffd420, old_libdr_base=<optimized out>,
    old_libdr_size=<optimized out>) at /home/runner/work/dynamorio/dynamorio/core/unix/loader.c:2255
#12 0x00007ffff7eeeede in reloaded_xfer ()
#13 0x0000000000000001 in ?? ()
#14 0x00007fffffffd934 in ?? ()
#15 0x0000000000000000 in ?? ()
(gdb) c
Continuing.
Successfully wrapped 'func_for_hook' function.
[app]: starting...
Solely invoking printf is ok

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff79e01d3 in ?? ()
(gdb) bt
#0  0x00007ffff79e01d3 in ?? ()
#1  0x00007ffff79c4534 in ?? ()
#2  0x0000000000000002 in ?? ()
#3  0x00007fffb3fd7020 in ?? ()
#4  0x0000003000000008 in ?? ()
#5  0x00007ffdb3e7ed20 in ?? ()
#6  0x00007ffdb3e7ec40 in ?? ()
#7  0x00007ffff7e954d9 in free_va_opnd_list (dcontext=0x7ffff7b69780, args=0x0, num_args=<optimized out>)
    at /home/runner/work/dynamorio/dynamorio/core/lib/instrument.c:5045
#8  dr_insert_clean_call_ex (drcontext=0x7ffff7b69780, ilist=0x7fffb3eac3d8, where=0x7ffdb3e7ed80, callee=0x7fffb3e1d0d0,
    save_flags=(DR_CLEANCALL_READS_APP_CONTEXT | DR_CLEANCALL_WRITES_APP_CONTEXT), num_args=4222419076)
    at /home/runner/work/dynamorio/dynamorio/core/lib/instrument.c:5372
#9  0xffffffffb3e1cc1f in ?? ()
#10 0x00007ffff3dce157 in ?? ()
#11 0xffffffffb3e87ba0 in ?? ()
#12 0x00007ffff79b1050 in ?? ()
#13 0x00007ffff7b69780 in ?? ()
#14 0x00007ffff3dce149 in ?? ()
#15 0x0000000000000000 in ?? ()
(gdb) c
Continuing.
<Application /home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/app (63391).  Tool internal crash at PC 0x00007ffff79e01d3.  Please report this at your tool's issue tracker.  Program aborted.
Received SIGSEGV at pc 0x00007ffff79e01d3 in thread 63391
Base: 0x00007ffff7ddd000
Registers:eax=0x0000000000000000 ebx=0x00000000fbad0084 ecx=0x0000000000000000 edx=0x00007ffdb3e7ec40
        esi=0x0000000000000025 edi=0x00007ffdb3e7e7c0 esp=0x00007ffdb3e7e6c8 ebp=0x00007ffff7b69780
        r8 =0x0000000000000000 r9 =0x0000000000000001 r10=0x00007fffb3fd7020 r11=0x0000000000000246
        r12=0x00007fffb3eac3d8 r13=0x0000000000000000 r14=0x0000000000000000 r15=0x00007ffdb3e7ed80
        eflags=0x0000000000010246
version 10.0.0, custom build
-no_dynamic_options -client_lib '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -client_lib64 '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -e
0x00007ffff7b69780 0x00007ffdb4d02fb0>
[Inferior 1 (process 63391) exited with code 0377]

[derekbruening]

Paste the lines in the message with the header Paste into GDB to debug DynamoRIO client to load the symbols into gdb (see https://dynamorio.org/page_debugging.html#autotoc_md142)

[wierton]

Paste the lines in the message with the header Paste into GDB to debug DynamoRIO client to load the symbols into gdb (see https://dynamorio.org/page_debugging.html#autotoc_md142)

Thanks for your guides, I tried your suggestions and get this stack dump:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff775b9ff in __GI___pthread_enable_asynccancel () at ./nptl/cancellation.c:34
34      ./nptl/cancellation.c: No such file or directory.
(gdb) bt
#0  0x00007ffff775b9ff in __GI___pthread_enable_asynccancel () at ./nptl/cancellation.c:34
#1  0x00007ffff77df6b7 in __GI___libc_write (nbytes=29, buf=0x7ffdb49ae410, fd=1) at ../sysdeps/unix/sysv/linux/write.c:26
#2  __GI___libc_write (fd=1, buf=0x7ffdb49ae410, nbytes=29) at ../sysdeps/unix/sysv/linux/write.c:24
#3  0x00007ffff7755eed in _IO_new_file_write (f=0x7ffff78e5780, data=0x7ffdb49ae410, n=29) at ./libio/fileops.c:1180
#4  0x00007ffff77579e1 in new_do_write (to_do=29,
    data=0x7ffdb49ae410 "Solely invoking printf is ok\n", '\315' <repeats 170 times>, <incomplete sequence \315>...,
    fp=0x7ffff78e5780) at ./libio/libioP.h:947
#5  _IO_new_do_write (to_do=29,
    data=0x7ffdb49ae410 "Solely invoking printf is ok\n", '\315' <repeats 170 times>, <incomplete sequence \315>...,
    fp=0x7ffff78e5780) at ./libio/fileops.c:425
#6  _IO_new_do_write (fp=fp@entry=0x7ffff78e5780,
    data=0x7ffdb49ae410 "Solely invoking printf is ok\n", '\315' <repeats 170 times>, <incomplete sequence \315>..., to_do=29)
    at ./libio/fileops.c:422
#7  0x00007ffff7757ec3 in _IO_new_file_overflow (f=0x7ffff78e5780, ch=10) at ./libio/fileops.c:783
#8  0x00007ffff774bfaa in __GI__IO_puts (str=0x7fffb3d4a000 "Solely invoking printf is ok") at ./libio/ioputs.c:41
#9  0x00007fffb3d49150 in libtest_main () at libtest.c:8
#10 0x00007fffb3b522cf in pre_func_for_hook (wrapcxt=0x7ffdb3bfad60, user_data=0x7fffb3cc90d8) at client.c:22
#11 0x00007fffb3b9b812 in drwrap_in_callee (arg1=0x7ffff3b4a149, xsp=140737488343816)
    at /home/runner/work/dynamorio/dynamorio/ext/drwrap/drwrap.c:2113
#12 0x00007fffb3d390ad in ?? ()
#13 0x00007ffff6a60a70 in ?? ()
#14 0x0000000000000001 in ?? ()
#15 0x00007fffffffd310 in ?? ()
#16 0x00007fffffffd308 in ?? ()
#17 0x0000000000000000 in ?? ()
(gdb) x/4i 0x00007ffff775b9ff
=> 0x7ffff775b9ff <__GI___pthread_enable_asynccancel+15>:       mov    %fs:0x972,%al
   0x7ffff775ba07 <__GI___pthread_enable_asynccancel+23>:       movzbl %al,%eax
   0x7ffff775ba0a <__GI___pthread_enable_asynccancel+26>:       movb   $0x1,%fs:0x972
   0x7ffff775ba13 <__GI___pthread_enable_asynccancel+35>:       mov    %fs:0x308,%edx
(gdb) c
Continuing.
<Application /home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/app (86641).  Tool internal crash at PC 0x00007ffff775b9ff.  Please report this at your tool's issue tracker.  Program aborted.
Received SIGSEGV at pc 0x00007ffff775b9ff in thread 86641
Base: 0x00007ffff7b65000
Registers:eax=0x0000000000000000 ebx=0x000000000000001d ecx=0x00007ffdb3beb700 edx=0x000000000000001d
        esi=0x00007ffdb49ae410 edi=0x0000000000000001 esp=0x00007ffdb3bfac08 ebp=0x00007ffdb49ae410
        r8 =0x0000000000000000 r9 =0x00007fffb3d482f4 r10=0x0000000000001000 r11=0x0000000000000202
        r12=0x000000000000001d r13=0x00007ffff78e5780 r14=0x00007ffff78e1600 r15=0x00007ffff78e0a00
        eflags=0x0000000000010246
version 10.0.0, custom build
-no_dynamic_options -client_lib '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -client_lib64 '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -e
0x00007ffdb49ae410 0x20676e696b6f766e>
[Inferior 1 (process 86641) exited with code 0377]
(gdb)

Source code at frame #9 and #10 is as below:

#9:

     7| int libtest_main() {
-->  8|   printf("Solely invoking printf is ok\n");
     9|   return printf("Obtaining return value is not ok\n");
    10| }

#10:

    16| static void pre_func_for_hook(void *wrapcxt, OUT void **user_data) {
    17|   void *mod = dr_load_aux_library("output/libtest.so", NULL, NULL);
    18| 
    19|   int (*libtest_main)() =
    20|       (void *)dr_lookup_aux_library_routine(mod, "libtest_main");
    21| 
--> 22|   libtest_main();
    23| }

[derekbruening]

What is the glibc version? Is it > 2.36? I'm afraid this is the continuation of #5437 where the glibc developers added hidden undocumented interfaces which break DynamoRIO's private loader. Workarounds were put in for glibc 2.34, 2.35, and 2.36: see e.g. #5902. But this glibc decision to not keep libraries separate with clean interfaces puts the whole private loader approach on shaky footing and means there have to be hacky workarounds every time glibc changes something. The long-term support for the private loader, and being able to just load some library that calls regular functions like printf like you did here, is unclear -- see discussion in #5437.

[wierton]

What is the glibc version? Is it > 2.36? I'm afraid this is the continuation of #5437 where the glibc developers added hidden undocumented interfaces which break DynamoRIO's private loader. Workarounds were put in for glibc 2.34, 2.35, and 2.36: see e.g. #5902. But this glibc decision to not keep libraries separate with clean interfaces puts the whole private loader approach on shaky footing and means there have to be hacky workarounds every time glibc changes something. The long-term support for the private loader, and being able to just load some library that calls regular functions like printf like you did here, is unclear -- see discussion in #5437.

My glibc version is 2.35, and I have some new observations about this crash.

Introducing large thread-local variables into libtest.so also causes DynamoRIO to crash, even without invoking printf. However, in this instance, DynamoRIO captures the crash and generates an assertion failure message (Internal Error: DynamoRIO debug check failure: ...:308 !dynamo_initialized), which I have pasted below:

(gdb) c
Continuing.
[app]: starting...
<Paste into GDB to debug DynamoRIO clients:
add-symbol-file './output/libtest.so' 0x00007fffb3d49040
>
Executing gdb command: add-symbol-file './output/libtest.so' 0x7fffb3d49040
<Application /home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/app (65864).  Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/unix/loader_linux.c:308 !dynamo_initialized
(Error occurred @2728 frags in tid 65864)
version 10.0.0, custom build
-no_dynamic_options -client_lib '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -client_lib64 '/home/wierton/research_playground/libcpp/DynamoRIO/patch-crash.reproduce/output/client.so;0;' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -e
...
[Inferior 1 (process 65864) exited with code 0377]

My modified libtest.c is presented below:

#include <stdio.h>

_Thread_local int bigarray[1024 + 128 + 16 + 8 + 1];
_Thread_local int local = 4578;

int libtest_main() {
  return 0;
}

Another observation worth noting:

In the stack dump presented in the previous response, the instruction which cause segfault is:

=> 0x7ffff775b9ff <__GI___pthread_enable_asynccancel+15>:       mov    %fs:0x972,%al

When I manually insert an inline assembly with the instruction into libtest_main using offset 0x972, the crash is reproducible. However, reducing the offset to a smaller value, like 0x1, won't triggers the crash.

I am uncertain whether these two crashes stem from the same underlying issue, but I hope this information will be helpful.

[derekbruening]

Given that, as explained in the referenced #5437, the long-term future of private loader support in DR is in doubt due to glibc changes making it difficult to keep maintaining that support, I'm not sure there are developers who are eager to spend time on corner cases of private libraries that are not impacting the regression tests (or their own primary projects). Also, you're referencing some pthreads symbols, and note that as documented private pthreads code has never been officially supported (part of the headache with 2.34+). If you'd like to work on support for the private loader issues you are hitting on glibc 2.35 (note that what you are doing may well work fine on versions < 2.34) such contributions are welcome. If it's like the other #5437 issues, there may be missing undocumented initialization calls or other pieces (see those PR's); you would have to study how these libraries work normally and compare to the private version.