Emscripten crashes when using thread_local variables wihtout -pthread and -flto enabled

[yosoymin]

Hello, I'm having problems with thread_local variables when the build is performed without -pthread and with -flto enabled.

I've noticed that when the runtime arrives at a thread_local variable, there is a crash. It seems like the variable was declared as a temporal variable (if inside a function scope) instead of a static variable. For global variables, it seems like the initialization value is not guaranteed to be 0.

With -pthread compile and link flag, the problem goes away, but I need a specific build without pthreads to support environments that don't support SharedArrayBuffer due to CORS problems.
The same thing happens if we remove the -flto compile and link flag, but we want to have the smallest possible binary.

As a workaround, I've defined a custom symbol to mark thread_local variables in our code. Something like this:

#if defined(__EMSCRIPTEN_PTHREADS__)
    #define au_thread_local thread_local
#else
    #define au_thread_local
#endif

But for other third-party libraries, or even Emscripten itself, this is not possible and the workaround is more hacky. In particular, Emscripten uses thread_local variables in <emscripten/val.h> and <emscripten/bind.h> header files. If you try to use coroutines, then you will have a crash inside the <emscripten/val.h> file:

template<EM_METHOD_CALLER_KIND Kind, typename ReturnType, typename... Args>
struct Signature {
  /*
  typedef typename BindingType<ReturnType>::WireType (*MethodCaller)(
      EM_VAL object,
      EM_VAL method,
      EM_DESTRUCTORS* destructors,
      typename BindingType<Args>::WireType...);
  */
  static EM_METHOD_CALLER get_method_caller() {
    static constexpr WithPolicies<>::ArgTypeList<ReturnType, Args...> args;
    thread_local EM_METHOD_CALLER mc = _emval_get_method_caller(args.getCount(), args.getTypes(), Kind);
    return mc;
  }
};

For this specific case, this is the workaround that is working for me. Instead of including directly the header <emscripten/val.h> I have to include a custom file called "emscriptenVal.h" with the next content:

#if !defined(__CAN_USE_THREADS)
	#pragma clang diagnostic push
	#pragma clang diagnostic ignored "-Wkeyword-macro"
	// In Emscripten if we activate LTO then the NoThreads build fails with thread_local variables treating them as
	// local temporal variables, instead of static variables
	#define thread_local static
#endif
#include <emscripten/val.h>
#if !defined(__CAN_USE_THREADS)
	#undef thread_local
	#pragma clang diagnostic pop
#endif

Version of emscripten/emsdk:
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.68 (ceee49d)
clang version 20.0.0git (https:/github.com/llvm/llvm-project 5cc64bf60bc04b9315de3c679eb753de4d554a8a)
Target: wasm32-unknown-emscripten
Thread model: posix

Full link command and output with -v appended:
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/upstream/bin\clang.exe" --version
"D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\tools\file_packager.bat" D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.data --from-emcc --embed D:/Code/Nebula/Nebula-Viewer/Code/Src/GoldenViewer/../../../_Output/desktop/chrome.gtpak@/chrome.gtpak D:/Code/Nebula/Nebula-Viewer/Code/Src/GoldenViewer/../../../_Output/scripts/chrome_scripts.gtpak@/chrome_scripts.gtpak --no-node --obj-output=C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\embedded_files.o
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/upstream/bin\wasm-ld.exe" -o D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.wasm -lembind-rtti Src/GoldenViewer/CMakeFiles/GoldenViewer.dir/Distribution/src/GenericEntryPoint.cpp.o Src/GoldenViewerLib/Distribution/libGoldenViewerLib.a Libs/GameEngine/Distribution/libGameEngine.a Libs/Aurora/Code/Libs/bullet/Distribution/libbullet.a clip_external-prefix/src/clip_external-build/Distribution/libclip.a Libs/Aurora/Code/Src/auVideo/Distribution/libauVideo.a Libs/Aurora/Code/Libs/freetype/Distribution/libfreetype.a ktx_read_external-prefix/src/ktx_read_external-build/Distribution/libktx_read.a Libs/Aurora/Code/Libs/imgui/Distribution/libimgui.a Libs/Aurora/Code/Src/auGeom/Distribution/libauGeom.a Libs/Aurora/Code/Src/auImage/Distribution/libauImage.a png_static_external-prefix/src/png_static_external-build/Distribution/libpng16.a webp_external-prefix/src/webp_external-build/Distribution/libwebp.a webp_external-prefix/src/webp_external-build/Distribution/libwebpdecoder.a webp_external-prefix/src/webp_external-build/Distribution/libsharpyuv.a avif_external-prefix/src/avif_external-build/Distribution/libavif.a aom_external-prefix/src/aom_external-build/Distribution/libaom.a yuv_external-prefix/src/yuv_external-build/Distribution/libyuv.a Libs/Aurora/Code/Src/auSound/Distribution/libauSound.a Libs/Aurora/Code/Libs/libvorbis-1.3.7/Distribution/liblibvorbis-1.3.7.a Libs/Aurora/Code/Src/auNetwork/Distribution/libauNetwork.a Libs/Aurora/Code/Src/auCore/Distribution/libauCore.a fmt_external-prefix/src/fmt_external-build/Distribution/libfmt.a zlibstatic_external-prefix/src/zlibstatic_external-build/Distribution/libz.a Libs/Aurora/Code/Libs/yajl/Distribution/libyajl.a Libs/Aurora/Code/Libs/lua/Distribution/liblua_static.a -LD:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\cache\sysroot\lib\wasm32-emscripten\lto -lfetch C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\embedded_files.o D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\cache\sysroot\lib\wasm32-emscripten\lto\libSDL2.a -lGL-webgl2-getprocaddr -lal -lhtml5 -lc_optz -lbulkmemory -lstubs -lc -ldlmalloc -lcompiler_rt -lc++-noexcept -lc++abi-noexcept -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr C:\Users\yosoy\AppData\Local\Temp\tmpn2i23s1dlibemscripten_js_symbols.so --strip-debug --export=_emscripten_stack_alloc --export=__getTypeName --export=__funcs_on_exit --export=__wasm_call_ctors --export=emscripten_stack_get_current --export=_emscripten_stack_restore --export=malloc --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=main --export-if-defined=__main_argc_argv --export-if-defined=fflush
--export-table -z stack-size=131072 --max-memory=1073741824 --initial-memory=67108864 --no-entry --table-base=1 --global-base=1024
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/upstream/bin\llvm-objcopy.exe" D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.wasm D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.wasm --remove-section=.debug* --remove-section=producers
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/node/20.18.0_64bit/bin/node.exe" D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\src\compiler.mjs C:\Users\yosoy\AppData\Local\Temp\tmp5blpl22o.json
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/node/20.18.0_64bit/bin/node.exe" D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\src\compiler.mjs C:\Users\yosoy\AppData\Local\Temp\tmp9w4a_2d5.json
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/node/20.18.0_64bit/bin/node.exe" C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\tsgen.js
"D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\node_modules.bin\tsc.cmd" --outFile C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\jsdoc.d.ts --declaration --emitDeclarationOnly --allowJs C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\jsdoc.js
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/upstream\bin\wasm-opt" --strip-target-features --post-emscripten -Oz --low-memory-unused --zero-filled-memory --pass-arg=directize-initial-contents-immutable --no-stack-ir D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.wasm -o D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.wasm --mvp-features --enable-threads --enable-bulk-memory --enable-multivalue --enable-mutable-globals --enable-reference-types --enable-sign-ext
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/node/20.18.0_64bit/bin/node.exe" D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\tools\acorn-optimizer.mjs C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\GoldenViewerNoThreads.js AJSDCE --minify-whitespace -o C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\GoldenViewerNoThreads.jso1.js
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/node/20.18.0_64bit/bin/node.exe" D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\tools\acorn-optimizer.mjs C:\Users\yosoy\AppData\Local\Temp\emcc_acorn_info_6m5d1w8o.js emitDCEGraph --no-print
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/upstream\bin\wasm-metadce" --graph-file=C:\Users\yosoy\AppData\Local\Temp\emcc_dce_graph_vod_qa8s.json D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.wasm -o D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.wasm --mvp-features --enable-threads --enable-bulk-memory --enable-multivalue --enable-mutable-globals --enable-reference-types --enable-sign-ext
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/node/20.18.0_64bit/bin/node.exe" D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\tools\acorn-optimizer.mjs C:\Users\yosoy\AppData\Local\Temp\emcc_acorn_info_ilu11rgi.js applyDCEGraphRemovals --minify-whitespace -o C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\GoldenViewerNoThreads.jso2.js
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/node/20.18.0_64bit/bin/node.exe" D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\tools\acorn-optimizer.mjs C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\GoldenViewerNoThreads.jso2.js AJSDCE --minify-whitespace -o C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\GoldenViewerNoThreads.jso3.js
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/upstream\bin\wasm-opt" --minify-imports-and-exports-and-modules --optimize-level=2 --shrink-level=2 --optimize-stack-ir D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.wasm -o D:\Code\Nebula\Nebula-Viewer_Output\html\wasm\GoldenViewerNoThreads.wasm --mvp-features --enable-threads --enable-bulk-memory --enable-multivalue --enable-mutable-globals --enable-reference-types --enable-sign-ext
"D:/Code/Nebula/Nebula-Viewer/Code/Libs/SDKs/emsdk/node/20.18.0_64bit/bin/node.exe" D:\Code\Nebula\Nebula-Viewer\Code\Libs\SDKs\emsdk\upstream\emscripten\tools\acorn-optimizer.mjs C:\Users\yosoy\AppData\Local\Temp\emcc_acorn_info_y_vvsusf.js applyImportAndExportNameChanges --minify-whitespace -o C:\Users\yosoy\AppData\Local\Temp\emscripten_temp_dylnnf24\GoldenViewerNoThreads.jso4.js

[kripken]

I don't see an issue in a trivial attempt to reproduce this:

thread_local int x = 10;

int main(int argc, char **argv) {
  x += argc;
  return x;
}

$ ./emcc a.cpp -flto -O3
$ nodejs a.out.js
$ echo $?
11

@yosoymin Can you create a standalone testcase that shows the issue?

[sbc100]

I also tried with a more complex example that used thread_local at the function scope:

$ cat test.cc
#include <stdio.h>                                                               
#include <string.h>                                                                 
#include <emscripten/stack.h>                                                       
                                                                                 
// set the entire stack region to 0xff                                           
void set_stack() {                                                               
  size_t stack_size = emscripten_stack_get_base() - emscripten_stack_get_end();  
  void* stack_low = (void*)emscripten_stack_get_end();                           
  memset(stack_low, 0xff, stack_size);                                           
}                                                                                
                                                                                 
int test(int add) {                                                              
  char foo[10];                                                                  
  thread_local int x;                                                            
  x += add;                                                                      
  printf("x => %d\n", x);                                                        
  printf("foo => %p\n", foo);                                                    
  printf("foo[0] => %x\n", foo[0]);                                              
  return x;                                                                      
}                                                                                
                                                                                 
int main(int argc, char **argv) {                                                
  test(argc);                                                                    
                                                                                 
  set_stack();                                                                   
  test(argc);                                                                    
                                                                                 
  set_stack();                                                                   
  test(argc);                                                                    
  return 0;                                                                      
} 
$ ./em++ -flto -O2 test.cc && node ./a.out.js 1 2 3 
x => 4
foo => 0x10bc6
foo[0] => 0
x => 8
foo => 0x10bc6
foo[0] => ffffffff
x => 12
foo => 0x10bc6
foo[0] => ffffffff

Here you can see that x is incremented from 0, by 4 each time test is called, and that x is not stored on the stack since (unlike foo) it is not clobbered by set_stack which sets the entire stack region to 0xff.

[yosoymin]

While reducing the code as much as possible I noticed that the only way to reproduce the error was to link with an external library. After that, I figured out the problem. I was setting -pthread compiler flag for all static libraries but compiling and linking the final executable without that flag.

[sbc100]

While reducing the code as much as possible I noticed that the only way to reproduce the error was to link with an external library. After that, I figured out the problem. I was setting -pthread compiler flag for all static libraries but compiling and linking the final executable without that flag.

Doing that (compiling with -pthread and linking without) should still work and produce perfectly valid programs that just run without any threads. If that doesn't work that would be considered a bug.

[yosoymin]

This small project reproduces the error. I have pointed out the key lines that cause this error to be reproduced, with any of them that are modified, the error disappears.
EmscriptenThreadLocalCrash.zip

In my complete project I have not been able to get the thread_local to work properly, not even being consistent with the -pthread flags :(. I hope this can be helpful.

[yosoymin]

Finally, I found the problem in my project: the library libwebp, which enables pthreads by default.
So, to summarize, runtime crashes occur if you use thread_local variables in a project with any static library using the -pthread flag, but you don't link the final executable with -pthread, enable -flto globally, and you also use --embed-file.

[sbc100]

So presumably we should be able to make the simple program in #22880 (comment) crash?

I can't seem to make it crash with or without using -pthread at link time.

[sbc100]

I updated my test program so that it asserts if the thread_local variable lives on the stack or get clobbered when I clobber the stack:

#include <assert.h>
#include <stdio.h>
#include <string.h>
#include <emscripten/stack.h>

// set the entire stack region to 0xff
void set_stack() {
  size_t stack_size = emscripten_stack_get_base() - emscripten_stack_get_end();
  void* stack_low = (void*)emscripten_stack_get_end();
  printf("stack: [%lx] %p -> %lx\n", emscripten_stack_get_current(), stack_low, emscripten_stack_get_base());
  memset(stack_low, 0xff, stack_size);
}

int test(int add, int expected) {
  char foo[10];
  thread_local int x;
  printf("foo => %p x => %p\n", foo, &x);
  printf("x => %d, expected => %d\n", x, expected);
  assert(x == expected);
  printf("foo[0] => %x\n", foo[0]);
  x += add;
  return x;
}

int main(int argc, char **argv) {
  int expected = 0;
  test(argc, expected);
  expected += argc;

  set_stack();
  test(argc, expected);
  expected += argc;

  set_stack();
  test(argc, expected);
  expected += argc;
  return 0;
}
$ ./em++ test.cc -O2 -flto -pthread -c && ./em++ -sASSERTIONS=0 test.o -flto --embed-file README.md && node ./a.out.js x y z
foo => 0xff66 x => 0x20000
x => 0, expected => 0
foo[0] => 0
stack: [feb0] 0 -> 10000
foo => 0xff66 x => 0x20000
x => 4, expected => 4
foo[0] => ffffffff
stack: [feb0] 0 -> 10000
foo => 0xff66 x => 0x20000
x => 8, expected => 8
foo[0] => ffffffff

You can see the TLS variable doesn't live in the stack range (it lives at 0x20000 and the stack 0x0-0x10000)

[yosoymin]

I've only been able to reproduce the error if I link with an static library that is compiled with -pthread and the final executable is not. Please, use the code I uploaded in the comment #22880 (comment). It is very sensible to modifications and the crash goes away, I recommend to use the supplied CMakeLists.txt file in order to reproduce.

[yosoymin]

I've simplified the code reproducing the error. No CMake, Ninja, or static library is needed. And I've uploaded to this Github repo: https://github.com/yosoymin/EmscriptenThreadLocalCrash

[sbc100]

Thanks for the simpler repro @yosoymin.

BTW do you know you can run/test this kind of stuff under node. My simplified setup:

$ cat build.sh
$ cat build.sh 
em++ -O1 -flto -pthread -c simpleLib.cpp
em++ -O1 -flto -c main.cpp
em++ -O1 -flto --embed-file=main.cpp@/ simpleLib.o main.o
$ ./build.sh
$ node ./a.out.js 
start main
wasm://wasm/00094376:1


RuntimeError: memory access out of bounds
    at wasm://wasm/00094376:wasm-function[68]:0x5d04
    at Module._main (/usr/local/google/home/sbc/dev/wasm/emscripten/EmscriptenThreadLocalCrash/a.out.js:3709:102)
    at callMain (/usr/local/google/home/sbc/dev/wasm/emscripten/EmscriptenThreadLocalCrash/a.out.js:3760:15)
    at doRun (/usr/local/google/home/sbc/dev/wasm/emscripten/EmscriptenThreadLocalCrash/a.out.js:3799:23)
    at run (/usr/local/google/home/sbc/dev/wasm/emscripten/EmscriptenThreadLocalCrash/a.out.js:3812:5)
    at runCaller (/usr/local/google/home/sbc/dev/wasm/emscripten/EmscriptenThreadLocalCrash/a.out.js:3739:19)
    at removeRunDependency (/usr/local/google/home/sbc/dev/wasm/emscripten/EmscriptenThreadLocalCrash/a.out.js:417:7)
    at receiveInstance (/usr/local/google/home/sbc/dev/wasm/emscripten/EmscriptenThreadLocalCrash/a.out.js:585:5)
    at receiveInstantiationResult (/usr/local/google/home/sbc/dev/wasm/emscripten/EmscriptenThreadLocalCrash/a.out.js:597:5)

Node.js v18.20.3

[sbc100]

I was able to narrow the test down more. Still fails even with thread_local at top level and without the getTest() helper:

$ cat main.cpp
#include <cstdio>

int simpleFunction();

struct Test {
  int* data = nullptr;
  void setValue(int v) {
    if (!data)
      data = new int();
    *data = v;
  }
};

thread_local Test test;

int main(int, char**) {
  printf("start main\n");
  // The program crashes here
  test.setValue(15);
  printf("end main -> %d\n", simpleFunction());
  return 0;
}

[sbc100]

I tracked this down to a linker bug! Fix is here: llvm/llvm-project#116136