[LLD] please support MSVC ABI in GNU toolchain and support GNU ABI in MSVC toolchain

[vhyijk]

Because of the two different name mangling rules (Visual C++ and MinGW-w64) on Windows, LLD cannot link against MSVC libraries in MinGW toolchain (x86_64-w64-windows-gnu) or link against GNU libraries in MSVC toolchain (x86_64-pc-windows-msvc).
Hope to support MSVC ABI in GNU toolchain and support GNU ABI in MSVC toolchain.

[alvinhochun]

What exactly is the use case you have in mind?

[llvmbot]

@llvm/issue-subscribers-lld-coff

[vhyijk]

What exactly is the use case you have in mind?

This problem does only appear on Windows.
MSVC and GCC have different C++ name mangling rules. For example, I have a C++ function int func(int , int);. MSVC will mangle it into ?func@@YAHHH@Z but GCC will mangle it into _Z4funcii.
When I use Clang as GCC (gcc.exe), I cannot link against the libraries compiled by MSVC. When I use Clang as MSVC (cl.exe & link.exe), I cannot link against the libraries compiled by GCC, either.
On Windows, I have known LLVM supports both MinGW ABI (aka. Itanium C++ ABI) and MSVC ABI but I cannot use these two ABIs at the same time.

[alvinhochun]

When I use Clang as GCC (gcc.exe), I cannot link against the libraries compiled by MSVC. When I use Clang as MSVC (cl.exe & link.exe), I cannot link against the libraries compiled by GCC, either.

Please mind that in almost all cases you do not want to mix MSVC static libraries (objects) with MinGW, even if they are plain C. Doing so is unsupported and is most certainly going to cause issues.

Linking against MSVC dynamic libraries (DLLs) through a import lib from MinGW code (and vice versa) does work, however this applies to C linkage only. When exporting APIs that are expected to be usable from different compilers, they should be limited to C ABI and use extern "C" linkage, which also disables name mangling.

On Windows, I have known LLVM supports both MinGW ABI (aka. Itanium C++ ABI) and MSVC ABI but I cannot use these two ABIs at the same time.

You can use those two ABIs together provided that they live in separate DLLs and you don't try to pass any C++ types between them ¹. The MinGW C++ ABI is very much incompatible with the MSVC ABI, and name mangling is only a very small part of it. Making the name mangling "compatible" does not magically make them compatible. (For simple functions it may happen to work, but definitely not when C++ types starts getting involved.)

Footnotes

1. COM objects implemented in C++ can cross the MSVC and MinGW ABI boundary, but only if you treat them strictly as COM objects. ↩