spdlog segfault on github windows CI

[narnaud]

When using spdlog with Qt, and having tests (using QtTest) in a project, the tests are failing on the Windows CI, only in release mode.

I was able to reduce the problem to a minimal project here: https://github.com/narnaud/tst-release
(don't mind what is in source, that's not the most important).

Spdlog integration

Nothing fancy, just using a git submodule and adding the directory with add_subdirectory(3rdparty/spdlog)
I'm using the latest spdlog version

Test

Here again, nothing fancy, that's a usual test, that could be reduced to running those lines:

    void printSomething()
    {
        spdlog::info("Hello, {}!", "World");
        spdlog::warn("Hello, {}!", "World");
        spdlog::error("Hello, {}!", "World");
    }

Error

Here is the error I got on the CI (see https://github.com/narnaud/tst-release/actions/runs/9419197367/job/25948425419?pr=5):

 A crash occurred in D:\a\tst-release\tst-release\build-ci\bin\tst_spdlog.exe.
While testing printSomething
Function time: 0ms Total time: 1ms
Exception address: 0x00007FFE83EB3278
Exception code   : 0xc0000005
Nearby symbol    : Thrd_yield
Stack:
#  1: QTest::toString() - 0x00007FFE5C0E9620
#  2: UnhandledExceptionFilter() - 0x00007FFE8EA3F990
#  3: memset() - 0x00007FFE91326140
#  4: _C_specific_handler() - 0x00007FFE9130EBB0
#  5: _chkstk() - 0x00007FFE91324290
#  6: RtlVirtualUnwind2() - 0x00007FFE912B1510
#  7: KiUserExceptionDispatcher() - 0x00007FFE91323380
#  8: Thrd_yield() - 0x00007FFE83EB3140
#  9: spdlog::sinks::wincolor_sink<spdlog::details::console_mutex>::log() - 0x00007FF70A3DE2B0
# 10: spdlog::logger::sink_it_() - 0x00007FF70A3CF550
# 11: spdlog::logger::log_it_() - 0x00007FF70A3CCA60
# 12: spdlog::logger::log_<char const (&)[6]>() - 0x00007FF70A372EE0
# 13: TestSpdlog::qt_static_metacall() - 0x00007FF70A3719E0
# 14: QMetaMethodInvoker::invokeImpl() - 0x00007FFE5C1A3460
# 15: QMetaMethod::invokeImpl() - 0x00007FFE5C1A3280
# 16: QTest::ignoreMessage() - 0x00007FFE5C0E1960
# 17: QTest::ignoreMessage() - 0x00007FFE5C0E1960
# 18: QTest::ignoreMessage() - 0x00007FFE5C0E1960
# 19: QTest::qRun() - 0x00007FFE5C0E7030
# 20: QTest::qExec() - 0x00007FFE5C0E4910
# 21: main() - 0x00007FF70A374F30
# 22: __scrt_common_main_seh() - 0x00007FF70A3DF294
# 23: BaseThreadInitThunk() - 0x00007FFE90354CA0
# 24: RtlUserThreadStart() - 0x00007FFE912FE880

It's working nicely on Mac and Linux, it's working nicely on my local Windows machine (same compiler/Qt version) and it's working nicely in debug mode...
I'm out of ideas right now.

[tt4g]

The top of the stacktrace is QTest::toString(), so it looks like a QTest problem.
Why did you think it was an spdlog problem?

[tt4g]

I have read the documentation for the QTEST_MAIN macro used in the following location and it says it creates QApplication object.

https://github.com/narnaud/tst-release/blob/898faf93af139b3fe07ea5a6c2d092d2b93632f2/tests/tst_spdlog.cpp#L26

Although this is the case with PyQt5, there have been reports of crashes with GItHub Action when using QApplication: https://stackoverflow.com/questions/60692711/cant-create-python-qapplication-in-github-action

I think the crash is caused by Qt.

[narnaud]

The top of the stacktrace is QTest::toString(), so it looks like a QTest problem. Why did you think it was an spdlog problem?

There are different reasons for that, but maybe my conclusions are wrong:

• the link I sent is just a small test-case, on my bigger project, if I remove spdlog all tests are passing
• a normal test not using spdlog just pass (I've added one here: https://github.com/narnaud/tst-release/actions/runs/9427392246/job/25971452412?pr=5)
• the stacktrace shows an exception raised after calling a spdlog function. I agree the tip of the stacktrace is QTest::toString, but at this point QTest is trying to handle the exception gracefully

Of course, I may be wrong, I need to create a test case without Qt. Maybe the problem is the combination of Qt/spdlog/Windows github CI?

[narnaud]

I can confirm it's the combination of Qt / spdlog / Windows github CI in release.

I tried with doctest, and the test is passing:
https://github.com/narnaud/tst-release/actions/runs/9427485839/job/25971652480?pr=6

Sorry for the noise. Now I wonder how I am going to find the problem...

[tt4g]

If the problem only occurs when using QTest, then that is the cause.
I don't think the Qt experts are checking here, so you should ask your question in the Qt community.

[narnaud]

Thank you for your time, I think we can close it... I'm still puzzled by all of this :)

[narnaud]

Ok, we are experiencing a problem on Windows runner in release mode on different repository (some where passing before).
May be related to: actions/runner-images#10019

So nothing to do with spdlog or Qt apparently...

[LeonMatthesKDAB]

For completeness sake: The issue seems to be resolved now with the newest Windows runners.
So time to revert the workarounds and reenable spdlog in our tests 😅

[zchrissirhcz]

Hi, @narnaud and @LeonMatthesKDAB , I meet a very similar crash on Windows. I paste my minimal reproducible example steps in #2902, which use a very simple code, no Qt stuffs:

#include <spdlog/spdlog.h>

int main()
{
    SPDLOG_INFO("x");
    return 0;
}

I'm also puzzled with what help solved the KDAB/knut. Just update github action's windows image? Well, my reproduce runs on a local machine, with latest Visual Studio 2022 installed (17.10.3). @narnaud Do you use a exactly same version Visual Studio 2022 in your github action and your local machine? If possible, would you please use my minimal code for reproduce trial? Note that I did not enable Address Sanitizer, and crash only occurs in Release and RelWithDebInfo configurations.

[narnaud]

Hi @zchrissirhcz
I didn't test you minimal example, but everything is working for us on the github workers or locally on my computer (latest version of VS 2022).
I would suggest to try a known version of spdlog instead of using master first.
Sorry, I can't help you more than that.

[zchrissirhcz]

@narnaud Thanks for your reply. I also tried spdlog versions 1.14.1 and 1.12.0, they both reproduce, then I switch to v1.x (the "master" you mentioned, I guess), also reproduce. Could you please help verify your local machines Visual Studio version? That is in Help -> About,

My reproduce uses "the latest version", which is 17.10.3:

And uninstall that version of VS2022, install an older version, 17.9.7, from https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-history#uninstalling-visual-studio-to-go-back-to-an-earlier-release , cannot reproduce it anymore:

[tt4g]

Resolved (MSVC breaking change): #2902 (comment)