Flaky SEH exceptions in tests involving bullet in Windows CI

[scpeters]

Environment

• OS Version: Windows Server 2022 ( https://build.osrfoundation.org/computer/win-ec2.nv.WinS22-e2b37850/ )
• Source or binary build? source build of 08dd8a7 from Mimic constraint feature using bullet-featherstone #517

Description

• Expected behavior: tests should pass reliably
• Actual behavior: some tests involving bullet fail in a flaky manner (fail sometimes, pass sometimes)

Example 1

• 08dd8a7 https://build.osrfoundation.org/job/ign_physics-pr-win/2295/

59: Test command: C:\J\workspace\ign_physics-pr-win\ws\build\gz-physics6\bin\Release\COMMON_TEST_joint_mimic_features.exe "C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/bin/Release/gz-physics6-bullet-featherstone-plugin.dll" "--gtest_output=xml:C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/test_results/COMMON_TEST_joint_mimic_features_bullet-featherstone.xml"
59: Test timeout computed to be: 240
59: [==========] Running 3 tests from 1 test suite.
59: [----------] Global test environment set-up.
59: [----------] 3 tests from JointMimicFeatureTest
59: [ RUN      ] JointMimicFeatureTest.PrismaticRevoluteMimicTest
59: JointMimicFeaturesTest::GetLibToTest() C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/bin/Release/gz-physics6-bullet-featherstone-plugin.dll
59: Testing plugin: gz::physics::bullet_featherstone::Plugin
59: No HOME defined in the environment. Will not log.
59: unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.
59: [  FAILED  ] JointMimicFeatureTest.PrismaticRevoluteMimicTest (275 ms)
59: [ RUN      ] JointMimicFeatureTest.PendulumMimicTest
59: JointMimicFeaturesTest::GetLibToTest() C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/bin/Release/gz-physics6-bullet-featherstone-plugin.dll
59: Testing plugin: gz::physics::bullet_featherstone::Plugin
59: [       OK ] JointMimicFeatureTest.PendulumMimicTest (172 ms)
59: [ RUN      ] JointMimicFeatureTest.PendulumsFastSlowMimicTest
59: JointMimicFeaturesTest::GetLibToTest() C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/bin/Release/gz-physics6-bullet-featherstone-plugin.dll
59: Testing plugin: gz::physics::bullet_featherstone::Plugin
59: unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.
59: [  FAILED  ] JointMimicFeatureTest.PendulumsFastSlowMimicTest (233 ms)
59: [----------] 3 tests from JointMimicFeatureTest (681 ms total)
59: 
59: [----------] Global test environment tear-down
59: [==========] 3 tests from 1 test suite ran. (681 ms total)
59: [  PASSED  ] 1 test.
59: [  FAILED  ] 2 tests, listed below:
59: [  FAILED  ] JointMimicFeatureTest.PrismaticRevoluteMimicTest
59: [  FAILED  ] JointMimicFeatureTest.PendulumsFastSlowMimicTest
59: 
59:  2 FAILED TESTS
 59/154 Test  #59: COMMON_TEST_joint_mimic_features_bullet-featherstone ......................***Failed    0.75 sec

Example 2

• 08dd8a7 https://build.osrfoundation.org/job/ign_physics-pr-win/2297/

59: Test command: C:\J\workspace\ign_physics-pr-win\ws\build\gz-physics6\bin\Release\COMMON_TEST_joint_mimic_features.exe "C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/bin/Release/gz-physics6-bullet-featherstone-plugin.dll" "--gtest_output=xml:C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/test_results/COMMON_TEST_joint_mimic_features_bullet-featherstone.xml"
59: Test timeout computed to be: 240
59: [==========] Running 3 tests from 1 test suite.
59: [----------] Global test environment set-up.
59: [----------] 3 tests from JointMimicFeatureTest
59: [ RUN      ] JointMimicFeatureTest.PrismaticRevoluteMimicTest
59: JointMimicFeaturesTest::GetLibToTest() C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/bin/Release/gz-physics6-bullet-featherstone-plugin.dll
59: Testing plugin: gz::physics::bullet_featherstone::Plugin
59: No HOME defined in the environment. Will not log.
59: [       OK ] JointMimicFeatureTest.PrismaticRevoluteMimicTest (319 ms)
59: [ RUN      ] JointMimicFeatureTest.PendulumMimicTest
59: JointMimicFeaturesTest::GetLibToTest() C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/bin/Release/gz-physics6-bullet-featherstone-plugin.dll
59: Testing plugin: gz::physics::bullet_featherstone::Plugin
59: [       OK ] JointMimicFeatureTest.PendulumMimicTest (239 ms)
59: [ RUN      ] JointMimicFeatureTest.PendulumsFastSlowMimicTest
59: JointMimicFeaturesTest::GetLibToTest() C:/J/workspace/ign_physics-pr-win/ws/build/gz-physics6/bin/Release/gz-physics6-bullet-featherstone-plugin.dll
59: Testing plugin: gz::physics::bullet_featherstone::Plugin
59: unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.
59: [  FAILED  ] JointMimicFeatureTest.PendulumsFastSlowMimicTest (241 ms)
59: [----------] 3 tests from JointMimicFeatureTest (800 ms total)
59: 
59: [----------] Global test environment tear-down
59: [==========] 3 tests from 1 test suite ran. (800 ms total)
59: [  PASSED  ] 2 tests.
59: [  FAILED  ] 1 test, listed below:
59: [  FAILED  ] JointMimicFeatureTest.PendulumsFastSlowMimicTest
59: 
59:  1 FAILED TEST
 59/154 Test  #59: COMMON_TEST_joint_mimic_features_bullet-featherstone ......................***Failed    0.87 sec

Steps to reproduce

1. Open a pull request targeting gz-physics6 or run ign_physics-gz-6-win

Output

[scpeters]

I've done some reading about SEH exceptions here and here, and it appears that they are a Microsoft extension to C and C++. Using the /EHsc compiler option is recommended and allows the exceptions to be handled in a standards-compliant way and is what we use in gz-cmake.

Seeing that this happens when using bullet in the two jenkins jobs listed above makes me think that perhaps the version of bullet we are using from vcpkg is not compiled with /EHsc. I did a quick search of both the bullet and vcpkg repositories and don't see /EHsc flags used for bullet.

[j-rivero]

I did a quick search of both the bullet and vcpkg repositories and don't see /EHsc flags used for bullet.

The (windows toolchain uses EHsc as part of the CMAKE_CXX_FLAGS](https://github.com/microsoft/vcpkg/blob/21bbb14c4113b89cd06402e852e075341722304f/scripts/toolchains/windows.cmake#L59]. Need to check the system logs to be fully sure that it is in use, I don't see anything in the port files that disable it.

[mjcarroll]

0xc0000005 is usually a memory access error. I'm assuming that you (the royal you, could be in Bullet, etc) are reading/writing from memory that has already been freed or you have an invalid pointer somewhere.

[scpeters]

I did a quick search of both the bullet and vcpkg repositories and don't see /EHsc flags used for bullet.

The (windows toolchain uses EHsc as part of the CMAKE_CXX_FLAGS]([https://github.com/microsoft/vcpkg/blob/21bbb14c4113b89cd06402e852e075341722304f/scripts/toolchains/windows.cmake#L59]](https://github.com/microsoft/vcpkg/blob/21bbb14c4113b89cd06402e852e075341722304f/scripts/toolchains/windows.cmake#L59%5D). Need to check the system logs to be fully sure that it is in use, I don't see anything in the port files that disable it.

ok, well that's good to know. The only other clue I see in the log files is

No HOME defined in the environment. Will not log.

early in the test, which I believe is from gz-common's Console.cc, but I don't know why that would cause memory access error

[azeey]

I think this is fixed by #539