Debugging hangs

[msimberg]

No description provided.

[msimberg]

cscs-ci run

[msimberg]

I suspect https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/4700071344751697/7514005670787789/-/jobs/8650776698#L2248 might be another case of SCOPED_TRACE while yielding. I'm going to wait for pika 0.31.0 and pika-org/pika#1373 before testing again. That PR should avoid yielding on async_rw_mutex access if the previous access has already been waited for.

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

With pika 0.30.1 I've been able to reproduce two segfaults on test_bt_reduction_to_band: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/4700071344751697/7514005670787789/-/jobs/8686983455 and https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/4700071344751697/7514005670787789/-/jobs/8686963129. The backtraces in these cases don't explicitly mention gtest, so I can't be 100% sure they're related to scoped trace, but the symptoms otherwise look the same.

So far I haven't been able to reproduce anything with pika 0.31.0, which is a good sign, but I'll run more tests.

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

It seems like pika 0.31.0 is still able to trigger something related to gtest/scoped_trace: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/4700071344751697/7514005670787789/-/jobs/8688842361.

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

So far so good with the pika branch I'm working on here: pika-org/pika#1379. That PR removes another source of potential yielding from async_rw_mutex compared to pika-org/pika#1373. I'll keep rerunning tests here, but pika-org/pika#1379 will likely anyway be available in pika 0.32.0.

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

cscs-ci run

[msimberg]

I've separated one fix into #1257. I still need to find a good solution for the debug bloat so that we can use pika 0.31.0 in CI as well.

[msimberg]

A patch to have gtest detect yielding during ScopedTrace is here: #1258.