Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ThreadSanitizer and RwLock #323

Closed
matanmarkind opened this issue Feb 25, 2022 · 6 comments · Fixed by #344
Closed

ThreadSanitizer and RwLock #323

matanmarkind opened this issue Feb 25, 2022 · 6 comments · Fixed by #344

Comments

@matanmarkind
Copy link

I am attempting to switch from the standard library's RwLock to parking_lot's. Unfortunately when making this switch thread sanitizer started to complain about data races. I was able to isolate that this is just a matter of the RwLock, since this diff fixes the issue by switching to std::sync::RwLock - matanmarkind/active_standby@bb1cd75.

Checking out the base commit, ee51de677f1722b9b1a0ec70ac9e9462d87bf4a1, and running the following results in failures about half the time with parking_lot, while never failing with the std RwLock.

RUST_BACKTRACE=full RUSTFLAGS="-Zsanitizer=thread -g" cargo +nightly bench shared_wguard_rw_contention -Z build-std --target x86_64-unknown-linux-gnu --quiet

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 127 filtered out; finished in 0.00s


running 1 test
==================
WARNING: ThreadSanitizer: data race (pid=516081)
  Write of size 4 at 0x7b04000016b8 by thread T11:
    #0 <bench::AddOne as active_standby::types::UpdateTables<i32,()>>::apply_first /home/matan/rust/active_standby/benches/bench.rs:29 (bench-d7c10858bf64896f+0xbbae9) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #1 <bench::AddOne as active_standby::types::UpdateTables<i32,()>>::apply_second /home/matan/rust/active_standby/benches/bench.rs:32 (bench-d7c10858bf64896f+0xbbae9)
    #2 active_standby::shared::aslock::AsLockWriteGuard<T>::update_tables::{{closure}} /home/matan/rust/active_standby/src/shared/aslock.rs:169 (bench-d7c10858bf64896f+0xbbae9)
    #3 core::ops::function::FnOnce::call_once{{vtable-shim}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xbbae9)
    #4 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1854 (bench-d7c10858bf64896f+0xc44cc) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #5 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1854 (bench-d7c10858bf64896f+0xc44cc) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #6 active_standby::shared::aslock::AsLock<T>::write /home/matan/rust/active_standby/src/shared/aslock.rs:99 (bench-d7c10858bf64896f+0xbbc50) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #7 bench::shared::AsLock::write /home/matan/rust/active_standby/src/macros.rs:231 (bench-d7c10858bf64896f+0xbebc3) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #8 bench::benchmarks::shared_wguard_rw_contention::{{closure}} /home/matan/rust/active_standby/benches/bench.rs:181 (bench-d7c10858bf64896f+0xbd50a) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #9 std::sys_common::backtrace::__rust_begin_short_backtrace /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:122 (bench-d7c10858bf64896f+0xbd50a)
    #10 std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (bench-d7c10858bf64896f+0xc3b6e) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #11 <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271 (bench-d7c10858bf64896f+0xc3b6e)
    #12 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0xc3b6e)
    #13 __rust_try.llvm.3183576308868158445 :? (bench-d7c10858bf64896f+0xc3d91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #14 __rust_try.llvm.3183576308868158445 :? (bench-d7c10858bf64896f+0xc3d91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #15 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0xbb9eb) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #16 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0xbb9eb)
    #17 std::thread::Builder::spawn_unchecked_::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:497 (bench-d7c10858bf64896f+0xb3ee2) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xb3ee2)
    #19 std::thread::Builder::spawn_unchecked_::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:497 (bench-d7c10858bf64896f+0xb3ee2) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #20 core::ops::function::FnOnce::call_once{{vtable-shim}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xb3ee2)
    #21 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1854 (bench-d7c10858bf64896f+0x131f7f) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1854 (bench-d7c10858bf64896f+0x131f7f)
    #23 std::sys::unix::thread::Thread::new::thread_start /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108 (bench-d7c10858bf64896f+0x1310bc) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)

  Previous read of size 4 at 0x7b04000016b8 by thread T4:
    #0 bench::benchmarks::shared_wguard_rw_contention::{{closure}}::{{closure}} /home/matan/rust/active_standby/benches/bench.rs:169 (bench-d7c10858bf64896f+0xbdb1a) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #1 std::sys_common::backtrace::__rust_begin_short_backtrace /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:122 (bench-d7c10858bf64896f+0xbdb1a)
    #2 std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (bench-d7c10858bf64896f+0xc368e) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #3 <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271 (bench-d7c10858bf64896f+0xc368e)
    #4 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0xc368e)
    #5 __rust_try.llvm.3183576308868158445 :? (bench-d7c10858bf64896f+0xc3d91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #6 __rust_try.llvm.3183576308868158445 :? (bench-d7c10858bf64896f+0xc3d91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #7 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0xbb4ab) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #8 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0xbb4ab)
    #9 std::thread::Builder::spawn_unchecked_::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:497 (bench-d7c10858bf64896f+0xb3ae2) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xb3ae2)
    #11 std::thread::Builder::spawn_unchecked_::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:497 (bench-d7c10858bf64896f+0xb3ae2) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #12 core::ops::function::FnOnce::call_once{{vtable-shim}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xb3ae2)
    #13 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1854 (bench-d7c10858bf64896f+0x131f7f) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #14 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1854 (bench-d7c10858bf64896f+0x131f7f)
    #15 std::sys::unix::thread::Thread::new::thread_start /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108 (bench-d7c10858bf64896f+0x1310bc) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)

  Location is heap block of size 16 at 0x7b04000016b0 allocated by main thread:
    #0 malloc /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:667 (bench-d7c10858bf64896f+0x29ac1) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #1 std::sys::unix::alloc::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::alloc /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/alloc.rs:14 (bench-d7c10858bf64896f+0x13c21c) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #2 __rdl_alloc /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/alloc.rs:355 (bench-d7c10858bf64896f+0x13c21c)
    #3 alloc::alloc::alloc /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:87 (bench-d7c10858bf64896f+0xbbdbe) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #4 alloc::alloc::Global::alloc_impl /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:169 (bench-d7c10858bf64896f+0xbbdbe)
    #5 <alloc::alloc::Global as core::alloc::Allocator>::allocate /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:229 (bench-d7c10858bf64896f+0xbbdbe)
    #6 alloc::alloc::exchange_malloc /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:318 (bench-d7c10858bf64896f+0xbbdbe)
    #7 alloc::boxed::Box<T>::new /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:195 (bench-d7c10858bf64896f+0xbbdbe)
    #8 active_standby::shared::aslock::AsLock<T>::from_identical /home/matan/rust/active_standby/src/shared/aslock.rs:61 (bench-d7c10858bf64896f+0xbbdbe)
    #9 active_standby::shared::aslock::AsLock<T>::new /home/matan/rust/active_standby/src/shared/aslock.rs:128 (bench-d7c10858bf64896f+0xbbdbe)
    #10 bench::shared::AsLock::new /home/matan/rust/active_standby/src/macros.rs:242 (bench-d7c10858bf64896f+0xbec20) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #11 bench::benchmarks::shared_wguard_rw_contention /home/matan/rust/active_standby/benches/bench.rs:162 (bench-d7c10858bf64896f+0xc5954) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #12 bench::benchmarks::shared_wguard_rw_contention::{{closure}} /home/matan/rust/active_standby/benches/bench.rs:161 (bench-d7c10858bf64896f+0xbcb55) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #13 core::ops::function::FnOnce::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xbcb55)
    #14 core::ops::function::FnMut::call_mut /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:150 (bench-d7c10858bf64896f+0xf0a6d) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #15 test::bench::Bencher::bench /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/bench.rs:56 (bench-d7c10858bf64896f+0xf0a6d)
    #16 test::bench::benchmark::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/bench.rs:208 (bench-d7c10858bf64896f+0x10c609) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #17 <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271 (bench-d7c10858bf64896f+0x10c609)
    #18 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0x10c609)
    #19 __rust_try.llvm.16455064722209264270 :? (bench-d7c10858bf64896f+0x10ca21) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #20 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0x10c3a3) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #21 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0xea395) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #22 test::bench::benchmark /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/bench.rs:208 (bench-d7c10858bf64896f+0xf1388) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #23 test::run_test /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:546 (bench-d7c10858bf64896f+0xce11f) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #24 test::run_tests /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:381 (bench-d7c10858bf64896f+0xcc9ff) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #25 test::console::run_tests_console /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/console.rs:286 (bench-d7c10858bf64896f+0x103ccd) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #26 test::test_main /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:116 (bench-d7c10858bf64896f+0xcaa60) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #27 test::test_main_static /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:135 (bench-d7c10858bf64896f+0xcac7c) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #28 bench::main /home/matan/rust/active_standby/benches/bench.rs:1 (bench-d7c10858bf64896f+0xbcc4c) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #29 core::ops::function::FnOnce::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xbdc0f) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #30 std::sys_common::backtrace::__rust_begin_short_backtrace /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:122 (bench-d7c10858bf64896f+0xbdc0f)
    #31 std::rt::lang_start::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:145 (bench-d7c10858bf64896f+0xbf75d) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #32 core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:259 (bench-d7c10858bf64896f+0x135f45) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #33 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0x135f45)
    #34 __rust_try.llvm.10611294404910455758 :? (bench-d7c10858bf64896f+0x136c91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #35 __rust_try.llvm.10611294404910455758 :? (bench-d7c10858bf64896f+0x136c91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #36 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0x135da0) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #37 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0x135da0)
    #38 std::rt::lang_start_internal::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128 (bench-d7c10858bf64896f+0x135da0)
    #39 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0x135da0)
    #40 __rust_try.llvm.10611294404910455758 :? (bench-d7c10858bf64896f+0x136c91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #41 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0x13464a) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #42 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0x13464a)
    #43 std::rt::lang_start_internal /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128 (bench-d7c10858bf64896f+0x123159) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #44 main ??:? (bench-d7c10858bf64896f+0xbd2fa) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #45 __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308 (libc.so.6+0x270b2) (BuildId: 099b9225bcb0d019d9d60884be583eb31bb5f44e)

  Thread T11 (tid=516093, running) created by main thread at:
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022 (bench-d7c10858bf64896f+0x2aecd) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #1 std::sys::unix::thread::Thread::new /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:87 (bench-d7c10858bf64896f+0x130ea1) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #2 std::thread::Builder::spawn_unchecked_ /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:526 (bench-d7c10858bf64896f+0xb55bf) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #3 std::thread::Builder::spawn_unchecked /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:456 (bench-d7c10858bf64896f+0xb55bf)
    #4 std::thread::Builder::spawn /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:388 (bench-d7c10858bf64896f+0xb55bf)
    #5 std::thread::spawn /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:649 (bench-d7c10858bf64896f+0xb55bf)
    #6 bench::benchmarks::shared_wguard_rw_contention /home/matan/rust/active_standby/benches/bench.rs:180 (bench-d7c10858bf64896f+0xc5a05) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #7 bench::benchmarks::shared_wguard_rw_contention::{{closure}} /home/matan/rust/active_standby/benches/bench.rs:161 (bench-d7c10858bf64896f+0xbcb55) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #8 core::ops::function::FnOnce::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xbcb55)
    #9 core::ops::function::FnMut::call_mut /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:150 (bench-d7c10858bf64896f+0xf0a6d) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #10 test::bench::Bencher::bench /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/bench.rs:56 (bench-d7c10858bf64896f+0xf0a6d)
    #11 test::bench::benchmark::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/bench.rs:208 (bench-d7c10858bf64896f+0x10c609) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #12 <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271 (bench-d7c10858bf64896f+0x10c609)
    #13 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0x10c609)
    #14 __rust_try.llvm.16455064722209264270 :? (bench-d7c10858bf64896f+0x10ca21) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #15 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0x10c3a3) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #16 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0xea395) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #17 test::bench::benchmark /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/bench.rs:208 (bench-d7c10858bf64896f+0xf1388) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #18 test::run_test /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:546 (bench-d7c10858bf64896f+0xce11f) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #19 test::run_tests /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:381 (bench-d7c10858bf64896f+0xcc9ff) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #20 test::console::run_tests_console /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/console.rs:286 (bench-d7c10858bf64896f+0x103ccd) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #21 test::test_main /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:116 (bench-d7c10858bf64896f+0xcaa60) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #22 test::test_main_static /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:135 (bench-d7c10858bf64896f+0xcac7c) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #23 bench::main /home/matan/rust/active_standby/benches/bench.rs:1 (bench-d7c10858bf64896f+0xbcc4c) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #24 core::ops::function::FnOnce::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xbdc0f) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #25 std::sys_common::backtrace::__rust_begin_short_backtrace /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:122 (bench-d7c10858bf64896f+0xbdc0f)
    #26 std::rt::lang_start::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:145 (bench-d7c10858bf64896f+0xbf75d) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #27 core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:259 (bench-d7c10858bf64896f+0x135f45) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #28 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0x135f45)
    #29 __rust_try.llvm.10611294404910455758 :? (bench-d7c10858bf64896f+0x136c91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #30 __rust_try.llvm.10611294404910455758 :? (bench-d7c10858bf64896f+0x136c91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #31 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0x135da0) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #32 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0x135da0)
    #33 std::rt::lang_start_internal::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128 (bench-d7c10858bf64896f+0x135da0)
    #34 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0x135da0)
    #35 __rust_try.llvm.10611294404910455758 :? (bench-d7c10858bf64896f+0x136c91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #36 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0x13464a) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #37 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0x13464a)
    #38 std::rt::lang_start_internal /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128 (bench-d7c10858bf64896f+0x123159) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #39 main ??:? (bench-d7c10858bf64896f+0xbd2fa) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #40 __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308 (libc.so.6+0x270b2) (BuildId: 099b9225bcb0d019d9d60884be583eb31bb5f44e)

  Thread T4 (tid=516086, running) created by main thread at:
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022 (bench-d7c10858bf64896f+0x2aecd) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #1 std::sys::unix::thread::Thread::new /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:87 (bench-d7c10858bf64896f+0x130ea1) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #2 std::thread::Builder::spawn_unchecked_ /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:526 (bench-d7c10858bf64896f+0xb7eff) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #3 std::thread::Builder::spawn_unchecked /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:456 (bench-d7c10858bf64896f+0xb7eff)
    #4 std::thread::Builder::spawn /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:388 (bench-d7c10858bf64896f+0xb7eff)
    #5 std::thread::spawn /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:649 (bench-d7c10858bf64896f+0xb7eff)
    #6 bench::benchmarks::shared_wguard_rw_contention::{{closure}} /home/matan/rust/active_standby/benches/bench.rs:167 (bench-d7c10858bf64896f+0xbf47a) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #7 core::iter::adapters::map::map_fold::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:84 (bench-d7c10858bf64896f+0xbf47a)
    #8 core::iter::traits::iterator::Iterator::fold /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2285 (bench-d7c10858bf64896f+0xbf47a)
    #9 <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:124 (bench-d7c10858bf64896f+0xbf47a)
    #10 core::iter::traits::iterator::Iterator::for_each /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:773 (bench-d7c10858bf64896f+0xc044b) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #11 <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/spec_extend.rs:40 (bench-d7c10858bf64896f+0xc044b)
    #12 <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter_nested.rs:62 (bench-d7c10858bf64896f+0xc044b)
    #13 <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter.rs:33 (bench-d7c10858bf64896f+0xc044b)
    #14 <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/vec/mod.rs:2552 (bench-d7c10858bf64896f+0xc59d4) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #15 core::iter::traits::iterator::Iterator::collect /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:1778 (bench-d7c10858bf64896f+0xc59d4)
    #16 bench::benchmarks::shared_wguard_rw_contention /home/matan/rust/active_standby/benches/bench.rs:164 (bench-d7c10858bf64896f+0xc59d4)
    #17 bench::benchmarks::shared_wguard_rw_contention::{{closure}} /home/matan/rust/active_standby/benches/bench.rs:161 (bench-d7c10858bf64896f+0xbcb55) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #18 core::ops::function::FnOnce::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xbcb55)
    #19 core::ops::function::FnMut::call_mut /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:150 (bench-d7c10858bf64896f+0xf0a6d) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #20 test::bench::Bencher::bench /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/bench.rs:56 (bench-d7c10858bf64896f+0xf0a6d)
    #21 test::bench::benchmark::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/bench.rs:208 (bench-d7c10858bf64896f+0x10c609) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #22 <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271 (bench-d7c10858bf64896f+0x10c609)
    #23 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0x10c609)
    #24 __rust_try.llvm.16455064722209264270 :? (bench-d7c10858bf64896f+0x10ca21) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #25 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0x10c3a3) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #26 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0xea395) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #27 test::bench::benchmark /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/bench.rs:208 (bench-d7c10858bf64896f+0xf1388) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #28 test::run_test /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:546 (bench-d7c10858bf64896f+0xce11f) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #29 test::run_tests /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:381 (bench-d7c10858bf64896f+0xcc9ff) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #30 test::console::run_tests_console /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/console.rs:286 (bench-d7c10858bf64896f+0x103ccd) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #31 test::test_main /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:116 (bench-d7c10858bf64896f+0xcaa60) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #32 test::test_main_static /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:135 (bench-d7c10858bf64896f+0xcac7c) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #33 bench::main /home/matan/rust/active_standby/benches/bench.rs:1 (bench-d7c10858bf64896f+0xbcc4c) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #34 core::ops::function::FnOnce::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (bench-d7c10858bf64896f+0xbdc0f) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #35 std::sys_common::backtrace::__rust_begin_short_backtrace /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:122 (bench-d7c10858bf64896f+0xbdc0f)
    #36 std::rt::lang_start::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:145 (bench-d7c10858bf64896f+0xbf75d) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #37 core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:259 (bench-d7c10858bf64896f+0x135f45) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #38 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0x135f45)
    #39 __rust_try.llvm.10611294404910455758 :? (bench-d7c10858bf64896f+0x136c91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #40 __rust_try.llvm.10611294404910455758 :? (bench-d7c10858bf64896f+0x136c91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #41 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0x135da0) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #42 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0x135da0)
    #43 std::rt::lang_start_internal::{{closure}} /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128 (bench-d7c10858bf64896f+0x135da0)
    #44 std::panicking::try::do_call /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:492 (bench-d7c10858bf64896f+0x135da0)
    #45 __rust_try.llvm.10611294404910455758 :? (bench-d7c10858bf64896f+0x136c91) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #46 std::panicking::try /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:456 (bench-d7c10858bf64896f+0x13464a) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #47 std::panic::catch_unwind /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137 (bench-d7c10858bf64896f+0x13464a)
    #48 std::rt::lang_start_internal /home/matan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128 (bench-d7c10858bf64896f+0x123159) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #49 main ??:? (bench-d7c10858bf64896f+0xbd2fa) (BuildId: aeaa5e27ed135c87da734c367fba412e16bdbc40)
    #50 __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308 (libc.so.6+0x270b2) (BuildId: 099b9225bcb0d019d9d60884be583eb31bb5f44e)

SUMMARY: ThreadSanitizer: data race /home/matan/rust/active_standby/benches/bench.rs:29 in <bench::AddOne as active_standby::types::UpdateTables<i32,()>>::apply_first
==================
test benchmarks::shared_wguard_rw_contention         ... bench:       5,859 ns/iter (+/- 4,566)

test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured; 20 filtered out; finished in 5.00s

ThreadSanitizer: reported 1 warnings
error: bench failed
@matanmarkind
Copy link
Author

Hi I was wondering if there was any update about getting parking_lot::RwLock to work with threadsanitizer? I'm happy to try out different configs or provide any logs if you'd like. Also let me know if you'd like to discuss the issue more and I'm happy to share some contact info so we can discuss :)

@Amanieu
Copy link
Owner

Amanieu commented May 17, 2022

I just tried your repo on ee51de677f1722b9b1a0ec70ac9e9462d87bf4a1 and it runs fine without producing any errors from ThreadSanitizer. I'm using rustc 1.62.0-nightly (de1026a67 2022-04-23).

@matanmarkind
Copy link
Author

Interesting, I just retried and still see the issue with rustc 1.62.0-nightly (ea92b0838 2022-05-07). I've tried this on a couple different computers running Ubuntu 20.04. What OS are you using?

Did you try running multiple times? The error is flaky (~50%).

@Amanieu
Copy link
Owner

Amanieu commented May 23, 2022

I tried running it 20 times and couldn't reproduce the error. I am on Arch Linux (kernel 5.17) with a Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz.

@matanmarkind
Copy link
Author

Hi Amanieu, thanks for trying I really appreciate it. I created a minimal example to repeat this issue and I just wanted to check that you don't see any failures before I send this to the TSAN team. Would you mind putting your cargo +nightly --version --verbose for me to include when I report?

$ cargo +nightly --version --verbose

cargo 1.63.0-nightly (3f052d8ee 2022-05-12)
release: 1.63.0-nightly
commit-hash: 3f052d8eed98c6a24f8b332fb2e6e6249d12d8c1
commit-date: 2022-05-12
host: x86_64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:OpenSSL/1.1.1n)
os: Ubuntu 20.04 (focal) [64-bit]

Run via:

RUSTFLAGS=-Zsanitizer=thread cargo +nightly run -Z build-std --target x86_64-unknown-linux-gnu

Code (parking_lot 0.12.0, rust edition 2021):

use std::sync::Arc;
use std::thread;

fn run_parking_lot() {
    let table = Arc::new(parking_lot::RwLock::new(1));

    let mut handles = Vec::new();
    for _ in 0..10 {
        let table = Arc::clone(&table);
        handles.push(thread::spawn(move || {
            while *table.read() != 0 {}
        }));
    }

    for i in 1..100000 {
        *table.write() = i;
    }
    *table.write() = 0;

    for h in handles {
        h.join().unwrap();
    }
}

fn run_std() { 
    let table = Arc::new(std::sync::RwLock::new(1));

    let mut handles = Vec::new();
    for _ in 0..10 {
        let table = Arc::clone(&table);
        handles.push(thread::spawn(move || {
            while *table.read().unwrap() != 0 {}
        }));
    }

    for i in 1..100000 {
        *table.write().unwrap() = i;
    }
    *table.write().unwrap() = 0;

    for h in handles {
        h.join().unwrap();
    }
}

fn main() {
    println!("run_parking_lot");
    run_parking_lot();

    println!("run_std");
    run_std();
}

@Amanieu
Copy link
Owner

Amanieu commented May 30, 2022

I managed to reproduce the error with your example. I've found the source of the bug, a fix is coming up. Thanks a lot for helping to track this down!

Amanieu added a commit that referenced this issue May 30, 2022
@Amanieu
Fix incorrect memory ordering in RwLock
d26c284 
Fixes #323
@Amanieu Amanieu mentioned this issue May 30, 2022
Merged
matanmarkind pushed a commit to matanmarkind/active_standby that referenced this issue May 31, 2022
@matan-starkware
Switch to parking_lot::RwLock with resolution of Amanieu/parking_lot#323
7720d5d 
.
Antiarchitect added a commit to Antiarchitect/async-broadcast that referenced this issue Jun 1, 2022
@Antiarchitect
Release 0.4.1
b93f3cd 
Update deps including `parking_lot` update to 0.12.1 fixing Amanieu/parking_lot#323
@Antiarchitect Antiarchitect mentioned this issue Jun 1, 2022
Merged
Antiarchitect added a commit to Antiarchitect/async-broadcast that referenced this issue Jun 1, 2022
@Antiarchitect
Update parking_lot to 0.12.1 fixing Amanieu/parking_lot#323
0d1b458
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix incorrect memory ordering in RwLock Amanieu/parking_lot
2 participants
@Amanieu @matanmarkind