[feat](lock)add deadlock detection tool and monitored lock implementations #39015
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
morningman
reviewed
Aug 7, 2024
fe/fe-core/src/main/java/org/apache/doris/common/lock/AbstractMonitoredLock.java
Outdated
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/common/lock/AbstractMonitoredLock.java
Outdated
Show resolved
Hide resolved
|
run buildall |
|
run buildall |
TPC-H: Total hot run time: 39822 ms |
|
run performance |
TPC-H: Total hot run time: 39892 ms |
morningman
reviewed
Aug 8, 2024
|
|
||
| @ConfField(description = {"表示最大锁持有时间，超过该时间会打印告警日志，单位秒", | ||
| "Maximum lock hold time; logs a warning if exceeded"}) | ||
| public static long max_lock_hold_threshold_seconds = 30; |
There was a problem hiding this comment.
Suggested change
| public static long max_lock_hold_threshold_seconds = 30; | |
| public static long max_lock_hold_threshold_seconds = 10; |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
approved
|
PR approved by anyone and no changes requested. |
|
run buildall |
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Aug 8, 2024
…tions apache#39015 (#22) ## Proposed changes ### Description: This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: #### AbstractMonitoredLock: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` #### DeadlockCheckerTool: Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ##### benchmark ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ```
TPC-H: Total hot run time: 39002 ms |
TPC-DS: Total hot run time: 203402 ms |
ClickBench: Total hot run time: 30.83 s |
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Aug 8, 2024
…tions apache#39015 (#22) ## Proposed changes ### Description: This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: #### AbstractMonitoredLock: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` #### DeadlockCheckerTool: Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ##### benchmark ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ```
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 14, 2024
…tions (apache#39015) ## Proposed changes ### Description: This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: #### AbstractMonitoredLock: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` #### DeadlockCheckerTool: Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ##### benchmark ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ```
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Aug 18, 2024
…tions apache#39015 (apache#39099) ## Proposed changes apache#39015 ### Description: This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: #### AbstractMonitoredLock: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` #### DeadlockCheckerTool: Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ##### benchmark ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
gavinchou
added
dev/3.0.x
dev/3.0.x-conflict
and removed
dev/3.0.2-merged
dev/3.0.x
labels
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Sep 13, 2024
…tions (apache#39015) ## Proposed changes ### Description: This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: #### AbstractMonitoredLock: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` #### DeadlockCheckerTool: Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ##### benchmark ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ```
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Sep 13, 2024
…k implementations (apache#39015) ## Proposed changes ### Description: This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: #### AbstractMonitoredLock: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` #### DeadlockCheckerTool: Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ##### benchmark ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ``` (cherry picked from commit f0ca491)
liutang123
pushed a commit
to liutang123/doris
that referenced
this pull request
Sep 14, 2024
…tions apache#39015 (apache#39099) This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ``` Issue Number: close #xxx <!--Describe your changes.-->
liutang123
pushed a commit
to liutang123/doris
that referenced
this pull request
Sep 14, 2024
…tions apache#39015 (apache#39099) This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ``` Issue Number: close #xxx <!--Describe your changes.-->
liutang123
pushed a commit
to liutang123/doris
that referenced
this pull request
Sep 14, 2024
…tions apache#39015 (apache#39099) This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ``` Issue Number: close #xxx <!--Describe your changes.-->
liutang123
pushed a commit
to liutang123/doris
that referenced
this pull request
Sep 14, 2024
…tions apache#39015 (apache#39099) This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ``` Issue Number: close #xxx <!--Describe your changes.-->
liutang123
pushed a commit
to liutang123/doris
that referenced
this pull request
Sep 19, 2024
…tions apache#39015 (apache#39099) This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ``` Issue Number: close #xxx <!--Describe your changes.-->
liutang123
pushed a commit
to liutang123/doris
that referenced
this pull request
Sep 20, 2024
liutang123
pushed a commit
to liutang123/doris
that referenced
this pull request
Sep 20, 2024
lide-reed
pushed a commit
that referenced
this pull request
Sep 20, 2024
…ock (#40746) Use the fe config like #39015, print table lock usage duration. In 2.0.6, I found `replayer` thread is blocked for a long time. ``` 2024-09-12 11:00:58,797 INFO (replayer|89) [DatabaseTransactionMgr.replayUpsertTransactionState():1977] replay a visible transaction TransactionState. transaction id: 471728922, label: flink_connector_20240912_103150_269f8d635ffb455bb4043eecd1a8406b, db id: 475180, table id list: 14473765, callback id: -1, coordinator: BE: 10.217.0.100, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1726108310505, commit time: 1726108310579, finish time: 1726108310604, reason: 2024-09-12 11:00:58,798 WARN (replayer|89) [Env.replayJournal():2575] entityCost:940256 loadJournalCost:940256 logId:1392353661 replayedJournalId:1392353661 code:100 size:562 ``` And, there are many table lock fail log: ``` 2024-09-12 10:56:53,167 WARN (mysql-nio-pool-32036|412611) [Table.tryReadLock():155] Failed to try table atdpersonpaycode's read lock. timeout 1 MINUTES. Current owner: null ``` But, I don't kown what thread hold the table lock. So, add some logs to print lock holding info.
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Sep 23, 2024
…branch-3.0-apache#39015 # Conflicts: # fe/fe-common/src/main/java/org/apache/doris/common/Config.java
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Oct 11, 2024
…k implementations (apache#39015) #
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Description:
This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage.
Features:
AbstractMonitoredLock:
A monitored version of Lock that tracks and logs lock acquisition and release times.
Functionality:
Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods.
Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout.
eg
DeadlockCheckerTool:
Uses ScheduledExecutorService for periodic deadlock checks.
Logs deadlock information including thread names, states, lock info, and stack traces.
ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network.
Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.
eg
benchmark