[enhancement](cloud) optimize block cache lock #41818
Conversation
freemandealer
commented
Oct 14, 2024
•
freemandealer
commented
- async deletion when do stale rowsets reclycle
- minimize lock critical size
- add cache lock held & wait time info for debug
1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug Signed-off-by: freemandealer <[email protected]>
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -19,6 +19,7 @@ | |||
|
|
|||
| #include <bvar/bvar.h> | |||
There was a problem hiding this comment.
warning: 'bvar/bvar.h' file not found [clang-diagnostic-error]
#include <bvar/bvar.h>
^
be/src/io/cache/block_file_cache.h
Outdated
| template <class Lock> | ||
| concept IsXLock = std::same_as<Lock, std::lock_guard<std::mutex>> || | ||
| std::same_as<Lock, std::unique_lock<std::mutex>>; | ||
|
|
||
| class FSFileCacheStorage; | ||
|
|
||
| class LockScopedTimer { | ||
| public: | ||
| LockScopedTimer() : start_(std::chrono::high_resolution_clock::now()) {} |
There was a problem hiding this comment.
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
| LockScopedTimer() : start_(std::chrono::high_resolution_clock::now()) {} | |
| LockScopedTimer() : start_(std::chrono::high_resolution_clock::now()) = default; |
Signed-off-by: freemandealer <[email protected]>
|
run buildall |
1 similar comment
|
run buildall |
freemandealer
force-pushed
the
opt-cache-lock
branch
from
9e44e1e to
1981d93
Compare
Signed-off-by: freemandealer <[email protected]>
freemandealer
force-pushed
the
opt-cache-lock
branch
from
1981d93 to
384b29d
Compare
|
run buildall |
Signed-off-by: freemandealer <[email protected]>
| template <class Lock> | ||
| concept IsXLock = std::same_as<Lock, std::lock_guard<std::mutex>> || | ||
| std::same_as<Lock, std::unique_lock<std::mutex>>; | ||
|
|
||
| class FSFileCacheStorage; | ||
|
|
||
| class LockScopedTimer { | ||
| public: | ||
| LockScopedTimer() : start_(std::chrono::steady_clock::now()) {} |
There was a problem hiding this comment.
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
| LockScopedTimer() : start_(std::chrono::steady_clock::now()) {} | |
| LockScopedTimer() : start_(std::chrono::steady_clock::now()) = default; |
|
run buildall |
|
TeamCity be ut coverage result: |
| std::chrono::duration_cast<std::chrono::milliseconds>(acq_time - start_time).count(); \ | ||
| if (duration > config::cache_lock_long_tail_threshold) \ | ||
| LOG(WARNING) << "Lock wait time " << std::to_string(duration) << "ms. " \ | ||
| << get_stack_trace_by_boost() << std::endl; \ |
There was a problem hiding this comment.
We should limit frequency of the log.
There was a problem hiding this comment.
@dataroaring It is limited already: only > config::cache_lock_long_tail_threshold (default:1s) will print the log. If not enough, enlarge the threshold.
|
run cloud_p0 |
|
run buildall |
|
TeamCity be ut coverage result: |
| while (_recycle_keys->pop(key)) { | ||
| Status st = _storage->remove(key); | ||
| if (!st.ok()) { | ||
| LOG_WARNING("").error(st); |
There was a problem hiding this comment.
put some key word in the log e.g.
"failed to remove when recycle_stale_rowset_async_bottom_half, key = {}" ...
| LOG_WARNING("Failed to push recycle key to queue, do it synchronously"); | ||
| Status st = _storage->remove(key); | ||
| if (!st.ok()) { | ||
| LOG_WARNING("").error(st); |
There was a problem hiding this comment.
put some key word in the log e.g.
"failed to remove async, key = {}" ...
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug
pick apache#41818 from master 1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug
…#43401) pick apache#41818 from master 1. async deletion when do stale rowsets reclycle 2. minimize lock critical size 3. add cache lock held & wait time info for debug
