Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The way Pikiwidb does snapshots. #207

Closed
dingxiaoshuai123 opened this issue Mar 14, 2024 · 2 comments
Closed

The way Pikiwidb does snapshots. #207

dingxiaoshuai123 opened this issue Mar 14, 2024 · 2 comments
Labels
☢️ Bug Something isn't working

Comments

@dingxiaoshuai123
Copy link
Collaborator

dingxiaoshuai123 commented Mar 14, 2024

  1. on_snapshot_save:变成空操作。需要braft暴露一个可以主动设置snapshot log index的接口。(panlei zhoukaiyan)
    1.1 改造 FSMCaller::do_snapshot_save(SaveSnapshotClosure* done) :修改赋予 last_applied_index 的值。
    1.2 故障恢复的回放点是可以动态设置的,替换掉为启动之后的apply-index
  2. 访问不平均时,memtable可以不进行flush,导致内存里最小的Log index一直不推进。需要一个机制主动去flush掉这些停留很久的log index。(ningchangyuan hjw)
    2.1 eventlisten 监听 flush 事件,在回调时检查 log 数量,超过阈值 x 之后, flush 一个 cf :选择出一个 (当前 seq 与 所有 mem 的 first_seq 的最大差值,且超过一个阈值)对应的 cf 。
    2.2 调优:找出合理的最大差值阈值。从 memtable 对应的 braft 日志条数来反推当前 cf 落后的总量。
  3. on_snapshot_load: 因为save是空操作,install snapshot会变成一个即时行为,需要重载一下snapshot reader和writer。(dxs wxr)
    3.1 https://luobuda.github.io/2022/02/15/braft-snapshot%E5%AE%9E%E7%8E%B0/ 借鉴看看
    3.2 通过迭代器生成 SST 文件并通过 ingest 方法导入 RocksDB 实例
    3.3 生成 checkpoint 将文件发送给其他 node 。
@dingxiaoshuai123 dingxiaoshuai123 added the ☢️ Bug Something isn't working label Mar 14, 2024
@super-Pan66
Copy link
Contributor

#238
#245

@panlei-coder
Copy link
Collaborator

#275

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
☢️ Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants