Disk buffer bugs tracking issue

[ktff]

Tracking issue for various bug issues related to disk buffer, their causes, and fixes.

Issues (work in progress):

• Many ldb files (probably low I/O ops) (#document) (separate uncompact compaction could help) (fragmentation? it would be a self perpetuating loop) (enhancement(buffers): Maintain working size of disk buffer #7264 aggravates it) (fix fix(buffers): Min uncompact size #7614)

  • issue with disk buffer - data_dir filling up with ldb files and data even though data is making it upstream #6602
  • High CPU and memory consumption with buffer enabled #7246 (comment)

• High idle cpu utilization (Periodically trigger disk buffer compaction #6597 could help) (fix fix(buffers): Batch delete #7615)

  • High CPU and memory consumption with buffer enabled #7246 (comment)
  • High CPU and memory consumption with buffer enabled #7246 (comment)
  • High CPU and memory consumption with buffer enabled #7246 (comment)

• Retaining ldb files, unbounded memory use (fix Periodically trigger disk buffer compaction #6597) (fix fix(buffers): Min uncompact size #7614)

  • High CPU and memory consumption with buffer enabled #7246 (comment)

• Large initial load (fix Compute disk buffer initial_size with Jean's method  #7408 unacceptable increase of startup time)(fix chore: Use a forked version of leveldb-sys #7522)

  • Stop sinking when buffer exceed max_size #6741 (comment)
  • High CPU and memory consumption with buffer enabled #7246 (comment)

• Blocked sink (Periodically trigger disk buffer compaction #6597 could help) (fixes made during move to future 0.3 have maybe fixed it) (slow disk) (partial fix fix(buffers): Batch delete #7615)

  • Stop sinking when buffer exceed max_size #6741

Todo

• Periodically trigger disk buffer compaction #6597
• chore: Use a forked version of leveldb-sys #7522
• Compute disk buffer initial_size with Jean's method  #7408 wasn't acceptable
• Batch deletes fix(buffers): Batch delete #7615
• Min uncompacted size fix(buffers): Min uncompact size #7614
• Document disk buffer memory behavior. chore(external docs): Document disk buffer behavior #7679
  * It will use memory proportional to disk size.
  * It can be slightly larger on disk than the configured max_size.
  * Clarify use case.
  * Mention low disk performance related issues.
• Notify relevant users when fixes get released to recheck them.

[ktff]

Residual ldb files

Bug where a large number of ldb files are never removed and just keep accumulating has been further aggravated by #7264 PR. With that PR the following config reproduces the issue

data_dir = "./../tmp"

[sources.source0]
  type = "generator"
  format = "shuffle" 
  lines = ["line"] 
  sequence = true
  interval = 0.1

[sinks.sink1]
  type = "blackhole"
  inputs = ["source0"] 
  buffer.max_size = 10000000000
  buffer.type = "disk" 
  buffer.when_full = "block" 

The main factor is the interval, that is if total emitted bytes is less than 1MB when the compaction happens there is a chance that it will leave behind a ldb file. The less bytes emitted, greater the chance. The above config leaves every ldb file while emitting 42KB in 1min

On our side, we can fix it by limiting compaction to happen only if we have 2MB or more of uncompacted bytes. We could go for a lower bound but, just to be safe, 2MB is the max size of ldb files and there is a chance that the bug happens for sizes greater than 1MB but it requires ever so more time to materialize.

[jszwedko]

On our side, we can fix it by limiting compaction to happen only if we have 2MB or more of uncompacted bytes. We could go for a lower bound but, just to be safe, 2MB is the max size of ldb files and there is a chance that the bug happens for sizes greater than 1MB but it requires ever so more time to materialize.

Is this right? I think the max size of the ldb files is actually 4 MB. Or at least that's what I've seen.

[ktff]

I think the max size of the ldb files is actually 4 MB

Hmm, then it's more dynamic than I thought. So let's go with 4MB.

[ktff]

Fixes for every issue have been merged and relevant users notified so I'll be closing this. Subsequent follow ups on the bugs/fixes can be done in individual issues.