RAM size increase on using RocksDB Java

[Aravindhan1995]

I am using RocksDB as Queue for my application, The load is 5000 inserts and deletes per minute, and continuous read using rocks iterator. The RAM size increases over time.
Configuration Details:
JVM RAM: 2GB
DB WRITE BUFFER SIZE: 256 MB
Each day 50-100 MB of RAM increases for the process. I am closing all the rocks objects, I am not sure what is the reason for the increase in memory.

This is a RocksDB option file.

[Version]
rocksdb_version=6.0.1
options_file_version=1.1

[DBOptions]
preserve_deletes=false
allow_ingest_behind=false
dump_malloc_stats=false
info_log_level=INFO_LEVEL
write_thread_max_yield_usec=100
avoid_flush_during_shutdown=false
enable_write_thread_adaptive_yield=true
wal_recovery_mode=kPointInTimeRecovery
fail_if_options_file_error=false
stats_history_buffer_size=1048576
stats_persist_period_sec=600
delete_obsolete_files_period_micros=21600000000
bytes_per_sync=0
enable_pipelined_write=false
max_subcompactions=1
db_log_dir=
wal_dir=/home/local/ZOHOCORP/aravind-4692/IOTBuild/Aug12Build/PersistentCache/RocksDB/data2
max_log_file_size=0
manifest_preallocation_size=4194304
delayed_write_rate=16777216
log_file_time_to_roll=0
avoid_flush_during_recovery=false
write_thread_slow_yield_usec=3
keep_log_file_num=5
table_cache_numshardbits=6
max_file_opening_threads=16
max_background_flushes=-1
base_background_compactions=-1
max_background_compactions=-1
use_fsync=false
use_adaptive_mutex=false
wal_bytes_per_sync=0
random_access_max_buffer_size=1048576
atomic_flush=false
compaction_readahead_size=0
manual_wal_flush=false
new_table_reader_for_compaction_inputs=false
max_total_wal_size=0
skip_stats_update_on_db_open=false
skip_log_error_on_recovery=false
max_manifest_file_size=1073741824
paranoid_checks=true
stats_dump_period_sec=600
recycle_log_file_num=0
is_fd_close_on_exec=true
error_if_exists=false
enable_thread_tracking=false
create_missing_column_families=false
WAL_ttl_seconds=0
create_if_missing=true
access_hint_on_compaction_start=NORMAL
max_background_jobs=2
allow_2pc=false
use_direct_io_for_flush_and_compaction=false
db_write_buffer_size=0
two_write_queues=false
use_direct_reads=false
allow_concurrent_memtable_write=true
allow_mmap_writes=false
writable_file_max_buffer_size=1048576
WAL_size_limit_MB=0
allow_fallocate=true
max_open_files=-1
allow_mmap_reads=false
advise_random_on_open=true

[CFOptions "default"]
compaction_pri=kMinOverlappingRatio
merge_operator=nullptr
compaction_filter_factory=nullptr
memtable_factory=SkipListFactory
memtable_insert_with_hint_prefix_extractor=nullptr
comparator=leveldb.BytewiseComparator
target_file_size_base=67108864
max_sequential_skip_in_iterations=8
compaction_style=kCompactionStyleLevel
max_bytes_for_level_base=268435456
bloom_locality=0
write_buffer_size=67108864
compression_per_level=
memtable_huge_page_size=0
max_successive_merges=0
arena_block_size=8388608
memtable_whole_key_filtering=false
target_file_size_multiplier=1
max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1
num_levels=7
min_write_buffer_number_to_merge=1
max_write_buffer_number_to_maintain=0
max_write_buffer_number=2
compression=kSnappyCompression
level0_stop_writes_trigger=36
level0_slowdown_writes_trigger=20
compaction_filter=nullptr
level0_file_num_compaction_trigger=1
max_compaction_bytes=1677721600
compaction_options_universal={allow_trivial_move=false;stop_style=kCompactionStopStyleTotalSize;compression_size_percent=-1;max_size_amplification_percent=200;max_merge_width=4294967295;min_merge_width=2;size_ratio=1;}
memtable_prefix_bloom_size_ratio=0.000000
hard_pending_compaction_bytes_limit=274877906944
ttl=0
table_factory=BlockBasedTable
soft_pending_compaction_bytes_limit=68719476736
prefix_extractor=nullptr
bottommost_compression=kDisableCompressionOption
force_consistency_checks=false
paranoid_file_checks=false
compaction_options_fifo={allow_compaction=false;max_table_files_size=1073741824;}
max_bytes_for_level_multiplier=10.000000
optimize_filters_for_hits=false
level_compaction_dynamic_level_bytes=false
inplace_update_num_locks=10000
inplace_update_support=false
disable_auto_compactions=false
report_bg_io_stats=false

[TableOptions/BlockBasedTable "default"]
pin_top_level_index_and_filter=true
enable_index_compression=true
read_amp_bytes_per_bit=8589934592
format_version=2
whole_key_filtering=true
block_align=false
metadata_block_size=4096
block_size_deviation=10
partition_filters=false
block_size=4096
index_block_restart_interval=1
no_block_cache=false
checksum=kCRC32c
data_block_index_type=kDataBlockBinarySearch
index_type=kBinarySearch
verify_compression=false
filter_policy=nullptr
data_block_hash_table_util_ratio=0.750000
pin_l0_filter_and_index_blocks_in_cache=false
block_restart_interval=16
cache_index_and_filter_blocks_with_high_priority=false
cache_index_and_filter_blocks=false
hash_index_allow_collision=true
flush_block_policy_factory=FlushBlockBySizePolicyFactory

[CFOptions "M2MDataProcessor"]
compaction_pri=kMinOverlappingRatio
merge_operator=nullptr
compaction_filter_factory=nullptr
memtable_factory=SkipListFactory
memtable_insert_with_hint_prefix_extractor=nullptr
comparator=leveldb.BytewiseComparator
target_file_size_base=67108864
max_sequential_skip_in_iterations=8
compaction_style=kCompactionStyleLevel
max_bytes_for_level_base=268435456
bloom_locality=0
write_buffer_size=67108864
compression_per_level=
memtable_huge_page_size=0
max_successive_merges=0
arena_block_size=8388608
memtable_whole_key_filtering=false
target_file_size_multiplier=1
max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1
num_levels=7
min_write_buffer_number_to_merge=1
max_write_buffer_number_to_maintain=0
max_write_buffer_number=2
compression=kSnappyCompression
level0_stop_writes_trigger=36
level0_slowdown_writes_trigger=20
compaction_filter=nullptr
level0_file_num_compaction_trigger=1
max_compaction_bytes=1677721600
compaction_options_universal={allow_trivial_move=false;stop_style=kCompactionStopStyleTotalSize;compression_size_percent=-1;max_size_amplification_percent=200;max_merge_width=4294967295;min_merge_width=2;size_ratio=1;}
memtable_prefix_bloom_size_ratio=0.000000
hard_pending_compaction_bytes_limit=274877906944
ttl=0
table_factory=BlockBasedTable
soft_pending_compaction_bytes_limit=68719476736
prefix_extractor=nullptr
bottommost_compression=kDisableCompressionOption
force_consistency_checks=false
paranoid_file_checks=false
compaction_options_fifo={allow_compaction=false;max_table_files_size=1073741824;}
max_bytes_for_level_multiplier=10.000000
optimize_filters_for_hits=false
level_compaction_dynamic_level_bytes=false
inplace_update_num_locks=10000
inplace_update_support=false
disable_auto_compactions=false
report_bg_io_stats=false

[TableOptions/BlockBasedTable "M2MDataProcessor"]
pin_top_level_index_and_filter=true
enable_index_compression=true
read_amp_bytes_per_bit=8589934592
format_version=2
whole_key_filtering=true
block_align=false
metadata_block_size=4096
block_size_deviation=10
partition_filters=false
block_size=4096
index_block_restart_interval=1
no_block_cache=false
checksum=kCRC32c
data_block_index_type=kDataBlockBinarySearch
index_type=kBinarySearch
verify_compression=false
filter_policy=nullptr
data_block_hash_table_util_ratio=0.750000
pin_l0_filter_and_index_blocks_in_cache=false
block_restart_interval=16
cache_index_and_filter_blocks_with_high_priority=false
cache_index_and_filter_blocks=false
hash_index_allow_collision=true
flush_block_policy_factory=FlushBlockBySizePolicyFactory

[CFOptions "M2MDataProcessorKeyVsDeviceIdentifier"]
compaction_pri=kMinOverlappingRatio
merge_operator=nullptr
compaction_filter_factory=nullptr
memtable_factory=SkipListFactory
memtable_insert_with_hint_prefix_extractor=nullptr
comparator=leveldb.BytewiseComparator
target_file_size_base=67108864
max_sequential_skip_in_iterations=8
compaction_style=kCompactionStyleLevel
max_bytes_for_level_base=268435456
bloom_locality=0
write_buffer_size=67108864
compression_per_level=
memtable_huge_page_size=0
max_successive_merges=0
arena_block_size=8388608
memtable_whole_key_filtering=false
target_file_size_multiplier=1
max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1
num_levels=7
min_write_buffer_number_to_merge=1
max_write_buffer_number_to_maintain=0
max_write_buffer_number=2
compression=kSnappyCompression
level0_stop_writes_trigger=36
level0_slowdown_writes_trigger=20
compaction_filter=nullptr
level0_file_num_compaction_trigger=1
max_compaction_bytes=1677721600
compaction_options_universal={allow_trivial_move=false;stop_style=kCompactionStopStyleTotalSize;compression_size_percent=-1;max_size_amplification_percent=200;max_merge_width=4294967295;min_merge_width=2;size_ratio=1;}
memtable_prefix_bloom_size_ratio=0.000000
hard_pending_compaction_bytes_limit=274877906944
ttl=0
table_factory=BlockBasedTable
soft_pending_compaction_bytes_limit=68719476736
prefix_extractor=nullptr
bottommost_compression=kDisableCompressionOption
force_consistency_checks=false
paranoid_file_checks=false
compaction_options_fifo={allow_compaction=false;max_table_files_size=1073741824;}
max_bytes_for_level_multiplier=10.000000
optimize_filters_for_hits=false
level_compaction_dynamic_level_bytes=false
inplace_update_num_locks=10000
inplace_update_support=false
disable_auto_compactions=false
report_bg_io_stats=false

[TableOptions/BlockBasedTable "M2MDataProcessorKeyVsDeviceIdentifier"]
pin_top_level_index_and_filter=true
enable_index_compression=true
read_amp_bytes_per_bit=8589934592
format_version=2
whole_key_filtering=true
block_align=false
metadata_block_size=4096
block_size_deviation=10
partition_filters=false
block_size=4096
index_block_restart_interval=1
no_block_cache=false
checksum=kCRC32c
data_block_index_type=kDataBlockBinarySearch
index_type=kBinarySearch
verify_compression=false
filter_policy=nullptr
data_block_hash_table_util_ratio=0.750000
pin_l0_filter_and_index_blocks_in_cache=false
block_restart_interval=16
cache_index_and_filter_blocks_with_high_priority=false
cache_index_and_filter_blocks=false
hash_index_allow_collision=true
flush_block_policy_factory=FlushBlockBySizePolicyFactory

[anand1976]

Can you obtain a heap profile? And is the DB size constant?

[koldat]

You should set max_open_files=100 or something like that. Otherwise it is unbounded. RocksDB is keeping indices of all opened files in memory.

[Aravindhan1995]

@koldat Thank you very much, will change the number and update you the result.

[Aravindhan1995]

@koldat after reducing the number of MAX_OPEN_FILES to 20, still the RAM keeps on increasing.

[koldat]

You can take a look on some of my tips in #4112 There are many reason and I posted what we use to stop memory growing. Key is:

• max_open_files
• Two level index
• Controlled cache
• Xms/Xmx on same value during start
• JEMalloc (do not use 5.x version that cause deadlock in JVM and no one knows why (Oracle). Use older 4.x version)

[jestan]

@Aravindhan1995 I also had a similar problem but after following @koldat 's suggestions and other issues reported here about memory usage helped me a lot to find a solution.

This is what i did after analyzing the memory usage and it lead to memory usage reduction without changing the application or the storage level change.

1. Controlled shared lru cache for all tables
2. Limiting write buffer size
3. Periodically flush memtable using a application thread
4. Using jemalloc memory allocator (i didn't know jemalloc 5.x has this dead lock issue with JVM, Thanks @koldat for mentioning it here)

(what i have tried and found was setting max_open_files will aggressively reduce the index memory usage and introduce more latency, if you have a latency sensitive application, then it may not be applicable, YMMV)

Before doing anything, measure the rocksdb memory usage periodically and log it to a file and analyze which component is using more memory and then decide about the solution.

You can use following methods in RocksDB object to collect statistics

long sharedBlockCacheUsage = Long.parseLong(rocksDb.property("rocksdb.block-cache-usage"));
long memTableUsage = Long.parseLong(rocksDb.property("rocksdb.size-all-mem-tables"));
long tableReaderUsage = Long.parseLong(rocksDb.property("rocksdb.estimate-table-readers-mem"));

[patkivikram]

Is block_cache_size the way to limit the lru cache?

[wzhiyog]

hello! I also encountered this problem, how did you solve it in the end?

[patkivikram]

@koldat I see that you are having heap memory issues as well. Our heap with our kafka streams app keeps on growing running in python and using centos. Our settings are
ROCKS_DB_OPTIONS = { "write_buffer_size": 8388608, "max_write_buffer_number": 2, "block_cache_size": 8388608, "bloom_filter_size": 3, "max_open_files": 500, "block_cache_compressed_size": 8388608, "compression": rocksdb.CompressionType.lz4_compression, "target_file_size_base": 2097152, "level0_file_num_compaction_trigger": 1, "max_background_flushes": 8, "max_background_compactions": 8, }

How are you able to get it to work?

[macduy]

Hi @jestan, sorry to revive an old thread. We are facing the same issue and want to try jemalloc but don't know how. There are so few users of the Java RocksDB library that information on how to do this is non existent. Your information could save weeks of frustration for all future developers unfortunate enough to pick the combo of RocksDB + Java.

[adamretter]

@macduy If you are compiling RocksJava on Linux and you have jemalloc library installed it should be detected and support will be compiled in.

[koldat]

@macduy To use jemalloc you do not need to compile it in. We simply use LD_PRELOAD to replace default allocator. Even Java is using jemalloc after that.

export LD_PRELOAD=/usr/local/jemalloc5.2.1/lib/libjemalloc.so.2
export MALLOC_CONF="narenas:16,lg_tcache_max:13"

This is what we use in production very long time. Memory is very stable after that. I tried many allocators and jemalloc is simply the best.

[koldat]

BTW my comment:
"JEMalloc (do not use 5.x version that cause deadlock in JVM and no one knows why (Oracle). Use older 4.x version)" is obsolete.

Jemalloc team have found issues together with Java team so it is fixed (a year). Just use latest Java builds.

[macduy]

Thanks @koldat , I'm new to this but it looks like this will help us get RocksDB working on Java in a stable manner without having to compile Rocks ourselves.

[areyohrahul]

Hey @macduy, were you able to solve this problem?

Also, @koldat how did you figure out whether LD_PRELOAD has been successful in changing the memory allocator? I tried doing some thing but it didn't work for me. So, I'm guessing I haven't done it right.

[macduy]

@areyohrahul we ended up just setting MALLOC_ARENA_MAX=2.

We were going to try switching to jemalloc but we found that the above was sufficient to solve the problem and we didn't dig any deeper.

[areyohrahul]

Thanks @macduy for responding, I'll give it a try.

[areyohrahul]

@macduy I tried using this setting, one quick question. Did your memory leak stop completely? or was it delayed by a very long time using this setting?

[macduy]

I can't answer that with 100% certainty. What I know is it reduced our crash rate due to OOM massively (from daily to about once a month) but I cannot conclusively say whether the memory leaks have stopped.

[areyohrahul]

Got it, thanks.