Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Something wrong with memory while doing compaction. #626

Closed
superwood opened this issue Jun 4, 2015 · 2 comments
Closed

Something wrong with memory while doing compaction. #626

superwood opened this issue Jun 4, 2015 · 2 comments

Comments

@superwood
Copy link

The size of rocksdb (RocksDB version: 3.11.0) is 300G . The memory used by rocksdb keeps going up over time while compacting. Our OOM killer kills the process when it hits 64GB.

the stack message is
#0 0x00000039f040e6fd in write () from /lib64/libpthread.so.0
#1 0x000000000053bace in rocksdb::(anonymous namespace)::PosixWritableFile::Flush (this=0x5eb2a00)

at util/env_posix.cc:790

#2 0x00000000005946ba in rocksdb::BlockBasedTableBuilder::Flush (this=Unhandled dwarf expression opcode 0xf3

) at table/block_based_table_builder.cc:584
#3 0x0000000000594870 in rocksdb::BlockBasedTableBuilder::Add (this=0x3cb9af400, key=..., value=...)

at table/block_based_table_builder.cc:546

---Type to continue, or q to quit---
#4 0x00000000005761bd in rocksdb::CompactionJob::ProcessKeyValueCompaction (this=0x7f373ddb6d00,

imm_micros=0x7f373ddb5f78, input=0x43a0ba1c0, is_compaction_v2=false) at db/compaction_job.cc:879

#5 0x00000000005795f2 in rocksdb::CompactionJob::Run (this=0x7f373ddb6d00) at db/compaction_job.cc:360
#6 0x00000000004c0335 in rocksdb::DBImpl::BackgroundCompaction (this=0x2a08000, madeProgress=0x7f373ddb714e,

job_context=0x7f373ddb7170, log_buffer=0x7f373ddb7330) at db/db_impl.cc:2395

#7 0x00000000004cbedf in rocksdb::DBImpl::BackgroundCallCompaction (this=0x2a08000) at db/db_impl.cc:2106
#8 0x000000000053b60f in BGThread (arg=Unhandled dwarf expression opcode 0xf3

) at util/env_posix.cc:1702
#9 rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper (arg=Unhandled dwarf expression opcode 0xf3

here is the options

[WARN] Options.create_if_missing: 1
[WARN] Options.paranoid_checks: 1
[WARN] Options.env: 0x839600
[WARN] Options.info_log: 0x29f8000
[WARN] Options.max_open_files: 5000
[WARN] Options.max_total_wal_size: 0
[WARN] Options.disableDataSync: 0
[WARN] Options.use_fsync: 0
[WARN] Options.max_log_file_size: 536870912
[WARN] Options.max_manifest_file_size: 18446744073709551615
[WARN] Options.log_file_time_to_roll: 0
[WARN] Options.keep_log_file_num: 1000
[WARN] Options.allow_os_buffer: 1
[WARN] Options.allow_mmap_reads: 0
[WARN] Options.allow_mmap_writes: 0
[WARN] Options.create_missing_column_families: 0
[WARN] Options.db_log_dir: ./log/
[WARN] Options.wal_dir: /ssd/data/morpheus/userprofile
[WARN] Options.table_cache_numshardbits: 4
[WARN] Options.delete_obsolete_files_period_micros: 21600000000
[WARN] Options.max_background_compactions: 2
[WARN] Options.max_background_flushes: 2
[WARN] Options.WAL_ttl_seconds: 0
[WARN] Options.WAL_size_limit_MB: 0
[WARN] Options.manifest_preallocation_size: 4194304
[WARN] Options.allow_os_buffer: 1
[WARN] Options.allow_mmap_reads: 0
[WARN] Options.allow_mmap_writes: 0
[WARN] Options.is_fd_close_on_exec: 1
[WARN] Options.stats_dump_period_sec: 600
[WARN] Options.advise_random_on_open: 1
[WARN] Options.db_write_buffer_size: 0
[WARN] Options.access_hint_on_compaction_start: NORMAL
[WARN] Options.use_adaptive_mutex: 0
[WARN] Options.rate_limiter: (nil)
[WARN] Options.bytes_per_sync: 0
[WARN] Options.wal_bytes_per_sync: 0
[WARN] Options.enable_thread_tracking: 0
Compression algorithms supported:
Snappy supported: 1
Zlib supported: 1
Bzip supported: 1
LZ4 supported: 1
Recovering from manifest
--------------- Options for
[WARN] Options.error_if_exists: 0
[WARN] Options.create_if_missing: 1
[WARN] Options.paranoid_checks: 1
[WARN] Options.env: 0x839600
[WARN] Options.info_log: 0x29f8000
[WARN] Options.max_open_files: 5000
[WARN] Options.max_total_wal_size: 0
[WARN] Options.disableDataSync: 0
[WARN] Options.use_fsync: 0
[WARN] Options.max_log_file_size: 536870912
[WARN] Options.max_manifest_file_size: 18446744073709551615
[WARN] Options.log_file_time_to_roll: 0
[WARN] Options.keep_log_file_num: 1000
[WARN] Options.allow_os_buffer: 1
[WARN] Options.allow_mmap_reads: 0
[WARN] Options.allow_mmap_writes: 0
[WARN] Options.create_missing_column_families: 0
[WARN] Options.db_log_dir: ./log/
[WARN] Options.wal_dir: /ssd/data/morpheus/userprofile
[WARN] Options.table_cache_numshardbits: 4
[WARN] Options.delete_obsolete_files_period_micros: 21600000000
[WARN] Options.max_background_compactions: 2
[WARN] Options.max_background_flushes: 2
[WARN] Options.WAL_ttl_seconds: 0
[WARN] Options.WAL_size_limit_MB: 0
[WARN] Options.manifest_preallocation_size: 4194304
[WARN] Options.allow_os_buffer: 1
[WARN] Options.allow_mmap_reads: 0
[WARN] Options.allow_mmap_writes: 0
[WARN] Options.is_fd_close_on_exec: 1
[WARN] Options.stats_dump_period_sec: 600
[WARN] Options.advise_random_on_open: 1
[WARN] Options.db_write_buffer_size: 0
[WARN] Options.access_hint_on_compaction_start: NORMAL
[WARN] Options.use_adaptive_mutex: 0
[WARN] Options.rate_limiter: (nil)
[WARN] Options.bytes_per_sync: 0
[WARN] Options.wal_bytes_per_sync: 0
[WARN] Options.enable_thread_tracking: 0
[WARN] Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator
[WARN] Options.merge_operator: rocks_merge
[WARN] Options.compaction_filter: None
[WARN] Options.compaction_filter_factory: DefaultCompactionFilterFactory
[WARN] Options.compaction_filter_factory_v2: DefaultCompactionFilterFactoryV2
[WARN] Options.memtable_factory: HashLinkListRepFactory
[WARN] Options.table_factory: BlockBasedTable
[WARN] table_factory options:

[WARN] Options.write_buffer_size: 536870912
[WARN] Options.max_write_buffer_number: 4
[WARN] Options.compression[0]: NoCompression
[WARN] Options.compression[1]: Snappy
[WARN] Options.compression[2]: Snappy
[WARN] Options.compression[3]: Zlib
[WARN] Options.compression[4]: Zlib
[WARN] Options.compression[5]: Zlib
[WARN] Options.compression[6]: Zlib
[WARN] Options.prefix_extractor: rocksdb.Noop
[WARN] Options.num_levels: 7
[WARN] Options.min_write_buffer_number_to_merge: 2
[WARN] Options.purge_redundant_kvs_while_flush: 1
[WARN] Options.compression_opts.window_bits: -14
[WARN] Options.compression_opts.level: -1
[WARN] Options.compression_opts.strategy: 0
[WARN] Options.level0_file_num_compaction_trigger: 4
[WARN] Options.level0_slowdown_writes_trigger: 20
[WARN] Options.level0_stop_writes_trigger: 48
[WARN] Options.max_mem_compaction_level: 2
[WARN] Options.target_file_size_base: 134217728
[WARN] Options.target_file_size_multiplier: 1
[WARN] Options.max_bytes_for_level_base: 1073741824
[WARN] Options.level_compaction_dynamic_level_bytes: 0
[WARN] Options.max_bytes_for_level_multiplier: 10
[WARN] Options.max_bytes_for_level_multiplier_addtl[0]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[1]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[2]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[3]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[4]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[5]: 1
[WARN] Options.max_bytes_for_level_multiplier_addtl[6]: 1
[WARN] Options.max_sequential_skip_in_iterations: 8
[WARN] Options.expanded_compaction_factor: 25
[WARN] Options.source_compaction_factor: 1
[WARN] Options.max_grandparent_overlap_factor: 10
[WARN] Options.arena_block_size: 53687091
[WARN] Options.soft_rate_limit: 0.00
[WARN] Options.hard_rate_limit: 0.00
[WARN] Options.rate_limit_delay_max_milliseconds: 1000
[WARN] Options.disable_auto_compactions: 0
[WARN] Options.purge_redundant_kvs_while_flush: 1
[WARN] Options.filter_deletes: 1
[WARN] Options.verify_checksums_in_compaction: 1
[WARN] Options.compaction_style: 0
[WARN] Options.compaction_options_universal.size_ratio: 1
[WARN] Options.compaction_options_universal.min_merge_width: 2
[WARN] Options.compaction_options_universal.max_merge_width: 4294967295
[WARN] Options.compaction_options_universal.max_size_amplification_percent: 200
[WARN] Options.compaction_options_universal.compression_size_percent: -1
[WARN] Options.compaction_options_fifo.max_table_files_size: 1073741824
[WARN] Options.table_properties_collectors:
[WARN] Options.inplace_update_support: 1
[WARN] Options.inplace_update_num_locks: 10000
@superwood
Copy link
Author

while doing Manu compcation ,the memory memory used by rocksdb keeps going up

@sagar0
Copy link
Contributor

sagar0 commented Oct 3, 2017

@superwood I hope your issue is resolved. If not, could you try with one of the latest RocksDB version to see if this is still a problem?
I am closing this issue for now as it is more than an year old. Please feel free to reopen if you think this issue is still relevant and/or has been closed in error.

@sagar0 sagar0 closed this as completed Oct 3, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sagar0 @superwood