Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow the user to reduce RAM usage #30

Closed
romanz opened this issue Jul 17, 2018 · 7 comments
Closed

Allow the user to reduce RAM usage #30

romanz opened this issue Jul 17, 2018 · 7 comments
Assignees
@romanz
Labels
enhancement New feature or request

Comments

@romanz
Copy link
Owner

romanz commented Jul 17, 2018 •
edited
Loading

It seems that most of the memory is used by RocksDB caches.

(see #11 for more details)
@romanz romanz added the enhancement New feature or request label Jul 17, 2018
@greenaddress greenaddress mentioned this issue Jul 22, 2018
Closed
@romanz romanz self-assigned this Jul 22, 2018
romanz referenced this issue Jul 26, 2018
@romanz
Explicitly specify 'max_open_files' to reduce RAM usage
fcb926a
@romanz
Copy link
Owner Author

romanz commented Jul 26, 2018 •
edited
Loading

Apparently, setting max_open_files = 2048 reduces steady-state RAM usage to ~160MB (from ~550MB).

@romanz
Copy link
Owner Author

romanz commented Jul 26, 2018 •
edited
Loading

Initial indexing with --skip-bulk-import takes ~250MB-400MB of RAM (at least for the first ~250K blocks).

@romanz
Copy link
Owner Author

romanz commented Aug 2, 2018 •
edited
Loading

Late phase of initial indexing and DB compaction seem to use significant amounts of RAM: #38 (comment)
It needs to be profiled and optimized for RAM (since currently it's optimized for speed).

@romanz
Copy link
Owner Author

romanz commented Aug 7, 2018

Running cb6196d with:

  • max_open_files=16
  • compaction_readahead_size=512K
  • write_buffer_size=128M
  • block_size=512K
  • --index-batch-size=10 --skip-bulk-import

selection_101

Indexing is using less than 400MB of RAM :)
Compaction is using more than 1GB of RAM :(

@romanz
Copy link
Owner Author

romanz commented Aug 10, 2018 •
edited
Loading

Running 9dcec98 with:

  • --index-batch-size=10
  • --skip-bulk-import

selection_102

Both indexing and compaction take now up to ~550MB of RAM :)

@romanz romanz changed the title Allow the user to control RAM usage Allow the user to reduce RAM usage Aug 11, 2018
@romanz romanz mentioned this issue Aug 29, 2018
Open
@romanz
Copy link
Owner Author

romanz commented Sep 10, 2018

Related to #8

@romanz
Copy link
Owner Author

romanz commented Sep 16, 2018

Merge to master at #58

@romanz romanz closed this as completed Sep 16, 2018
@dpc dpc mentioned this issue Mar 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@romanz romanz

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@romanz