Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to enable RocksDB ZSTD compression to save 15% storage space/IO #230

Open
eugenesan opened this issue Jan 16, 2024 · 0 comments

Comments

@eugenesan
Copy link

eugenesan commented Jan 16, 2024

I've been experimenting with enabling ZSTD compression and I beehive ZSTD should be offered as an option for those who might benefit from it.

Benefits:

  1. DB size is 15% smaller (as of Jan 2024 fully synced ZSTD compressed DB is 112GB while uncompressed one ~134GB)
  2. Reduced I/O which might help with slower SSDs
  3. Depending on host, sync speed shouldn't increase much if at all (on i5-6200u with 8GB RAM, sync took 29H)

I am not asking to enable it by default (as I did in the patch below) but having a config option for people to utilize would be really nice.

diff --git a/src/Storage.cpp b/src/Storage.cpp
index 8ee8a52..1001eb2 100644
--- a/src/Storage.cpp
+++ b/src/Storage.cpp
@@ -1806,7 +1806,7 @@ void Storage::startup()
         opts.error_if_exists = false;
         opts.max_open_files = options->db.maxOpenFiles <= 0 ? -1 : options->db.maxOpenFiles; ///< this affects memory usage see: https://github.com/facebook/rocksdb/issues/4112
         opts.keep_log_file_num = options->db.keepLogFileNum;
-        opts.compression = rocksdb::CompressionType::kNoCompression; // for now we test without compression. TODO: characterize what is fastest and best..
+        opts.compression = rocksdb::CompressionType::kZSTD; // for now we test without compression. TODO: characterize what is fastest and best..
         opts.use_fsync = options->db.useFsync; // the false default is perfectly safe, but Jt asked for this as an option, so here it is.
 
         shistOpts = opts; // copy what we just did (will implicitly copy over the shared table_factory and write_buffer_manager)
@@ -1834,7 +1834,7 @@ void Storage::startup()
             Debug() << "DB \"" << name << "\" mem: " << QString::number(mem / 1024. / 1024., 'f', 2) << " MiB";
             opts.OptimizeLevelStyleCompaction(mem);
             for (auto & comp : opts.compression_per_level)
-                comp = rocksdb::CompressionType::kNoCompression; // paranoia -- enforce no compression since our data compresses so poorly
+                comp = rocksdb::CompressionType::kZSTD; // paranoia -- enforce no compression since our data compresses so poorly
             memTotal += mem;
             rocksdb::Status s;
             // try and open database
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant