Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix config set compression type didn't take effect #1576

Merged
merged 11 commits into from
Jul 16, 2023

Conversation

xiaobiaozhao
Copy link
Contributor

@xiaobiaozhao xiaobiaozhao commented Jul 10, 2023

In this PR, we also refactor the c-style ConfigEnum array into the std::vector, the main reason is to make it easier to adopt dynamic config enums like the compression types. And for the test case, it's a bit hard to test if the compression type takes effect in underlying storage because we only enabled it when the levels >= 2. But will check it manually to make sure if works.

This closes #1560

src/storage/storage.cc Outdated Show resolved Hide resolved
@aleksraiden aleksraiden changed the title Enable dynamin compression Enable dynamic compression Jul 11, 2023
@git-hulk git-hulk changed the title Enable dynamic compression Fix config set compression type didn't take effect Jul 11, 2023
src/storage/storage.cc Outdated Show resolved Hide resolved
src/config/config.cc Outdated Show resolved Hide resolved
src/config/config.h Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
Copy link
Member

@PragmaTwice PragmaTwice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code does not look well to me, some fixes need to be pushed

src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
@caipengbo
Copy link
Contributor

I think the SetCompressionOption parameter value is the name of the compression algorithm. We get rocksdb::CompressionType by map[name](name->CompressionType), and then pass it to SetOptions, in SetOptions static_cast it to rocksdb::CompressionType.

@git-hulk
Copy link
Member

@PragmaTwice @xiaobiaozhao @caipengbo I minor refactor the PR, PTAL again.

@git-hulk git-hulk marked this pull request as draft July 14, 2023 11:57
@git-hulk
Copy link
Member

git-hulk commented Jul 14, 2023

I have tested if the config set command works on my side:

Set the compression to LZ4:

2023/07/14-20:45:08.518174 6158610432 [db/db_impl/db_impl.cc:1195] SetOptions() on column family [default], inputs:
2023/07/14-20:45:08.518210 6158610432 [db/db_impl/db_impl.cc:1198] compression_per_level: kNoCompression:kNoCompression:kLZ4Compression:kLZ4Compression:kLZ4Compression:kLZ4Compression:kLZ4Compression

then use redis-benchmark -p 6666 -n 1000000 -r 5000000 -t set -d 1024 to write the data, and check the db LOG to see if the compression was used:

2023/07/14-20:50:15.181736 6156890112 (Original Log Time 2023/07/14-20:50:15.181594) [db/compaction/compaction_job.cc:888] [metadata] compacted to: files[0 1 13 0 0 0 0] 
max score 0.93, MB/sec: 39.1 rd, 4.5 wr, level 2, files in(1, 1) out(1 +0 blob) MB 
in(58.0, 2.6 +0.0 blob) out(7.1 +0.0 blob), read-write-amplify(1.2) write-amplify(0.1) OK, 
records in: 89927, records dropped: 3779 output_compression: LZ4

@git-hulk git-hulk marked this pull request as ready for review July 14, 2023 12:53
@git-hulk git-hulk self-requested a review July 14, 2023 14:46
src/config/config.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
caipengbo
caipengbo previously approved these changes Jul 16, 2023
Copy link
Contributor

@caipengbo caipengbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@git-hulk git-hulk merged commit b67f268 into apache:unstable Jul 16, 2023
git-hulk added a commit to git-hulk/kvrocks that referenced this pull request Jul 30, 2023
@git-hulk git-hulk mentioned this pull request Jul 31, 2023
git-hulk added a commit that referenced this pull request Aug 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants