Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Adaptively Decide Compression Algorithm for integers data type #8982

Closed
Lloyd-Pottiger opened this issue Apr 24, 2024 · 0 comments · Fixed by #8903, #9293, #9299, #9312 or #9416
Closed

Support Adaptively Decide Compression Algorithm for integers data type #8982

Lloyd-Pottiger opened this issue Apr 24, 2024 · 0 comments · Fixed by #8903, #9293, #9299, #9312 or #9416
Assignees
@Lloyd-Pottiger
Labels
type/feature-request Categorizes issue or PR as related to a new feature.

Comments

@Lloyd-Pottiger
Copy link
Contributor

Feature Request

Is your feature request related to a problem? Please describe:

Currently, TiFlash only supports general compression LZ4, ZSTD, and DeflateQpl (not available in the release version). It is decided by the configuration file, which means all data in the same TiFlash instance will use the same compression algorithm.

By detecting specific patterns, specialized compression algorithms can be significantly more lightweight, providing much faster compression and decompression. In addition, they can be effective on much smaller data sizes. This allows us to decompress a few rows at a time, rather than requiring large blocks of data to be decompressed at once. These specialized compression algorithms can also offer efficient support for random seeks, making data access through an index significantly faster.

Describe the feature you'd like:

Support more lightweight compression algorithms and adaptively decide compression algorithms for each data part in TiFlash.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:
@Lloyd-Pottiger Lloyd-Pottiger added the type/feature-request Categorizes issue or PR as related to a new feature. label Apr 24, 2024
@Lloyd-Pottiger Lloyd-Pottiger mentioned this issue Apr 24, 2024
Merged
12 tasks
@Lloyd-Pottiger Lloyd-Pottiger self-assigned this Apr 26, 2024
@Lloyd-Pottiger Lloyd-Pottiger mentioned this issue Apr 26, 2024
Closed
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue May 8, 2024
@Lloyd-Pottiger
Compression: support FOR & replace Delta with DeltaFOR (#8983)
2251037 
ref #8982
This was referenced May 8, 2024
Merged
Closed
@ti-chi-bot ti-chi-bot bot closed this as completed in #8903 Jul 8, 2024
@ti-chi-bot ti-chi-bot bot closed this as completed in 1ad8a21 Jul 8, 2024
@JaySon-Huang JaySon-Huang changed the title Support Adaptively Decide Compression Algorithm Support Adaptively Decide Compression Algorithm for integers data type Jul 8, 2024
This was referenced Aug 6, 2024
Closed
Merged
ti-chi-bot bot pushed a commit that referenced this issue Aug 7, 2024
@Lloyd-Pottiger
Compression: add microbenchmark (#9293)
16fb1d9 
close #8982

Signed-off-by: Lloyd-Pottiger <[email protected]>
@Lloyd-Pottiger Lloyd-Pottiger mentioned this issue Aug 7, 2024
Merged
12 tasks
guo-shaoge pushed a commit to guo-shaoge/tiflash that referenced this issue Aug 7, 2024
@Lloyd-Pottiger @guo-shaoge
Compression: add microbenchmark (pingcap#9293)
c3be1a9 
close pingcap#8982

Signed-off-by: Lloyd-Pottiger <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue Aug 12, 2024
@Lloyd-Pottiger
Compression: optimize the CompressionCodecFactory (#9299)
5e9e6c2 
close #8982

Compression: optimize the CompressionCodecFactory

Signed-off-by: Lloyd-Pottiger <[email protected]>
@Lloyd-Pottiger Lloyd-Pottiger mentioned this issue Aug 12, 2024
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 6, 2024
@Lloyd-Pottiger @JinheLin
Compression: Support enabling lightweight compression (#9312)
627fd1f 
close #8982, close #9336

Compression: Support enabling lightweight compression

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: jinhelin <[email protected]>
@Lloyd-Pottiger Lloyd-Pottiger mentioned this issue Sep 9, 2024
Merged
12 tasks
ti-chi-bot bot added a commit that referenced this issue Sep 11, 2024
@Lloyd-Pottiger @JaySon-Huang @ti-chi-bot
Compression: enable lightweight temporarily (#9416)
305dba4 
close #8982

Compression: enable lightweight temporarily

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: JaySon <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@Lloyd-Pottiger Lloyd-Pottiger mentioned this issue Sep 24, 2024
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 26, 2024
@Lloyd-Pottiger
Compression: still using LZ4 by default (#9468)
d91bcde 
close #8982

Compression: still using LZ4 by default

Signed-off-by: Lloyd-Pottiger <[email protected]>
@Lloyd-Pottiger Lloyd-Pottiger mentioned this issue Oct 25, 2024
Draft
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Lloyd-Pottiger Lloyd-Pottiger

Labels
type/feature-request Categorizes issue or PR as related to a new feature.
Projects
None yet
Milestone
No milestone
1 participant
@Lloyd-Pottiger