feat: port in more from the C++ code #24
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
a10y
commented
Aug 23, 2024
| bytes: [u8; 8], | ||
| num: u64, | ||
| } | ||
| pub struct Symbol(u64); |
There was a problem hiding this comment.
going from union -> newtype struct shows up in the cargo asm. before symbol was getting spilled into stack, now it's passed in registers
a10y
commented
Aug 23, 2024
a10y
commented
Sep 3, 2024
| #[cfg(miri)] | ||
| const MAX_GENERATIONS: usize = 2; | ||
| /// Entrypoint for building a new `Compressor`. | ||
| pub struct CompressorBuilder { |
There was a problem hiding this comment.
much of this is just moving the old methods from Compressor into a new struct
a10y
commented
Sep 3, 2024
a10y
commented
Sep 3, 2024
benches/micro.rs
Outdated
Comment on lines
28
to
54
| group.bench_function("compress-hashtab", |b| { | ||
| // We create a symbol table and an input that will execute exactly one iteration, | ||
| // in the fast compress_word pathway. | ||
| let mut compressor = CompressorBuilder::new(); | ||
| compressor.insert(Symbol::from_slice(b"abcdefgh"), 8); | ||
| let compressor = compressor.build(); | ||
|
|
||
| b.iter(|| unsafe { | ||
| compressor.compress_into( | ||
| b"abcdefghabcdefghabcdefghabcdefghabcdefghabcdefghabcdefghabcdefgh", | ||
| &mut output_buf, | ||
| ); | ||
| }); | ||
| }); | ||
|
|
||
| group.bench_function("compress-twobytes", |b| { | ||
| // We create a symbol table and an input that will execute exactly one iteration, | ||
| // in the fast compress_word pathway. | ||
| let mut compressor = CompressorBuilder::new(); | ||
| compressor.insert(Symbol::from_slice(&[b'a', b'b', 0, 0, 0, 0, 0, 0]), 8); | ||
| let compressor = compressor.build(); | ||
|
|
||
| b.iter(|| unsafe { | ||
| compressor.compress_into(b"abababababababab", &mut output_buf); | ||
| }); | ||
| }); | ||
| group.finish(); |
There was a problem hiding this comment.
These test the relative speeds of the two compression pathways: one where we're able to skip the hashtable lookup and one where we are not. Skipping the lookup is nearly twice as fast
a10y
changed the title
a10y
commented
Sep 3, 2024
| /// a load of `N` values from the pointer in a minimum number of instructions into | ||
| /// an output `u64`. | ||
| #[inline] | ||
| unsafe fn extract_u64<const N: usize>(ptr: *const u8) -> u64 { |
There was a problem hiding this comment.
With better benchmarking it became clear this had no effect on performance VS just doing a copy_non_overlapping, and was less clear in general
Merged
a10y
pushed a commit
that referenced
this pull request
Sep 3, 2024
## 🤖 New release * `fsst-rs`: 0.2.3 -> 0.3.0 <details><summary><i><b>Changelog</b></i></summary><p> <blockquote> ## [0.3.0](v0.2.3...v0.3.0) - 2024-09-03 ### Added - port in more from the C++ code ([#24](#24)) ### Other - centering ([#26](#26)) </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/MarcoIeni/release-plz/). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR ports in some more functionality based on the MIT-licensed C++ code from CWI.
In particular, it implements the following:
makeSamplefunction from C++ to build a sample of ~16KB from the input datasuffix limitoptimization and its correspondingfinalizemethod needed when building the symbol table, including changes to thecompress_wordfunction we have that more directly corresponds to thecompressVariantfrom the C++ codebyteCodesfrom C++, which we implement here ascodes_one_byte. Note that before this PR, one-byte codes would not be found unless the byte occurred at the end of the plaintext stringCompressorbuild state into a newCompressorBuilderstruct, which has all methods that take&mut self. This also means that we can in theory construct aCompressornow from a symbol table, though that logic is not implemented.Additional things in this PR:
compress_wordmethod comparing the relative speeds of both code paths, see feat: port in more from the C++ code #24 (comment)dbtextcompression benchmarks from the CWI paper. Here's a table of the compression factors:I'll follow up to figure out how to close the gap with those 1-2% differences