Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

s2: Add concurrent stream decompression #397

Closed
klauspost opened this issue Jun 11, 2021 · 0 comments · Fixed by #602
Closed

s2: Add concurrent stream decompression #397

klauspost opened this issue Jun 11, 2021 · 0 comments · Fixed by #602
Labels
enhancement

Comments

@klauspost
Copy link
Owner

No description provided.

klauspost added a commit that referenced this issue May 25, 2022
@klauspost
s2: Add concurrent stream decompression
a188606 
Add concurrent stream decompression of snappy/s2 streams

Fixes #397

```
e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost)
λ s2d -bench=3 -cpu=1 enwik10.snappy
Reading enwik10.snappy...
Decompressing... 4340323220 -> 10000000000 [230.40%]; 8.681s, 1098.6MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 8.744s, 1090.7MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 8.711s, 1094.8MB/s

e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost)
λ s2d -bench=3 -cpu=2 enwik10.snappy
Reading enwik10.snappy...
Decompressing... 4340323220 -> 10000000000 [230.40%]; 5.259s, 1813.4MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 5.241s, 1819.8MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 5.255s, 1814.7MB/s

e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost)
λ s2d -bench=3 -cpu=4 enwik10.snappy
Reading enwik10.snappy...
Decompressing... 4340323220 -> 10000000000 [230.40%]; 2.63s, 3625.6MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 2.646s, 3604.8MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 2.644s, 3606.5MB/s

e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost)
λ s2d -bench=3 -cpu=8 enwik10.snappy
Reading enwik10.snappy...
Decompressing... 4340323220 -> 10000000000 [230.40%]; 1.38s, 6910.6MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 1.387s, 6873.4MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 1.387s, 6875.8MB/s

e:\gopath\src\github.com\klauspost\compress\s2\cmd\s2d (master -> klauspost)
λ s2d -bench=3 -cpu=16 enwik10.snappy
Reading enwik10.snappy...
Decompressing... 4340323220 -> 10000000000 [230.40%]; 894ms, 10668.5MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 882ms, 10818.2MB/s
Decompressing... 4340323220 -> 10000000000 [230.40%]; 903ms, 10558.8MB/s
```
@klauspost klauspost mentioned this issue May 25, 2022
Merged
klauspost added a commit that referenced this issue May 26, 2022
@klauspost
s2: Add concurrent stream decompression (#602)
a1f8123 
Add concurrent stream decompression of snappy/s2 streams

Fixes #397

For full stream decompression S2 offers a `DecodeConcurrent` function that will decode a full stream using multiple goroutines.

Example scaling, AMD Ryzen 3950X, 16 cores, decompression using `s2d -bench=3 <input>`, best of 3: 

| Input                                     | `-cpu=1`   | `-cpu=2`   | `-cpu=4`   | `-cpu=8`   | `-cpu=16`   |
|-------------------------------------------|------------|------------|------------|------------|-------------|
| enwik10.snappy                            | 1098.6MB/s | 1819.8MB/s | 3625.6MB/s | 6910.6MB/s | 10818.2MB/s |
| enwik10.s2                                | 1303.5MB/s | 2606.1MB/s | 4847.9MB/s | 8878.4MB/s | 9592.1MB/s  |
| sofia-air-quality-dataset.tar.snappy      | 1302.0MB/s | 2165.0MB/s | 4244.5MB/s | 8241.0MB/s | 12920.5MB/s |
| sofia-air-quality-dataset.tar.s2          | 1399.2MB/s | 2463.2MB/s | 5196.5MB/s | 9639.8MB/s | 11439.5MB/s |
| sofia-air-quality-dataset.tar.s2 (no asm) | 837.5MB/s  | 1652.6MB/s | 3183.6MB/s | 5945.0MB/s | 9620.7MB/s  |

Scaling can be expected to be pretty linear until memory bandwidth is saturated. 

For now the `DecodeConcurrent` can only be used for full streams without seeking or combining with regular reads.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

s2: Add concurrent stream decompression klauspost/compress
1 participant
@klauspost