Support for ZLIB compression in ORC writer #11036
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vuule
added
feature request
github-actions
bot
added
Python
Codecov Report
@@ Coverage Diff @@
## branch-22.08 #11036 +/- ##
===============================================
Coverage ? 86.34%
===============================================
Files ? 144
Lines ? 22738
Branches ? 0
===============================================
Hits ? 19632
Misses ? 3106
Partials ? 0 Continue to review full report at Codecov.
|
vuule
commented
Jun 4, 2022
| @@ -1233,7 +1232,7 @@ __global__ void __launch_bounds__(1024) | |||
| ? statuses[ss.first_block + b].bytes_written | |||
| : src_len; | |||
| uint32_t blk_size24{}; | |||
| if (dst_len >= src_len) { | |||
| if (statuses[ss.first_block + b].status == 0) { | |||
There was a problem hiding this comment.
Previous condition was implicit, led to wrong output when the compressed data is larger.
vuule
commented
Jun 4, 2022
vuule
requested review from
galipremsagar,
isVoid,
robertmaynard and
karthikeyann
Fixed this issue when I switched to 22.06 and back to 22.08 as base. Also updated reviewers. |
Co-authored-by: Karthikeyan <[email protected]>
github-actions
bot
removed
gpuCI
CMake
|
rerun tests |
1 similar comment
|
rerun tests |
|
@gpucibot merge |
rapids-bot bot
pushed a commit
that referenced
this pull request
Jun 20, 2022
Some recently merged PRs (#11018 + #11036) do not include enough header which may cause compile error in some systems (in particular, CUDA 11.7 + gcc-11.2). This PR adds the missing header (`<optional>`) to fix the compile issue. Authors: - Nghia Truong (https://github.com/ttnghia) Approvers: - Karthikeyan (https://github.com/karthikeyann) - Yunsong Wang (https://github.com/PointKernel) URL: #11126
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #11023
Expands the nvcomp adapter to cover compression. Supports SNAPPY and ZLIB.
Moves the ORC writer nvcomp compression from
stripe_enc.cuto the adapter and add ZLIB support.Adds padding to compressed blocks in writer to ensure alignment of output pointer alignment that is required for nvcomp ZLIB compression.
Minor changes:
batched_argsso it also makes sense for compression.