Use nvcomp's snappy compressor in parquet writer #8229
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Cmake changes (excluding changes needed in nvcomp's cmake) Replace cuIO's snappy compressor with nvcomp
github-actions
bot
added
CMake
devavret
added
improvement
vuule
requested changes
May 12, 2021
github-actions
bot
added
CMake
robertmaynard
requested changes
May 13, 2021
This was referenced May 24, 2021
rapids-bot bot
pushed a commit
that referenced
this pull request
May 26, 2021
Updates the Java bindings to nvcomp to statically link libnvcomp. This will help avoid libnvcomp v1.x and v2.x conflicts when libcudf starts pulling in libnvcomp 2.x as part of #8229. Switching to a statically-linked libnvcomp requires a small patch to the nvcomp source, as it is hard-coded to only produce a shared library. The patch changes the target to a static library compiled with position-independent code so it can be linked into a shared object like libcudfjni.so. Authors: - Jason Lowe (https://github.com/jlowe) Approvers: - Robert (Bobby) Evans (https://github.com/revans2) URL: #8334
When writing statistics, there's not enough space allocated in chunk's compressed buffer. This results in the compressed buffer being written into another chunk's memory.
|
rerun tests |
Codecov Report
@@ Coverage Diff @@
## branch-21.10 #8229 +/- ##
===============================================
Coverage ? 10.81%
===============================================
Files ? 115
Lines ? 18775
Branches ? 0
===============================================
Hits ? 2030
Misses ? 16745
Partials ? 0 Continue to review full report at Codecov.
|
vuule
reviewed
Sep 8, 2021
kkraus14
reviewed
Sep 8, 2021
vuule
requested changes
Sep 8, 2021
vuule
approved these changes
Sep 9, 2021
|
rerun tests |
robertmaynard
approved these changes
Sep 14, 2021
|
@gpucibot merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds nvcomp dependency and uses nvcomp's batched snappy compression functions in parquet writer.
Adds an environment variable
LIBCUDF_USE_NVCOMPto switch between cuIO's internal snappy compressor and nvcomp's compressor.Using nvcomp is disabled by default.
Use
export LIBCUDF_USE_NVCOMP=1to switch to nvcomp's compressor.