Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] ORC ZLIB tests fail with nvCOMP 2.4 #11812

Closed
vuule opened this issue Sep 28, 2022 · 1 comment · Fixed by #12077
Closed

[BUG] ORC ZLIB tests fail with nvCOMP 2.4 #11812

vuule opened this issue Sep 28, 2022 · 1 comment · Fixed by #12077
Assignees
@vuule
Labels
2 - In Progress Currently a work in progress bug Something isn't working cuIO cuIO issue

Comments

@vuule
Copy link
Contributor

vuule commented Sep 28, 2022

nvCOMP introduced changes to the DEFLATE compress/decompress (for example, decompress chunk size is limited to 64KB). ORC tests now fail, further triage is required to root cause.
@vuule vuule added bug Something isn't working Needs Triage Need team to review and classify cuIO cuIO issue labels Sep 28, 2022
@GregoryKimball GregoryKimball added 2 - In Progress Currently a work in progress and removed Needs Triage Need team to review and classify labels Oct 21, 2022
@vuule
Copy link
Contributor Author

vuule commented Oct 21, 2022

Dug into this further. The output from nvcomp 2.4 differs from the output we get from the internal deflate decompression, in some cases. Decompressed output buffer is much smaller than the correct output and the content differs (not just cropped).
Both the input and output are smaller then 64KB, so the size limit should not be the issue here.

Provided repro steps to nvcomp team.

@vuule vuule mentioned this issue Nov 4, 2022
Merged
3 tasks
@vuule vuule self-assigned this Nov 8, 2022
rapids-bot bot pushed a commit that referenced this issue Nov 11, 2022
@vuule
Fix alignment of compressed blocks in ORC writer (#12077)
d335aa3 
Closes #11812
Fixed alignment of compressed blocks in ORC writer - impacted ZLIB compression.
Re-enabled nvCOMP DEFLATE compression in ORC - nvCOMP 2.5+ only.

Refactored the nvCOMP feature status(enabled/disabled in cuDF) checks to include reason why features are not enabled (if not enabled).
Refactored call sites to return the detailed error message if an operation fails because of nvCOMP integration config.
Refactored nvCOMP adapter macros to allow mocking of the parameters that determine if an nvCOMP feature is enabled (env var, GPU compute capability, nvCOMP version).
Added tests to verify the logic of the newly refactored feature status checks (allowed by the mocking above).
Fix a Parquet test that was calling ORC reader/writer 😬

Authors:
  - Vukasin Milovanovic (https://github.com/vuule)

Approvers:
  - Jim Brennan (https://github.com/jbrennan333)
  - Mike Wilson (https://github.com/hyperbolic2346)
  - Bradley Dice (https://github.com/bdice)

URL: #12077
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vuule vuule

Labels
2 - In Progress Currently a work in progress bug Something isn't working cuIO cuIO issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix alignment of compressed blocks in ORC writer vuule/cudf
2 participants
@GregoryKimball @vuule