Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix encode/decode of negative timestamps in ORC reader/writer #11586

Merged
merged 14 commits into from
Sep 13, 2022
Merged

Fix encode/decode of negative timestamps in ORC reader/writer #11586

merged 14 commits into from
Sep 13, 2022

Conversation

vuule
Copy link
Contributor

@vuule vuule commented Aug 24, 2022 •
edited
Loading

Description

Fixes #11525

Contains a chain of fixes:

  1. Allow negative nanoseconds in negative timestamps - aligns writer with pyorc;
  2. Limit seconds adjustment to positive nanoseconds - fixes the off-by-one issue reported in [BUG] ORC read/write is incompatible with Spark in some corner case(s) #11525;
  3. Fix the decode of large uint64_t values (>max int64_t) - fixes reading of cuDF encoded negative nanoseconds;
  4. Avoid mode 2 encode when the base value is larger than max int64_t - follows the specs and fixes reading of negative nanoseconds using non-cuDF readers.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@vuule vuule added bug Something isn't working cuIO cuIO issue non-breaking Non-breaking change labels Aug 24, 2022
@vuule vuule self-assigned this Aug 24, 2022
@github-actions github-actions bot added Python Affects Python cuDF API. libcudf Affects libcudf (C++/CUDA) code. labels Aug 24, 2022
@codecov
Copy link

codecov bot commented Sep 8, 2022 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.10@dca285b). Click here to learn what that means.
Patch has no changes to coverable lines.

❗ Current head c86fa32 differs from pull request most recent head e1a27f1. Consider uploading reports for the commit e1a27f1 to get more accurate results

Additional details and impacted files 
@@               Coverage Diff               @@
##             branch-22.10   #11586   +/-   ##
===============================================
  Coverage                ?   86.42%           
===============================================
  Files                   ?      145           
  Lines                   ?    23009           
  Branches                ?        0           
===============================================
  Hits                    ?    19885           
  Misses                  ?     3124           
  Partials                ?        0

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@vuule vuule marked this pull request as ready for review September 8, 2022 21:00
@vuule vuule requested review from a team as code owners September 8, 2022 21:00
PointKernel
PointKernel approved these changes Sep 9, 2022
View reviewed changes
Copy link
Member

@PointKernel PointKernel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with two non-blocking nits

cpp/src/io/orc/stripe_data.cu Outdated Show resolved Hide resolved
cpp/src/io/orc/stripe_data.cu Outdated Show resolved Hide resolved
cpp/src/io/orc/stripe_data.cu Show resolved Hide resolved
@ttnghia ttnghia mentioned this pull request Sep 12, 2022
Merged
@ttnghia
Copy link
Contributor

ttnghia commented Sep 12, 2022

FYI: I have tested this with Spark and no longer see the issue. Thanks for working on it.

@vuule vuule requested a review from bdice September 12, 2022 20:01
@vuule
Copy link
Contributor Author

vuule commented Sep 12, 2022

rerun tests

bdice
bdice approved these changes Sep 12, 2022
View reviewed changes
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test refactoring is 🔥, nice work!

@vuule
Copy link
Contributor Author

vuule commented Sep 13, 2022

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 7b0d597 into rapidsai:branch-22.10 Sep 13, 2022
@vuule vuule deleted the bug-orc-negative-timestamp branch September 13, 2022 00:10
@pxLi pxLi mentioned this pull request Sep 14, 2022
Closed
@vyasr vyasr added 4 - Needs Review Waiting for reviewer to review or respond and removed 4 - Needs cuDF (Python) Reviewer labels Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PointKernel PointKernel PointKernel approved these changes

@galipremsagar galipremsagar galipremsagar approved these changes

@ttnghia ttnghia ttnghia approved these changes

@bdice bdice bdice approved these changes

@shwina shwina Awaiting requested review from shwina shwina is a code owner automatically assigned from rapidsai/cudf-python-codeowners

@isVoid isVoid Awaiting requested review from isVoid isVoid is a code owner automatically assigned from rapidsai/cudf-python-codeowners

@harrism harrism Awaiting requested review from harrism harrism is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@vuule vuule

Labels
4 - Needs Review Waiting for reviewer to review or respond bug Something isn't working cuIO cuIO issue libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] ORC read/write is incompatible with Spark in some corner case(s)
6 participants
@vuule @ttnghia @bdice @galipremsagar @PointKernel @vyasr