Support nanosecond timestamps in parquet #10063
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PointKernel
added
non-breaking
|
This PR has been labeled |
PointKernel
force-pushed
the
parquet-nanosecond-support
branch
from
6af6cc7 to
2467672
Compare
PointKernel
force-pushed
the
parquet-nanosecond-support
branch
from
98d241b to
3113ede
Compare
PointKernel
added
3 - Ready for Review
devavret
suggested changes
Mar 14, 2022
robertmaynard
approved these changes
Mar 14, 2022
|
@vuule requesting your review as well. |
Codecov Report
@@ Coverage Diff @@
## branch-22.04 #10063 +/- ##
================================================
- Coverage 86.13% 86.01% -0.13%
================================================
Files 139 139
Lines 22438 22435 -3
================================================
- Hits 19328 19298 -30
- Misses 3110 3137 +27
Continue to review full report at Codecov.
|
vuule
requested changes
Mar 18, 2022
vuule
approved these changes
Mar 18, 2022
devavret
approved these changes
Mar 21, 2022
karthikeyann
approved these changes
Mar 21, 2022
|
@gpucibot merge |
rapids-bot bot
pushed a commit
that referenced
this pull request
Mar 22, 2022
…0483) Includes `<cstddef>` for `ptrdiff_t` in `parquet/compact_protocol_reader.hpp`. Compilation fails on GCC 11 without this include. Targeting 22.04 since this was broken yesterday in #10063. Error output: ``` cudf/cpp/src/io/parquet/compact_protocol_reader.hpp:51:17: error: 'ptrdiff_t' does not name a type 51 | [[nodiscard]] ptrdiff_t bytecount() const noexcept { return m_cur - m_base; } | cudf/cpp/src/io/parquet/compact_protocol_reader.hpp:22:1: note: 'ptrdiff_t' is defined in header '<cstddef>'; did you forget to '#include <cstddef>'? ``` Also includes `<optional>` in `cpp/include/cudf/table/experimental/row_operators.cuh`, which was broken by #10164. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Conor Hoekstra (https://github.com/codereport) - Yunsong Wang (https://github.com/PointKernel) URL: #10483
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #9393
This PR was intended to support nanoseconds for both duration and timestamp types in parquet. It introduces
LogicalType-handling on both reader and writer sides. This PR also includes code cleanups like movingCompactProtocolReaderto its own file. Finally, nanosecond durations remain unchanged since it's not fully supported bypyarrowi.e. nanosecond durations are truncated to microseconds (see here).