Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Add DELTA_BINARY_PACKED decoding support to Parquet reader #13637

Merged
merged 85 commits into from
Aug 23, 2023
+1,376 −627
Merged

[FEA] Add DELTA_BINARY_PACKED decoding support to Parquet reader #13637

Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
5e9cf26
add DELTA_BINARY_PACKED decoder
etseidl Jun 26, 2023
9326321
start merging in changes from #13622
etseidl Jun 27, 2023
ee7511d
get reduce working on device
etseidl Jun 27, 2023
2cafe62
use functor for transform iterator
etseidl Jun 27, 2023
219ff0b
change filter functors to use kernel_mask
etseidl Jun 27, 2023
0e181a8
pull in changes from #13622
etseidl Jun 28, 2023
129d9ab
Merge branch 'rapidsai:branch-23.08' into feature/delta_binary
etseidl Jun 28, 2023
2b5a25d
Merge branch 'rapidsai:branch-23.08' into feature/delta_binary
etseidl Jun 28, 2023
b58e55c
use less shared memory for delta binary decoder
etseidl Jun 28, 2023
c63e503
Merge branch 'feature/delta_binary' of github.com:etseidl/cudf into f…
etseidl Jun 28, 2023
2c5e087
spelling
etseidl Jun 28, 2023
996893e
change encoding to unsupported type
etseidl Jun 29, 2023
f1f74dc
add python test of delta parser
etseidl Jun 29, 2023
639b8ab
test delta with nulls
etseidl Jun 29, 2023
02cd2be
Merge branch 'rapidsai:branch-23.08' into feature/delta_binary
etseidl Jun 29, 2023
9871d66
Merge branch 'rapidsai:branch-23.08' into feature/delta_binary
etseidl Jun 29, 2023
7debd29
Merge branch 'rapidsai:branch-23.08' into feature/delta_binary
etseidl Jun 29, 2023
c1bbb84
add comments to skip_values and decode_batch
etseidl Jun 30, 2023
8e66a08
revert east volatile changes
etseidl Jun 30, 2023
7b09c4f
update doc string
etseidl Jul 7, 2023
5e05872
Merge branch 'feature/delta_binary' of github.com:etseidl/cudf into f…
etseidl Jul 7, 2023
6576aa3
Merge branch 'rapidsai:branch-23.08' into feature/delta_binary
etseidl Jul 7, 2023
1d6efbc
Merge branch 'branch-23.08' into feature/delta_binary
etseidl Jul 12, 2023
334fb53
Merge branch 'branch-23.08' into feature/delta_binary
vuule Jul 13, 2023
d18dabf
Merge branch 'rapidsai:branch-23.08' into feature/delta_binary
etseidl Jul 17, 2023
05eb40f
Merge branch 'rapidsai:branch-23.08' into feature/delta_binary
etseidl Jul 18, 2023
a957ecc
Merge branch 'branch-23.08' into feature/delta_binary
etseidl Jul 19, 2023
9b636c7
fix for header location
etseidl Jul 19, 2023
6a53d43
Merge branch 'rapidsai:branch-23.08' into feature/delta_binary
etseidl Jul 19, 2023
ceb22ab
fix some short-circuit logic
etseidl Jul 19, 2023
c00be0b
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Jul 25, 2023
e30aa11
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Jul 26, 2023
7ef4be2
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Jul 26, 2023
2410e47
Merge remote-tracking branch 'origin/branch-23.10' into feature/delta…
etseidl Jul 26, 2023
4aa783d
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Jul 28, 2023
94afb8d
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Jul 28, 2023
12b9bab
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Jul 31, 2023
a6f7957
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 1, 2023
b7dbf47
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Aug 2, 2023
ea49a23
rename function
etseidl Aug 2, 2023
3a9f186
clean up kernel_mask_for_page()
etseidl Aug 2, 2023
d7671d7
remove TODO
etseidl Aug 2, 2023
7084d89
add some documentation to kernel_mask_bits
etseidl Aug 2, 2023
f60bc1c
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Aug 2, 2023
9ebeca9
use rand_dataframe() to produce test data
etseidl Aug 3, 2023
07e73ac
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 3, 2023
410ab67
Merge remote-tracking branch 'origin/branch-23.10' into feature/delta…
etseidl Aug 3, 2023
f779455
Merge branch 'branch-23.10' into feature/delta_binary
vuule Aug 3, 2023
761393f
formatting
etseidl Aug 4, 2023
ced688d
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 4, 2023
4846032
implement suggestion from review
etseidl Aug 4, 2023
d615625
more suggestions from review
etseidl Aug 4, 2023
c1ce34c
Merge remote-tracking branch 'origin/branch-23.10' into feature/delta…
etseidl Aug 4, 2023
835e866
restore old unrolled loop for testing
etseidl Aug 5, 2023
6353a4a
add note to revisit bit unpacker with delta_byte_array
etseidl Aug 7, 2023
9ffea01
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 7, 2023
0904418
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Aug 7, 2023
616d3fc
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Aug 8, 2023
e924a97
fix and test int8 and int16 handling
etseidl Aug 8, 2023
d0bf0cd
fix for single row files
etseidl Aug 8, 2023
d3b0c09
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 9, 2023
3c83b89
Merge branch 'branch-23.10' into feature/delta_binary
vuule Aug 9, 2023
5ac20a0
clean up some docstrings
etseidl Aug 9, 2023
9914e1f
Merge remote-tracking branch 'origin/branch-23.10' into feature/delta…
etseidl Aug 14, 2023
7d077f7
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 14, 2023
62e0493
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 15, 2023
5725c61
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Aug 16, 2023
0ab8c15
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Aug 17, 2023
b404de7
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 17, 2023
1d137ec
Apply suggestions from code review
etseidl Aug 17, 2023
6cd3e00
fix docstring
etseidl Aug 17, 2023
a774ac1
need to pass num_threads as template param to make constexpr
etseidl Aug 17, 2023
60b45b3
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 17, 2023
fb44e80
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 18, 2023
1d9cc7d
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Aug 18, 2023
6655e86
more descriptive name and docstring
etseidl Aug 21, 2023
9dbc3ff
Merge remote-tracking branch 'origin/branch-23.10' into feature/delta…
etseidl Aug 21, 2023
ee4385c
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 22, 2023
373bdd9
Apply suggestions from code review
etseidl Aug 22, 2023
67be449
formatting
etseidl Aug 22, 2023
e6068d7
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 22, 2023
bbf957e
Merge branch 'branch-23.10' into feature/delta_binary
etseidl Aug 23, 2023
ed5dcdb
Merge branch 'rapidsai:branch-23.10' into feature/delta_binary
etseidl Aug 23, 2023
eb4ff08
implement suggestion from review
etseidl Aug 23, 2023
9afd0fb
Merge remote-tracking branch 'origin/branch-23.10' into feature/delta…
etseidl Aug 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cpp/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,10 +390,12 @@ add_library(
src/io/orc/writer_impl.cu
src/io/parquet/compact_protocol_reader.cpp
src/io/parquet/compact_protocol_writer.cpp
src/io/parquet/decode_preprocess.cu
src/io/parquet/page_data.cu
src/io/parquet/chunk_dict.cu
src/io/parquet/page_enc.cu
src/io/parquet/page_hdr.cu
src/io/parquet/page_delta_decode.cu
src/io/parquet/page_string_decode.cu
src/io/parquet/predicate_pushdown.cpp
src/io/parquet/reader.cpp
Expand Down
Loading