Code

Implement chunked Parquet reader#11867

Code

Merged

rapids-bot[bot] merged 181 commits intorapidsai:branch-22.12rapidsai/cudf:branch-22.12from ttnghia:parquet_readerttnghia/cudf:parquet_readerCopy head branch name to clipboardNov 18, 2022

+2,320-351

Commits

Commits on Sep 23, 2022

• Fix an issue where using num_rows and skip_rows on a parquet file containing a mix of nested and non-nested types would

  Show description for c2e8e87
  nvdbaranec

  committed
  c2e8e87

• Merge branch 'branch-22.12' into reader_preprocess_fix_and_opt

  nvdbaranec

  committed
  f330431

Commits on Sep 27, 2022

• Fixed an issue with the tests: input columns cannot have unsanitary lists. Fixed an additional issue in the decoding where flat column types underneath

  Show description for eadfd63
  nvdbaranec

  committed
  eadfd63

• Merge branch 'branch-22.12' into reader_preprocess_fix_and_opt

  nvdbaranec

  committed
  c4de038

Commits on Sep 30, 2022

• Copy parquet_reader_* into chunked_parquet_reader_*

  Show description for 222c9fe
  ttnghia

  committed
  222c9fe

• Modify chunked_parquet_reader_options

  Show description for f49cfed
  ttnghia

  committed
  f49cfed

Commits on Oct 4, 2022

• Exploit inheritance to extend the options and options_builder classes

  Show description for dd39804
  ttnghia

  committed
  dd39804

Commits on Oct 5, 2022

• Remove unnecessary variable

  Show description for 81bc68f
  ttnghia

  committed
  81bc68f

• Misc

  Show description for f8126be
  ttnghia

  committed
  f8126be

• Add docs

  Show description for 0e7692c
  ttnghia

  committed
  0e7692c

• PR feedback changes.

  nvdbaranec

  committed
  9f9eeb0

• Merge branch 'branch-22.12' into reader_preprocess_fix_and_opt

  nvdbaranec

  committed
  9b3ea62

• Fixed some compile errors from merging.

  nvdbaranec

  committed
  d2e409a

• Add chunked_parquet_reader

  Show description for ed41ac1
  ttnghia

  committed
  ed41ac1

• Add empty implementation

  Show description for be782f2
  ttnghia

  committed
  be782f2

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  7908b66

• Add a destructor and close

  Show description for a7175c8
  ttnghia

  committed
  a7175c8

Commits on Oct 6, 2022

• Update docs

  Show description for 63a7bd6
  ttnghia

  committed
  63a7bd6

• Fix comment

  Show description for 16c12d9
  ttnghia

  committed
  16c12d9

• Construct chunked_parquet_reader

  Show description for cd85385
  ttnghia

  committed
  cd85385

• Add comment

  Show description for 5944beb
  ttnghia

  committed
  5944beb

Commits on Oct 7, 2022

• Rename function and implementing

  Show description for 7cfa72a
  ttnghia

  committed
  7cfa72a

• MISC

  Show description for 4696bd3
  ttnghia

  committed
  4696bd3

Commits on Oct 10, 2022

• Bare bones implementation. Many types still not working.

  nvdbaranec

  committed
  99dc786

Commits on Oct 11, 2022

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  ad9c399

• Merge branch 'chunked_reader_gpu' into parquet_reader

  Show description for ecf225d
  ttnghia

  committed
  ecf225d

• Add test

  Show description for 583a7ef
  ttnghia

  committed
  583a7ef

• Cleanup

  Show description for b250d6f
  ttnghia

  committed
  b250d6f

• Modify docs

  Show description for e7a9e3e
  ttnghia

  committed
  e7a9e3e

• Cleanup

  Show description for 811354a
  ttnghia

  committed
  811354a

• Add TODO

  Show description for 12ba72e
  ttnghia

  committed
  12ba72e

• Add read_intermediate_data

  Show description for 45668ff
  ttnghia

  committed
  45668ff

• Use read_intermediate_data

  Show description for 1bb8254
  ttnghia

  committed
  1bb8254

Commits on Oct 12, 2022

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  b1c44dd

• Fix bug

  Show description for 56715ef
  ttnghia

  committed
  56715ef

• Simplify code

  Show description for a7e7e93
  ttnghia

  committed
  a7e7e93

• Implement file_intermediate_data

  Show description for 8fe87b1
  ttnghia

  committed
  8fe87b1

• Add make_output

  Show description for 464f4f9
  ttnghia

  committed
  464f4f9

• Implement read_chunk

  Show description for 56756d6
  ttnghia

  committed
  56756d6

• Cleanup

  Show description for 3044ac5
  ttnghia

  committed
  3044ac5

• Fix bug when skip_rows and num_rows are modified inside a called function

  Show description for ffb8a19
  ttnghia

  committed
  ffb8a19

Commits on Oct 13, 2022

• Fix comment

  Show description for baf3603
  ttnghia

  committed
  baf3603

• Store preprocess data

  Show description for 8bdab44
  ttnghia

  committed
  8bdab44

• Implement chunked_reader detail class

  Show description for ec4abfb
  ttnghia

  committed
  ec4abfb

• Refactoring

  Show description for cb1dea4
  ttnghia

  committed
  cb1dea4

• Rename structs

  Show description for a8dfd82
  ttnghia

  committed
  a8dfd82

• Increment current_read_chunk

  Show description for 7889e5a
  ttnghia

  committed
  7889e5a

• Call preprocessing in read_chunk

  Show description for 63a6511
  ttnghia

  committed
  63a6511

Commits on Oct 14, 2022

• Fix has_next

  Show description for c1269d1
  ttnghia

  committed
  c1269d1

• Refactoring

  Show description for 95e6c1d
  ttnghia

  committed
  95e6c1d

• Fix errors

  Show description for bd7b510
  ttnghia

  committed
  bd7b510

Commits on Oct 17, 2022

• Merge branch 'branch-22.12' into chunked_reader_gpu. Also: work towards string columns.

  nvdbaranec

  committed
  eb78526

Commits on Oct 18, 2022

• Change param

  Show description for 66aeaf4
  ttnghia

  committed
  66aeaf4

• Rename variables

  Show description for 1d700e3
  ttnghia

  committed
  1d700e3

• Remove intermediate variables

  Show description for 4af948b
  ttnghia

  committed
  4af948b

• Modify tests

  Show description for 28cfc6f
  ttnghia

  committed
  28cfc6f

• First pass of string support.

  nvdbaranec

  committed
  8135ed5

• Fix bug

  Show description for fbeabfc
  ttnghia

  committed
  fbeabfc

• Remove debug print

  Show description for df074e0
  ttnghia

  committed
  df074e0

• Merge branch 'chunked_reader_gpu' into parquet_reader

  Show description for 5653090
  ttnghia

  committed
  5653090

• Merge branch 'branch-22.12' into parquet_reader

  Show description for 3071fd7
  ttnghia

  committed
  3071fd7

• Fix tests

  Show description for 631eff1
  ttnghia

  committed
  631eff1

• Fix chunk size limit

  Show description for d1b4e4c
  ttnghia

  committed
  d1b4e4c

• Turn back to do preprocess once

  Show description for 3f2f8a4
  ttnghia

  committed
  3f2f8a4

Commits on Oct 19, 2022

• The read limit parameter is now no longer const but truely runtime parameter

  Show description for 974e7ef
  ttnghia

  committed
  974e7ef

• Add new test file

  Show description for 0be096b
  ttnghia

  committed
  0be096b

• Reverse parquet_test.cpp

  Show description for f7018fe
  ttnghia

  committed
  f7018fe

• Modify read to add exception and preprocess once

  Show description for 81097eb
  ttnghia

  committed
  81097eb

• Rewrite tests

  Show description for fcffac8
  ttnghia

  committed
  fcffac8

• Store decomp_page_data

  Show description for 43dd802
  ttnghia

  committed
  43dd802

• Rewrite tests

  Show description for eeec023
  ttnghia

  committed
  eeec023

• Simple test

  Show description for 14dfd3f
  ttnghia

  committed
  14dfd3f

• Store raw_page_data

  Show description for 66e9f09
  ttnghia

  committed
  66e9f09

• Cleanup test

  Show description for 669b8cf
  ttnghia

  committed
  669b8cf

• Fix empty output

  Show description for 001c6c7
  ttnghia

  committed
  001c6c7

• Add preprocess_file_and_columns

  Show description for f50603a
  ttnghia

  committed
  f50603a

• Misc

  Show description for 66976aa
  ttnghia

  committed
  66976aa

• Fixed some incorrect logic in preprocess tep.

  nvdbaranec

  committed
  0b0040a

• Removed debug stuff. Added some comments.

  nvdbaranec

  committed
  467de78

Commits on Oct 20, 2022

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  d68bf80

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  1888aa3

• Change function

  Show description for 5861747
  ttnghia

  committed
  5861747

• Disable debug printing

  Show description for 721c052
  ttnghia

  committed
  721c052

• Fixed an issue with non-first reads in the chunked reader. Made an attempt to cleanup and make several pieces of logic

  Show description for 7cda8c2
  nvdbaranec

  committed
  7cda8c2

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  6bc073d

• Fix off-by-one bug

  Show description for 2aef5cc
  ttnghia

  committed
  2aef5cc

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  89ca1a6

• Fix an issue related to aliased output pointers in the chunked read case. We were checking for nullptr to indicate whether or not

  Show description for 5245b9b
  nvdbaranec

  committed
  5245b9b

Commits on Oct 21, 2022

• Do not keep reference---copy object instead

  ttnghia

  committed
  62283c7

Commits on Oct 24, 2022

• Optimization: don't do any decoding or page size computation for pages that have been trimmed.

  nvdbaranec

  committed
  44424a2

• Merge branch 'branch-22.12' into chunked_reader_gpu

  nvdbaranec

  committed
  95df356

• Fix build issue for spark-rapids-jni

  nvdbaranec

  committed
  445db9b

• Cleanup: Remove chunked_parquet_reader_options and chunked_parquet_reader_options_builder

  Show description for db56908
  ttnghia

  committed
  db56908

• Move preprocess_file into reader_preprocess.cu

  Show description for ef5eaee
  ttnghia

  committed
  ef5eaee

• Move common implementation into reader_impl_helpers.*

  Show description for d19260d
  ttnghia

  committed
  d19260d

Commits on Oct 25, 2022

• Cleanup

  Show description for 6569d62
  ttnghia

  committed
  6569d62

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  0a1e2c3

• More cleanup

  Show description for 04e1320
  ttnghia

  committed
  04e1320

• Rewrite docs for parquet.hpp files

  Show description for 395413d
  ttnghia

  committed
  395413d

• Extract functions for reader and chunked_reader

  Show description for e3e19e8
  ttnghia

  committed
  e3e19e8

• Fix issues with string length computation.

  nvdbaranec

  committed
  52339da

• Merge branch 'nghia_parquet_reader' into chunked_reader_gpu

  nvdbaranec

  committed
  f7e8694

• Remove redundant changes

  Show description for 83fa31a
  ttnghia

  committed
  83fa31a

• Add simple structs test

  Show description for 02ccdec
  ttnghia

  committed
  02ccdec

• Rewrite tests

  Show description for ee0ffad
  ttnghia

  committed
  ee0ffad

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  09a89e4

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  034a5b7

• Add lists test

  Show description for c149d64
  ttnghia

  committed
  c149d64

• MISC

  Show description for 9335bb7
  ttnghia

  committed
  9335bb7

• Cleanup comments

  Show description for 3769fff
  ttnghia

  committed
  3769fff

• Construct output table metadata just once

  Show description for dc9ef5c
  ttnghia

  committed
  dc9ef5c

Commits on Oct 26, 2022

• Construct _output_columns just once

  Show description for 0366b7a
  ttnghia

  committed
  0366b7a

• Remove options member variable

  Show description for ea2fe9c
  ttnghia

  committed
  ea2fe9c

• Make the chunked_read_limit a soft limit - if we can't find a split, do the best we can. Fix a potential out of bounds

  Show description for 1804056
  nvdbaranec

  committed
  1804056

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  5671c36

• Add tests for structs of lists and lists of structs

  Show description for 3d6e13b
  ttnghia

  committed
  3d6e13b

• Fixed an issue in split generation code causing indexing off the end of the input array. Added some specific page boundary check tests.

  nvdbaranec

  committed
  826c46f

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  36c1972

• Merge branch 'nghia_parquet_reader' into chunked_reader_gpu

  nvdbaranec

  committed
  d32c602

• Just reformat

  Show description for 88ca034
  ttnghia

  committed
  88ca034

• Change variable names in tests

  Show description for d76ee06
  ttnghia

  committed
  d76ee06

Commits on Oct 27, 2022

• Merge branch 'nghia_parquet_reader' into chunked_reader_gpu

  nvdbaranec

  committed
  cf7b786

• Optimization: store off global nesting sizes per page so that during trim passes we only have to fully process

  Show description for b806323
  nvdbaranec

  committed
  b806323

• Adding doxygen, refactoring and cleaning up

  Show description for 0c2178d
  ttnghia

  committed
  0c2178d

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  f759103

• Fixed issues with list, and validity size calculations.

  nvdbaranec

  committed
  c2bf7f5

• More refactoring

  Show description for e7e74c5
  ttnghia

  committed
  e7e74c5

• Merge branch 'chunked_reader_gpu' into parquet_reader

  Show description for ee9edbc
  ttnghia

  committed
  ee9edbc

• Add more tests

  Show description for 6fd3e90
  ttnghia

  committed
  6fd3e90

Commits on Oct 28, 2022

• Add test with empty data

  Show description for 9dda980
  ttnghia

  committed
  9dda980

• Add tests

  Show description for bb35f9f
  ttnghia

  committed
  bb35f9f

• Rewrite null tests

  Show description for 59166cf
  ttnghia

  committed
  59166cf

• Add more extreme tests

  Show description for eb6e996
  ttnghia

  committed
  eb6e996

• Rewrite tests to generate input files just once

  Show description for 1c56794
  ttnghia

  committed
  1c56794

• Fix tests with structs of lists

  Show description for 4777419
  ttnghia

  committed
  4777419

• Handle nulls for more complex types

  Show description for aedc37a
  ttnghia

  committed
  aedc37a

• Fix another nulls handling bug for strings

  Show description for c00eb3c
  ttnghia

  committed
  c00eb3c

• Simplify the null purging process

  Show description for 4d24f88
  ttnghia

  committed
  4d24f88

Commits on Oct 31, 2022

• Cleanup.

  nvdbaranec

  committed
  b15bb39

• Merge branch 'nghia_parquet_reader' into chunked_reader_gpu

  nvdbaranec

  committed
  7e38a56

• Fleshed out list-of-structs and struct-of-lists tests.

  nvdbaranec

  committed
  321815d

• Docs and cleanup.

  nvdbaranec

  committed
  8cf95e8

Commits on Nov 1, 2022

• Update doxygen

  Show description for af35c4d
  ttnghia

  committed
  af35c4d

Commits on Nov 2, 2022

• Cleaning up

  Show description for fb1bd73
  ttnghia

  committed
  fb1bd73

• Add doxygen

  Show description for 842f9ea
  ttnghia

  committed
  842f9ea

• Clean up reader_impl.hpp

  Show description for 34e3777
  ttnghia

  committed
  34e3777

• More cleanup

  Show description for 40e463c
  ttnghia

  committed
  40e463c

• Cleanup allocate_nesting_info

  Show description for 7252b0d
  ttnghia

  committed
  7252b0d

• Reformat

  Show description for 696182c
  ttnghia

  committed
  696182c

• Further cleanup

  Show description for 4c353fd
  ttnghia

  committed
  4c353fd

Commits on Nov 3, 2022

• Rename compute_chunk_read_info into preprocess_pages

  Show description for 31590cb
  ttnghia

  committed
  31590cb

Commits on Nov 10, 2022

• Re-adding an optimization that somehow got nuked during a merge.

  nvdbaranec

  committed
  d91c690

• Optimization: store off global nesting sizes per page so that during trim passes we only have to fully process

  Show description for c5e73ce
  ttnghia

  committed
  c5e73ce

• Merge branch 'branch-22.12' into parquet_reader

  Show description for 0b31b95
  ttnghia

  committed
  0b31b95

• Merge branch 'chunked_reader_gpu' into parquet_reader

  Show description for c8e34ba
  ttnghia

  committed
  c8e34ba

Commits on Nov 14, 2022

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  450e6d5

• Fix several warnings that show up in the spark-rapids-jni build.

  nvdbaranec

  committed
  4fca0c0

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  fd9280c

Commits on Nov 15, 2022

• Several changes from PR review.

  nvdbaranec

  committed
  cedcc07

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  16f78b6

• Fix typo

  Show description for 73503bb
  ttnghia

  committed
  73503bb

• Fix test

  Show description for e9905b8
  ttnghia

  committed
  e9905b8

• Address review comments

  Show description for 4072a80
  ttnghia

  committed
  4072a80

• Small optimization

  Show description for 95a97fb
  ttnghia

  committed
  95a97fb

• Implement column_buffer::empty_like

  Show description for 8390d4b
  ttnghia

  committed
  8390d4b

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  b4b131f

• Optimize unit tests: Only call cudf::concatenate once

  Show description for 5035bf4
  ttnghia

  committed
  5035bf4

• Remove redundant check

  Show description for 912d86c
  ttnghia

  committed
  912d86c

• Fix cudaLaunchKernel error in DecodePageData

  Show description for bb2e26e
  ttnghia

  committed
  bb2e26e

• Add assertion to make sure not to decode/parse empty page array

  Show description for 36c7ec2
  ttnghia

  committed
  36c7ec2

Commits on Nov 16, 2022

• Address some review comments

  Show description for 7203c67
  ttnghia

  committed
  7203c67

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  520448b

• Fix #endif

  Show description for 96eed8e
  ttnghia

  committed
  96eed8e

• Merge branch 'branch-22.12' into parquet_reader

  Show description for 6697e3b
  ttnghia

  committed
  6697e3b

Commits on Nov 17, 2022

• PR review changes. Updated some incorrect/incomplete function docs.

  nvdbaranec

  committed
  70f4fde

• Made the logic in the row_total_size functor much more readable.

  nvdbaranec

  committed
  4547483

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  fe7d6d1

• Fix the tests

  Show description for db21bc3
  ttnghia

  committed
  db21bc3

• Variable renaming for clarity.

  nvdbaranec

  committed
  36043d8

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  83f0703

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  3499bda