Code

Refactor Parquet reader#12046

Code

Merged

rapids-bot[bot] merged 172 commits intorapidsai:branch-22.12rapidsai/cudf:branch-22.12from ttnghia:refactor_parquet_readerttnghia/cudf:refactor_parquet_readerCopy head branch name to clipboardNov 10, 2022

+2,179-2,029

Commits

Commits on Sep 23, 2022

• Fix an issue where using num_rows and skip_rows on a parquet file containing a mix of nested and non-nested types would

  Show description for c2e8e87
  nvdbaranec

  committed
  c2e8e87

• Merge branch 'branch-22.12' into reader_preprocess_fix_and_opt

  nvdbaranec

  committed
  f330431

Commits on Sep 27, 2022

• Fixed an issue with the tests: input columns cannot have unsanitary lists. Fixed an additional issue in the decoding where flat column types underneath

  Show description for eadfd63
  nvdbaranec

  committed
  eadfd63

• Merge branch 'branch-22.12' into reader_preprocess_fix_and_opt

  nvdbaranec

  committed
  c4de038

Commits on Sep 30, 2022

• Copy parquet_reader_* into chunked_parquet_reader_*

  Show description for 222c9fe
  ttnghia

  committed
  222c9fe

• Modify chunked_parquet_reader_options

  Show description for f49cfed
  ttnghia

  committed
  f49cfed

Commits on Oct 4, 2022

• Exploit inheritance to extend the options and options_builder classes

  Show description for dd39804
  ttnghia

  committed
  dd39804

Commits on Oct 5, 2022

• Remove unnecessary variable

  Show description for 81bc68f
  ttnghia

  committed
  81bc68f

• Misc

  Show description for f8126be
  ttnghia

  committed
  f8126be

• Add docs

  Show description for 0e7692c
  ttnghia

  committed
  0e7692c

• PR feedback changes.

  nvdbaranec

  committed
  9f9eeb0

• Merge branch 'branch-22.12' into reader_preprocess_fix_and_opt

  nvdbaranec

  committed
  9b3ea62

• Fixed some compile errors from merging.

  nvdbaranec

  committed
  d2e409a

• Add chunked_parquet_reader

  Show description for ed41ac1
  ttnghia

  committed
  ed41ac1

• Add empty implementation

  Show description for be782f2
  ttnghia

  committed
  be782f2

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  7908b66

• Add a destructor and close

  Show description for a7175c8
  ttnghia

  committed
  a7175c8

Commits on Oct 6, 2022

• Update docs

  Show description for 63a7bd6
  ttnghia

  committed
  63a7bd6

• Fix comment

  Show description for 16c12d9
  ttnghia

  committed
  16c12d9

• Construct chunked_parquet_reader

  Show description for cd85385
  ttnghia

  committed
  cd85385

• Add comment

  Show description for 5944beb
  ttnghia

  committed
  5944beb

Commits on Oct 7, 2022

• Rename function and implementing

  Show description for 7cfa72a
  ttnghia

  committed
  7cfa72a

• MISC

  Show description for 4696bd3
  ttnghia

  committed
  4696bd3

Commits on Oct 10, 2022

• Bare bones implementation. Many types still not working.

  nvdbaranec

  committed
  99dc786

Commits on Oct 11, 2022

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  ad9c399

• Merge branch 'chunked_reader_gpu' into parquet_reader

  Show description for ecf225d
  ttnghia

  committed
  ecf225d

• Add test

  Show description for 583a7ef
  ttnghia

  committed
  583a7ef

• Cleanup

  Show description for b250d6f
  ttnghia

  committed
  b250d6f

• Modify docs

  Show description for e7a9e3e
  ttnghia

  committed
  e7a9e3e

• Cleanup

  Show description for 811354a
  ttnghia

  committed
  811354a

• Add TODO

  Show description for 12ba72e
  ttnghia

  committed
  12ba72e

• Add read_intermediate_data

  Show description for 45668ff
  ttnghia

  committed
  45668ff

• Use read_intermediate_data

  Show description for 1bb8254
  ttnghia

  committed
  1bb8254

Commits on Oct 12, 2022

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  b1c44dd

• Fix bug

  Show description for 56715ef
  ttnghia

  committed
  56715ef

• Simplify code

  Show description for a7e7e93
  ttnghia

  committed
  a7e7e93

• Implement file_intermediate_data

  Show description for 8fe87b1
  ttnghia

  committed
  8fe87b1

• Add make_output

  Show description for 464f4f9
  ttnghia

  committed
  464f4f9

• Implement read_chunk

  Show description for 56756d6
  ttnghia

  committed
  56756d6

• Cleanup

  Show description for 3044ac5
  ttnghia

  committed
  3044ac5

• Fix bug when skip_rows and num_rows are modified inside a called function

  Show description for ffb8a19
  ttnghia

  committed
  ffb8a19

Commits on Oct 13, 2022

• Fix comment

  Show description for baf3603
  ttnghia

  committed
  baf3603

• Store preprocess data

  Show description for 8bdab44
  ttnghia

  committed
  8bdab44

• Implement chunked_reader detail class

  Show description for ec4abfb
  ttnghia

  committed
  ec4abfb

• Refactoring

  Show description for cb1dea4
  ttnghia

  committed
  cb1dea4

• Rename structs

  Show description for a8dfd82
  ttnghia

  committed
  a8dfd82

• Increment current_read_chunk

  Show description for 7889e5a
  ttnghia

  committed
  7889e5a

• Call preprocessing in read_chunk

  Show description for 63a6511
  ttnghia

  committed
  63a6511

Commits on Oct 14, 2022

• Fix has_next

  Show description for c1269d1
  ttnghia

  committed
  c1269d1

• Refactoring

  Show description for 95e6c1d
  ttnghia

  committed
  95e6c1d

• Fix errors

  Show description for bd7b510
  ttnghia

  committed
  bd7b510

Commits on Oct 17, 2022

• Merge branch 'branch-22.12' into chunked_reader_gpu. Also: work towards string columns.

  nvdbaranec

  committed
  eb78526

Commits on Oct 18, 2022

• Change param

  Show description for 66aeaf4
  ttnghia

  committed
  66aeaf4

• Rename variables

  Show description for 1d700e3
  ttnghia

  committed
  1d700e3

• Remove intermediate variables

  Show description for 4af948b
  ttnghia

  committed
  4af948b

• Modify tests

  Show description for 28cfc6f
  ttnghia

  committed
  28cfc6f

• First pass of string support.

  nvdbaranec

  committed
  8135ed5

• Fix bug

  Show description for fbeabfc
  ttnghia

  committed
  fbeabfc

• Remove debug print

  Show description for df074e0
  ttnghia

  committed
  df074e0

• Merge branch 'chunked_reader_gpu' into parquet_reader

  Show description for 5653090
  ttnghia

  committed
  5653090

• Merge branch 'branch-22.12' into parquet_reader

  Show description for 3071fd7
  ttnghia

  committed
  3071fd7

• Fix tests

  Show description for 631eff1
  ttnghia

  committed
  631eff1

• Fix chunk size limit

  Show description for d1b4e4c
  ttnghia

  committed
  d1b4e4c

• Turn back to do preprocess once

  Show description for 3f2f8a4
  ttnghia

  committed
  3f2f8a4

Commits on Oct 19, 2022

• The read limit parameter is now no longer const but truely runtime parameter

  Show description for 974e7ef
  ttnghia

  committed
  974e7ef

• Add new test file

  Show description for 0be096b
  ttnghia

  committed
  0be096b

• Reverse parquet_test.cpp

  Show description for f7018fe
  ttnghia

  committed
  f7018fe

• Modify read to add exception and preprocess once

  Show description for 81097eb
  ttnghia

  committed
  81097eb

• Rewrite tests

  Show description for fcffac8
  ttnghia

  committed
  fcffac8

• Store decomp_page_data

  Show description for 43dd802
  ttnghia

  committed
  43dd802

• Rewrite tests

  Show description for eeec023
  ttnghia

  committed
  eeec023

• Simple test

  Show description for 14dfd3f
  ttnghia

  committed
  14dfd3f

• Store raw_page_data

  Show description for 66e9f09
  ttnghia

  committed
  66e9f09

• Cleanup test

  Show description for 669b8cf
  ttnghia

  committed
  669b8cf

• Fix empty output

  Show description for 001c6c7
  ttnghia

  committed
  001c6c7

• Add preprocess_file_and_columns

  Show description for f50603a
  ttnghia

  committed
  f50603a

• Misc

  Show description for 66976aa
  ttnghia

  committed
  66976aa

• Fixed some incorrect logic in preprocess tep.

  nvdbaranec

  committed
  0b0040a

• Removed debug stuff. Added some comments.

  nvdbaranec

  committed
  467de78

Commits on Oct 20, 2022

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  d68bf80

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  1888aa3

• Change function

  Show description for 5861747
  ttnghia

  committed
  5861747

• Disable debug printing

  Show description for 721c052
  ttnghia

  committed
  721c052

• Fixed an issue with non-first reads in the chunked reader. Made an attempt to cleanup and make several pieces of logic

  Show description for 7cda8c2
  nvdbaranec

  committed
  7cda8c2

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  6bc073d

• Fix off-by-one bug

  Show description for 2aef5cc
  ttnghia

  committed
  2aef5cc

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  89ca1a6

• Fix an issue related to aliased output pointers in the chunked read case. We were checking for nullptr to indicate whether or not

  Show description for 5245b9b
  nvdbaranec

  committed
  5245b9b

Commits on Oct 21, 2022

• Do not keep reference---copy object instead

  ttnghia

  committed
  62283c7

Commits on Oct 24, 2022

• Optimization: don't do any decoding or page size computation for pages that have been trimmed.

  nvdbaranec

  committed
  44424a2

• Merge branch 'branch-22.12' into chunked_reader_gpu

  nvdbaranec

  committed
  95df356

• Fix build issue for spark-rapids-jni

  nvdbaranec

  committed
  445db9b

• Cleanup: Remove chunked_parquet_reader_options and chunked_parquet_reader_options_builder

  Show description for db56908
  ttnghia

  committed
  db56908

• Move preprocess_file into reader_preprocess.cu

  Show description for ef5eaee
  ttnghia

  committed
  ef5eaee

• Move common implementation into reader_impl_helpers.*

  Show description for d19260d
  ttnghia

  committed
  d19260d

Commits on Oct 25, 2022

• Cleanup

  Show description for 6569d62
  ttnghia

  committed
  6569d62

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  0a1e2c3

• More cleanup

  Show description for 04e1320
  ttnghia

  committed
  04e1320

• Rewrite docs for parquet.hpp files

  Show description for 395413d
  ttnghia

  committed
  395413d

• Extract functions for reader and chunked_reader

  Show description for e3e19e8
  ttnghia

  committed
  e3e19e8

• Fix issues with string length computation.

  nvdbaranec

  committed
  52339da

• Merge branch 'nghia_parquet_reader' into chunked_reader_gpu

  nvdbaranec

  committed
  f7e8694

• Remove redundant changes

  Show description for 83fa31a
  ttnghia

  committed
  83fa31a

• Add simple structs test

  Show description for 02ccdec
  ttnghia

  committed
  02ccdec

• Rewrite tests

  Show description for ee0ffad
  ttnghia

  committed
  ee0ffad

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  09a89e4

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  034a5b7

• Add lists test

  Show description for c149d64
  ttnghia

  committed
  c149d64

• MISC

  Show description for 9335bb7
  ttnghia

  committed
  9335bb7

• Cleanup comments

  Show description for 3769fff
  ttnghia

  committed
  3769fff

• Construct output table metadata just once

  Show description for dc9ef5c
  ttnghia

  committed
  dc9ef5c

Commits on Oct 26, 2022

• Construct _output_columns just once

  Show description for 0366b7a
  ttnghia

  committed
  0366b7a

• Remove options member variable

  Show description for ea2fe9c
  ttnghia

  committed
  ea2fe9c

• Make the chunked_read_limit a soft limit - if we can't find a split, do the best we can. Fix a potential out of bounds

  Show description for 1804056
  nvdbaranec

  committed
  1804056

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  5671c36

• Add tests for structs of lists and lists of structs

  Show description for 3d6e13b
  ttnghia

  committed
  3d6e13b

• Fixed an issue in split generation code causing indexing off the end of the input array. Added some specific page boundary check tests.

  nvdbaranec

  committed
  826c46f

• Merge branch 'chunked_reader_gpu' into parquet_reader

  ttnghia

  committed
  36c1972

• Merge branch 'nghia_parquet_reader' into chunked_reader_gpu

  nvdbaranec

  committed
  d32c602

• Just reformat

  Show description for 88ca034
  ttnghia

  committed
  88ca034

• Change variable names in tests

  Show description for d76ee06
  ttnghia

  committed
  d76ee06

Commits on Oct 27, 2022

• Merge branch 'nghia_parquet_reader' into chunked_reader_gpu

  nvdbaranec

  committed
  cf7b786

• Optimization: store off global nesting sizes per page so that during trim passes we only have to fully process

  Show description for b806323
  nvdbaranec

  committed
  b806323

• Adding doxygen, refactoring and cleaning up

  Show description for 0c2178d
  ttnghia

  committed
  0c2178d

• Merge branch 'branch-22.12' into parquet_reader

  ttnghia

  committed
  f759103

• Fixed issues with list, and validity size calculations.

  nvdbaranec

  committed
  c2bf7f5

• More refactoring

  Show description for e7e74c5
  ttnghia

  committed
  e7e74c5

• Merge branch 'chunked_reader_gpu' into parquet_reader

  Show description for ee9edbc
  ttnghia

  committed
  ee9edbc

• Add more tests

  Show description for 6fd3e90
  ttnghia

  committed
  6fd3e90

Commits on Oct 28, 2022

• Add test with empty data

  Show description for 9dda980
  ttnghia

  committed
  9dda980

• Add tests

  Show description for bb35f9f
  ttnghia

  committed
  bb35f9f

• Rewrite null tests

  Show description for 59166cf
  ttnghia

  committed
  59166cf

• Add more extreme tests

  Show description for eb6e996
  ttnghia

  committed
  eb6e996

• Rewrite tests to generate input files just once

  Show description for 1c56794
  ttnghia

  committed
  1c56794

• Fix tests with structs of lists

  Show description for 4777419
  ttnghia

  committed
  4777419

• Handle nulls for more complex types

  Show description for aedc37a
  ttnghia

  committed
  aedc37a

• Fix another nulls handling bug for strings

  Show description for c00eb3c
  ttnghia

  committed
  c00eb3c

• Simplify the null purging process

  Show description for 4d24f88
  ttnghia

  committed
  4d24f88

Commits on Oct 31, 2022

• Cleanup.

  nvdbaranec

  committed
  b15bb39

• Merge branch 'nghia_parquet_reader' into chunked_reader_gpu

  nvdbaranec

  committed
  7e38a56

• Fleshed out list-of-structs and struct-of-lists tests.

  nvdbaranec

  committed
  321815d

• Docs and cleanup.

  nvdbaranec

  committed
  8cf95e8

Commits on Nov 1, 2022

• Update doxygen

  Show description for af35c4d
  ttnghia

  committed
  af35c4d

• Remove new test file

  Show description for e31a767
  ttnghia

  committed
  e31a767

• Remove chunked reader

  Show description for a232241
  ttnghia

  committed
  a232241

• Remove chunked reading

  Show description for c3d0097
  ttnghia

  committed
  c3d0097

• Reversing changes...

  Show description for e7808f9
  ttnghia

  committed
  e7808f9

• Fix page decoding

  Show description for 9da46c7
  ttnghia

  committed
  9da46c7

• Remove chunk_intermediate_data

  Show description for c77893e
  ttnghia

  committed
  c77893e

• Cleanup

  Show description for f63b0b5
  ttnghia

  committed
  f63b0b5

• Merge branch 'branch-22.12' into refactor_parquet_reader

  Show description for 78cd349
  ttnghia

  committed
  78cd349

Commits on Nov 2, 2022

• Correct const

  Show description for fd4df91
  ttnghia

  committed
  fd4df91

• Move allocate_nesting_info

  Show description for 8198ce8
  ttnghia

  committed
  8198ce8

• Add doxygen

  Show description for 2cc8235
  ttnghia

  committed
  2cc8235

• Clean up reader_impl.hpp

  Show description for aa47f23
  ttnghia

  committed
  aa47f23

• Add doxygen

  Show description for f560ab7
  ttnghia

  committed
  f560ab7

• Add const

  Show description for e7f80ec
  ttnghia

  committed
  e7f80ec

• Type alias for load_and_decompress_data

  Show description for 1c0308a
  ttnghia

  committed
  1c0308a

• Revert "Move allocate_nesting_info"

  Show description for c791faa
  ttnghia

  committed
  c791faa

• Cleanup allocate_nesting_info

  Show description for 1503674
  ttnghia

  committed
  1503674

• Further cleanup

  Show description for 5440662
  ttnghia

  committed
  5440662

• Cleanup headers

  Show description for 78e3f10
  ttnghia

  committed
  78e3f10

Commits on Nov 3, 2022

• Merge branch 'branch-22.12' into refactor_parquet_reader

  ttnghia

  committed
  cbdaf52

• Extract device code out of parquet_gpu.hpp

  Show description for c35aebe
  ttnghia

  committed
  c35aebe

Commits on Nov 4, 2022

• Change std::vector<> const& to span<>

  Show description for aa64f81
  ttnghia

  committed
  aa64f81

• Merge branch 'branch-22.12' into refactor_parquet_reader

  ttnghia

  committed
  a4547bf

• Merge branch 'branch-22.12' into refactor_parquet_reader

  ttnghia

  committed
  e85b1da

Commits on Nov 7, 2022

• Merge branch 'branch-22.12' into refactor_parquet_reader

  ttnghia

  committed
  2a37e5c

Commits on Nov 8, 2022

• Remove using namespace

  Show description for 453a1b7
  ttnghia

  committed
  453a1b7

• Add nodiscard and fix const

  Show description for bdb15be
  ttnghia

  committed
  bdb15be

• Fix input check

  Show description for dcdeee8
  ttnghia

  committed
  dcdeee8

Commits on Nov 10, 2022

• Merge branch 'branch-22.12' into refactor_parquet_reader

  ttnghia

  committed
  72c888f