[FEA] The final step of the parquet preprocess step can get very slow for highly complicated schemas.

[nvdbaranec]

For parquet files that contain columns with lists in them, we do a preprocess step to compute column sizes (and other information). As a very last step, we iterate over the input columns and march through them by depth here:

cudf/cpp/src/io/parquet/page_data.cu

Line 1827 in e91d7d9

for (size_t idx = 0; idx < input_columns.size(); idx++) {

For very complicated schemas, this can result in a large number of calls to thrust::reduce() and thrust::exclusive_scan_by_key(). In some cases, cases, so much so that it can dominate the rest of the decompress/decode work.

We should spend some time figuring out how to coalesce this into fewer (perhaps 1 each of reduce_by_key and exclusive_scan_by_key) calls.

[nvdbaranec]

@abellina

[vyasr]

@SrikarVanavasam FYI Github supports certain keywords to automate linking issues to PRs. If you had included Closes #11922 instead of Addresses #11922 in the PR description for #12931 Github would have automatically linked and closed this issue when the PR merged. For future reference 😄