[FEA] Performance issue with the Parquet reader for very large schemas (especially when containing strings)

[nvdbaranec]

For parquet files that contain very large schemas with strings (either large numbers of columns, or large numbers of nested columns) we pay a very heavy price postprocessing the string data after the core decode kernels runs.

Essentially, the "decode" process for strings is just emitting a large array of pointer/size pairs that are then passed to other cudf functions to reconstruct actual columns. The problem is that we are doing this with no batching - each output string column results in an entire cudf function call (make_strings_column) with multiple internal kernel calls each. In situations with thousands of columns, this gets very expensive.

In the image above, the green span represents the time spent in the decode kernel and the time spent in all of the make_strings_column calls afterwards. The time is totally dominated by the many many calls to make_strings_column (the red span).

Ideally, we would have some kind of batched interface to make_strings_column (make_strings_columns ?) that can do the work for the thousands of output columns coalesced into fewer kernels.

On a related note, the area under the blue line represents a similar problem involving preprocessing the file (thousands of calls to thrust::reduce and thrust::exclusive_scan_by_key). This has been largely addressed by this PR #12931

[etseidl]

@nvdbaranec I'm curious if #13302 had any impact on the file used to generate the profile above.

[GregoryKimball]

@vuule recently conducted some experiments using an internal stream pool to hide latencies during column buffer allocation. Perhaps evaluating string types with larger column counts would show a bigger signal.

[mhaseeb123]

Investigated this issue by profiling the overhead of many make_strings_columns on read_parquet using a custom NVTX range over read_chunk_internal function. Observed that 1024 calls of make_strings_columns take roughly ~1.5ms to complete in the current state which seems negligible compared to the decode time (roughly 1.3ms for table size of 64MB across 1024 cols, 17ms for table size of 1GB across 1024 string cols and 178ms for table size of 12GB across 1024 string cols). This is because since @etseidl's #13303, the make_strings_column simply uses the inline_column_buffer to trivially create the column on the host instead of launching anything on the GPU.

From the investigation, it seems like large number of make_strings_column calls is not a significant performance issue in read_parquet anymore but using a thread pool to create these columns in parallel may help further optimize. @GregoryKimball @nvdbaranec @vuule

[mhaseeb123]

Closing the issue based on above results.