[BUG] q9 regression between 21.10 and 21.12

[abellina]

We had a baseline of 25s for q9 in 21.10, but we have lost around 1.5 or 2 seconds in 21.12 when running in the spark2a environment.

This is to investigate the differences and figure out if we need to raise an issue in cuDF, the plugin or if there's an environmental issue.

"queryTimes" : [ 26977 ],
"queryTimes" : [ 27164 ],
"queryTimes" : [ 27016 ],
"queryTimes" : [ 26903 ],
"queryTimes" : [ 26542 ],

[abellina]

Note that we have identified other issues with q9 that could yield significant performance improvements: #4189, #4186, #4164.

The issue here is somewhat separate from the other issues, but there may be some parallels.

[jbrennan333]

I believe the difference here may be due to enabling nvcomp compression/decompression by default in CUDF. In branch-22.02, I got times similar to 21.10 when I disabled nvcomp:

spark.executorEnv.LIBCUDF_NVCOMP_POLICY=OFF

Results:

q9-22.02-nonvcomp/tpcds-testjtb_no_nvcomp-gpu-aqe-on-ucx-off-16-cores-decimals-false-q9-1638656812726.json:  "queryTimes" : [ 24593 ],
q9-22.02-nonvcomp/tpcds-testjtb_no_nvcomp-gpu-aqe-on-ucx-off-16-cores-decimals-false-q9-1638656886471.json:  "queryTimes" : [ 25379 ],
q9-22.02-nonvcomp/tpcds-testjtb_no_nvcomp-gpu-aqe-on-ucx-off-16-cores-decimals-false-q9-1638657043868.json:  "queryTimes" : [ 25976 ],

When I run branch-22.02 with default setting (LIBCUDF_NVCOMP_POLICY=STABLE).
I get:

q9-22.02/tpcds-testjtb-gpu-aqe-on-ucx-off-16-cores-decimals-false-q9-1638658371600.json:  "queryTimes" : [ 27203 ],
q9-22.02/tpcds-testjtb-gpu-aqe-on-ucx-off-16-cores-decimals-false-q9-1638658437224.json:  "queryTimes" : [ 27284 ],
q9-22.02/tpcds-testjtb-gpu-aqe-on-ucx-off-16-cores-decimals-false-q9-1638658657588.json:  "queryTimes" : [ 27032 ],

[jbrennan333]

This is the PR for enabling nvcomp in cudf: rapidsai/cudf#9582

[abellina]

Same comment as I had for q88: #4280 (comment)

[jbrennan333]

As noted in #4280, the change to nvcomp as the default snappy compressor is in branch-21.12.

[jbrennan333]

I think the difference reported here is explained by the change to make nvcomp the default snappy compressor in branch-21.12. The nvcomp team is investigating that issue separately. I think we can close this issue - any objections @abellina or @jlowe?

[jlowe]

I think we can close this issue

No objection from me.

[abellina]

+1 closing this.