[FEA] build cudf jars with NVTX enabled

[tgravescs]

Is your feature request related to a problem? Please describe.
Right now if you want to get an nvtx trace you have to go build cudf yourself. This is very time consuming if you are just using one of the base versions. It would be really nice if we can either have another set of jars with it enabled or if we can just enable it in our regular builds.

I know it was shut off because there was performance hit, do we need to revisit that to make sure actually a problem.

[revans2]

In the past I saw what I thought was a very significant performance difference when running some queries. I tried to reproduce this by running TPCH q1 at scale factor 100 against 4 different cudf-0.19 jars each compiled slightly differently.

I compiled one like I normally do for development (NVTX enabled and RMM_LOGGING set to the default)
I complied a separate one with NVTX disabled
Another with NVTX enabled and RMM_LOGGING turned off
and finally the last one I downloaded from the nightly build repo that has both RMM logging disabled and NVTX disabled.

I ran the test using the benchmark tool included in the integration tests jar. The only thing that I changed between the runs was the cudf jar.

$ egrep 'queryTimes|class' *.json
tpch-q1-parquet-1613596781989.json:  "queryTimes" : [ 146709, 144134 ],
tpch-q1-parquet-1613597155688.json:  "queryTimes" : [ 145989, 144112 ],
tpch-q1-parquet-1613602719033.json:  "queryTimes" : [ 146123, 143993 ],
tpch-q1-parquet-1613603478326.json:  "queryTimes" : [ 146179, 144077 ],

To me the only difference in the times that I can see appears to be within the noise between runs. I will try with a few more queries to see if I can find a difference at all. @rongou when you first asked to disabled NVTX in CUDF what was the query that you were running that showed a big enough difference that you wanted to disable it?

[revans2]

I ran all of the TPCH queries with the jar from the nightly build and for my build. All of the hot runs were within what appeared to be noise. The largest difference was for Q21 in which the jar with RMM logging and NVTX enabled happened to be 3.7 seconds (1.7%) faster. My guess is that it was a hiccup.

Unless I can get a reproducible use case I am going to assume that NVTX enabled vs disabled has no impact on real world run times. But I will leave this alone for a while to see if someone else can come up with a query where it does have a measurable impact.

[rongou]

I'm not sure it had a big effect on query performance to begin with, I think I was just trying to eliminate variables that might have affected PTDS. The option cudf had for turning off NVTX didn't work, so my fix was mainly about correctness. I believe the NVTX api itself has also gone through some major changes, so it's quite possible currently there is no perceivable performance difference with it on or off.

[revans2]

@rongou thanks, That is what we heard from the developer of the API. They expected the overhead to be a function call followed by a single global variable lookup. That is what we are seeing in practice, so to make life simpler for customers I think we want to turn on NVTX by default in our builds.

[abellina]

+1 on this

[sameerz]

@tgravescs can this be closed?

[tgravescs]

yes its turned on in rapids build