[FEA] Use nvbench for new benchmarks and transition old benchmarks to use nvbench

[jrhemstad]

Is your feature request related to a problem? Please describe.

nvbench is a new library for writing benchmarks of GPU accelerated applications. It's similar to Google Benchmark, but has several quality of life improvements when doing GPU benchmarking, e.g., it eliminates the need for us to do manual timing in GBench with the raii_event_timer object. It also can show things like % of peak BW achieved and other details about the GPU hardware.

Describe the solution you'd like

We have quite a few benchmarks right now, so it would take a while to transition everything to use nvbench. That said, any new benchmarks can and should be written with nvbench.

[harrism]

Oops, I've added several new benchmarks in the past two weeks before I saw this.

[harrism]

I think before we can switch nvbench would need a replacement for gbench's compare.py, which we rely on heavily.

[jrhemstad]

I think before we can switch nvbench would need a replacement for gbench's compare.py, which we rely on heavily.

Yeah, @allisonvacanti has that on the P0 list already: NVIDIA/nvbench#9

[vyasr]

I think before we can switch nvbench would need a replacement for gbench's compare.py, which we rely on heavily.

Yeah, @allisonvacanti has that on the P0 list already: NVIDIA/nvbench#9

Do we know what next steps are on that? @shwina and I worked on getting a version of compare.py ready and it was merged a while ago, but I don't recall how much more work is necessary to bring it to feature parity with gbench's compare.py so that we can close NVIDIA/nvbench#9. I can follow up if we don't.

[karthikeyann]

I think before we can switch nvbench would need a replacement for gbench's compare.py, which we rely on heavily.

This is implemented in NVIDIA/nvbench#14

[karthikeyann]

Basic version is fine. Documentation is required. CLI could be improved.
benchmarksfiltered feature might be needed.

[vyasr]

@harrism @jrhemstad Now that we're trying to get nightly benchmarks going I think we'll need to prioritize this conversion so that we can standardize the data that will be available and start building more tooling for our dashboards. I'm going to follow up on the comparison script work now to see what next steps are there.
CC @GregoryKimball @randerzander @robertmaynard

[bdice]

I updated our Unit Benchmarking in libcudf Guide to reflect the policy that new benchmarks should use NVBench in PR #10093. I think most of the information there still holds.

[vyasr]

At this point PR reviewers are reasonably consistent at requiring that new benchmark code use NVbench or convert existing benchmarks. We should continue to encourage that so that we achieve a gradual transition. The infrastructure for NVbench is also all in place, and the benchmarking docs indicate that NVbench is our preferred tool. If we decide to make a concerted push to convert all remaining benchmarks to NVBench, we can reopen this issue or open a new one as a checklist of files to convert, but for now I'm going to close this as it's not particularly actionable.