Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Compile/Run All Benchmarks #2784

Closed
2 tasks done
dillon-cullinan opened this issue Sep 12, 2019 · 3 comments · Fixed by #9879
Closed
2 tasks done

[FEA] Compile/Run All Benchmarks #2784

dillon-cullinan opened this issue Sep 12, 2019 · 3 comments · Fixed by #9879
Labels
CMake CMake build issue feature request New feature or request
Milestone
C++ Benchmark Run...

Comments

@dillon-cullinan
Copy link
Contributor

dillon-cullinan commented Sep 12, 2019 •
edited by karthikeyann
Loading

Is your feature request related to a problem? Please describe.
Currently all benchmarks must be manually ran after compilation by doing ./<benchmark> --benchmark_out=<benchmark>.json --benchmark_out_format=json after passing -DBUILD_BENCHMARKS=ON to cmake which isn't too friendly for job automation in Jenkins. Source building has to be edited manually on the job currently since I cannot use the root build script.

Describe the solution you'd like

  1. A flag/option in root build.sh to build the benchmarks via this script. All this flag needs to do is pass -DBUILD_BENCHMARKS=ON to libcudf cmake.
  2. A make target for benchmarks that builds/runs all the benchmarks given options described above (output dirname and file type (csv, json, console)). A single output file would be ideal, but outputting several files similar to GTest is fine as well.
@dillon-cullinan dillon-cullinan added feature request New feature or request Needs Triage Need team to review and classify CMake CMake build issue labels Sep 12, 2019
@dillon-cullinan dillon-cullinan mentioned this issue Sep 12, 2019
Merged
@kkraus14 kkraus14 removed the Needs Triage Need team to review and classify label Sep 20, 2019
@karthikeyann
Copy link
Contributor

@robertmaynard is it possible to run benchmarks with make benchmarks or ninja benchmarks ? (or ninja run_benchmarks)
AFAIK, we don't have any such rule now in cmake.

@robertmaynard
Copy link
Contributor

We could write such a rule via a add_custom_target where we have multiple COMMAND inputs which map to each benchmark we have generated

@karthikeyann
Copy link
Contributor

Is there a way to all benchmarks like we run ninja test ?
Perhaps using Ctest scripting https://gitlab.kitware.com/cmake/community/-/wikis/doc/ctest/Scripting-Of-CTest ?

@karthikeyann karthikeyann mentioned this issue Dec 9, 2021
Merged
rapids-bot bot pushed a commit that referenced this issue Dec 10, 2021
@karthikeyann
add run_benchmarks target for running benchmarks with json output (#9879
b545df4 
)

closes #2784
Adds run_benchmarks target for running benchmarks with json output

You can run all benchmarks with  `make run_benchmarks` or `ninja run_benchmarks`
json will be stored in `benchmarks/results/*.json`
The benchmark stdout will still print.

Authors:
  - Karthikeyan (https://github.com/karthikeyann)

Approvers:
  - Robert Maynard (https://github.com/robertmaynard)
  - David Wendt (https://github.com/davidwendt)

URL: #9879
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
CMake CMake build issue feature request New feature or request
Projects
None yet
Milestone
C++ Benchmark Runtime Improvements
Development

Successfully merging a pull request may close this issue.

add run_benchmarks target for running benchmarks with json output karthikeyann/cudf
4 participants
@robertmaynard @kkraus14 @karthikeyann @dillon-cullinan