-
Notifications
You must be signed in to change notification settings - Fork 246
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance monitoring; GitHub integration #1472
Performance monitoring; GitHub integration #1472
Conversation
This patch introduces two new testing targets to the verification suite based on a small configuration based on the `benchmark` regression test. The profile test is saved a `p0` in `.testing`. Future tests can be included if appropriate. The new targets: * `make profile`: Run the model and record the FMS timings. * `make perf`: Run the model through the `perf` tool and record timings for the resolvable functions (as symbols). In both cases, the timings are converted to JSON output files and the top results are reported to stdout, and readable in GitHub actions output. It can also be run locally. Support Python scripts have been included to do this work. This will require a functional Python environment. Some system and configuration data is logged alongside the timings, but this is still rather incomplete and needs some further planning. Times are compared to the target build (usually dev/gfdl). ANSI terminal highlighting (i.e. color) is to used to highlight excessive differences. Current issues: - Model configuration - GitHub timings are still rather unreliable, and should currently only be treated as crude estimates. This should be considered a work in progress. - The GitHub profiling rule still builds the standard configuration, evem though it is unused. - Additional tools are required to push the timings to some database, either a local sqlite3 or an external one.
Although this work does not yet resemble what we would like to see, I think it's is developed enough to at least be included in the repository. Future considerations include the following:
|
Codecov Report
@@ Coverage Diff @@
## dev/gfdl #1472 +/- ##
=========================================
Coverage 29.13% 29.13%
=========================================
Files 235 235
Lines 71061 71061
=========================================
Hits 20707 20707
Misses 50354 50354 Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks reasonable enough, and it has passed the TC testing and the pipeline testing at https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/pipelines/13469.
This patch introduces two new testing targets to the verification suite
based on a small configuration based on the
benchmark
regression test.The profile test is saved as
p0
in.testing
. Future tests can beincluded if appropriate.
The new targets:
make profile
: Run the model and record the FMS timings.make perf
: Run the model through theperf
tool and record timingsfor the resolvable functions (as symbols).
In both cases, the timings are converted to JSON output files and the
top results are reported to stdout, and readable in GitHub actions
output. It can also be run locally.
Support Python scripts have been included to do this work. This will
require a functional Python environment.
Some system and configuration data is logged alongside the timings, but
this is still rather incomplete and needs some further planning.
Times are compared to the target build (usually dev/gfdl). ANSI
terminal highlighting (i.e. color) is to used to highlight excessive
differences.
Current issues:
Model configuration
GitHub timings are still rather unreliable, and should currently only
be treated as crude estimates. This should be considered a work in
progress.
The GitHub profiling rule still builds the standard configuration,
evem though it is unused.
Additional tools are required to push the timings to some database,
either a local sqlite3 or an external one.