Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance monitoring; GitHub integration #1472

Merged
merged 3 commits into from
Aug 27, 2021

Conversation

marshallward
Copy link
Collaborator

This patch introduces two new testing targets to the verification suite
based on a small configuration based on the benchmark regression test.

The profile test is saved as p0 in .testing. Future tests can be
included if appropriate.

The new targets:

  • make profile: Run the model and record the FMS timings.

  • make perf: Run the model through the perf tool and record timings
    for the resolvable functions (as symbols).

In both cases, the timings are converted to JSON output files and the
top results are reported to stdout, and readable in GitHub actions
output. It can also be run locally.

Support Python scripts have been included to do this work. This will
require a functional Python environment.

Some system and configuration data is logged alongside the timings, but
this is still rather incomplete and needs some further planning.

Times are compared to the target build (usually dev/gfdl). ANSI
terminal highlighting (i.e. color) is to used to highlight excessive
differences.

Current issues:

  • Model configuration

  • GitHub timings are still rather unreliable, and should currently only
    be treated as crude estimates. This should be considered a work in
    progress.

  • The GitHub profiling rule still builds the standard configuration,
    evem though it is unused.

  • Additional tools are required to push the timings to some database,
    either a local sqlite3 or an external one.

This patch introduces two new testing targets to the verification suite
based on a small configuration based on the `benchmark` regression test.

The profile test is saved a `p0` in `.testing`.  Future tests can be
included if appropriate.

The new targets:

* `make profile`: Run the model and record the FMS timings.

* `make perf`: Run the model through the `perf` tool and record timings
  for the resolvable functions (as symbols).

In both cases, the timings are converted to JSON output files and the
top results are reported to stdout, and readable in GitHub actions
output.  It can also be run locally.

Support Python scripts have been included to do this work.  This will
require a functional Python environment.

Some system and configuration data is logged alongside the timings, but
this is still rather incomplete and needs some further planning.

Times are compared to the target build (usually dev/gfdl).  ANSI
terminal highlighting (i.e. color) is to used to highlight excessive
differences.

Current issues:

- Model configuration

- GitHub timings are still rather unreliable, and should currently only
  be treated as crude estimates.  This should be considered a work in
  progress.

- The GitHub profiling rule still builds the standard configuration,
  evem though it is unused.

- Additional tools are required to push the timings to some database,
  either a local sqlite3 or an external one.
@marshallward
Copy link
Collaborator Author

Although this work does not yet resemble what we would like to see, I think it's is developed enough to at least be included in the repository.

Future considerations include the following:

  • Better tuning to report a "slow" or "fast" timing.
  • Better stability on GitHub Actions: (though this is probably not an option)
  • Integration with a database for automatic logging.

@codecov
Copy link

codecov bot commented Aug 18, 2021

Codecov Report

Merging #1472 (e85f291) into dev/gfdl (e0e70e3) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##           dev/gfdl    #1472   +/-   ##
=========================================
  Coverage     29.13%   29.13%           
=========================================
  Files           235      235           
  Lines         71061    71061           
=========================================
  Hits          20707    20707           
  Misses        50354    50354           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0e70e3...e85f291. Read the comment docs.

Copy link
Collaborator

@Hallberg-NOAA Hallberg-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks reasonable enough, and it has passed the TC testing and the pipeline testing at https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/pipelines/13469.

@Hallberg-NOAA Hallberg-NOAA merged commit 79fcdfb into mom-ocean:dev/gfdl Aug 27, 2021
@marshallward marshallward deleted the make_prof_merge branch October 20, 2021 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants