Feature #2550 compute_diagnostics

[JohnHalleyGotway]

Expected Differences

This PR provides major upgrades to the TC-Diag tool through a large number of modified files. While this is a significant amount of work, more changes to TC-Diag are needed in the beta3 development cycle and are described in issue #2729.

Here's an overview of the changes in this PR:

• Incorporates CSU CIRA Python Diagnostics code:

  • scripts/python/tc_diag is code pulled from https://github.com/robertdemariacira/tc_diag_driver
  • scripts/python/tc_diag/compute_tc_diag.py is the Python script that MET actually calls to interface with the CIRA Python diagnostics code.
  • scripts/python/pyembed contains read_tmp_diag.py and write_tmp_diag.py scripts to facilitate running TC-Diag with MET_PYTHON_EXE pointing to a custom Python installation.
  • Changes to configure.ac, configure, and many Makefiles are caused by adding new directories.
  • Note that data/tc_data/v2023-04-07_gdland_table.dat is added to define the distance to land. I added a note to Fix the wind-based diagnostics computations in TC-Diag #2729 about how this overlaps the TC-DLand tool.

• Made a handful of changes to common library code for convenience during development.

• Changes to the TC-Diag source code in src/tools/tc_utils/tc_diag.

  • Refine how/what data is passed from Python to the C++ code, support separate sections for specifying storm diagnostics, sounding diagnostics, and custom diagnostics. Also let the Python script define units, long names, and comments to be appended to the bottom of the ASCII output.

• Do these changes introduce new tools, command line arguments, or configuration file options? [Yes]
  
  If yes, please describe:
  
  In TCDiagConfig_default:

  • Adds override_diags config option to specify which diagnostics from the nest should override the parents.
  • Remove output_prefix and replace it with output_base_format.
  • Rename nc_rng_azi_flag with nc_cyc_grid_flag.
  • By default, specify diag_script separately for the "parent" and "nest" domains passing in different yml configuration files.

• Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [Yes]
  
  If yes, please describe:
  

• Adds new TC-Diag NetCDF output file.

• Adds new TC-Diag ASCII output file.

Pull Request Testing

• Describe testing already performed for these changes:
  
  Confirmed that the existing unit test in unit_tc_diag.xml runs and produces the two new output files.
  Carefully compared the newly generated ASCII output to exist CIRA Diagnostics output to ensure consistent formatting. I do note some differences in the actual computed diagnostics values, but those diffs will be investigated and fixed through Fix the wind-based diagnostics computations in TC-Diag #2729.

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  
  First, read through Fix the wind-based diagnostics computations in TC-Diag #2729 to note what additional changes are needed.
  Review these code changes, existing documentation changes, and note that the GHA runs pass with the only difference being the creation of new TC-Diag output files.

This feature branch is compiled and available for your use on seneca in:
/d1/projects/MET/MET_pull_requests/met-12.0.0/beta2/MET-feature_2550_compute_diagnostics

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
  Updates to the TC-Diag chapter, Appendix F about Python embedding, and an unrelated change to the TC-Stat chapter. Please review these on the Files changed tab or in this PR 2728 RTD build.

Further needed updates to documentation are described in #2729.

• Do these changes include sufficient testing updates? [Yes]
  The internal/test_unit/xml/unit_tc_diag.xml file continues running the same Ian case, but the expected output files are updated to include the newly generated diag.nc and diag.txt output file.

Note that the TC-Diag unit tests should really be expanded to include an example of nesting, as mentioned in #2729.

• Will this PR result in changes to the test suite? [Yes]
  
  If yes, describe the new output and/or changes to the existing output:
  
  Creates 2 new TC-Diag output files.

• Please complete this pull request review by [Fri 11/17/23].
  

Pull Request Checklist

See the METplus Workflow for details.

• Review the source issue metadata (required labels, projects, and milestone).
• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the version that will include these changes
  Select: Coordinated METplus-X.Y Support project for bugfix releases or MET-X.Y.Z Development project for official releases
• After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[JohnHalleyGotway]

Testing notes:

1. Test whether local build and GHA build produce the same python-embedding result:

• On 11/8/23, created METbaseimage v3.1 tag from Enhance METbaseimage to install the YAML Python package METbaseimage#15 feature branch.
• GHA automatically built that tag and pushed to DockerHub.
• Reran the GHA pull request testing which failed as before.
• Building the locally on my laptop and through GHA produces the same result - as it should.

2. Test whether Python embedding still works when v3.0 is rebuilt:

• On 11/8/23, rebuilt the existing v3.0 tag on my laptop:

cd /d1/projects/METbaseimage
git checkout v3.0
docker build -t dtcenter/met-base:v3.0_local .

• The build failed because installing NumPy package version 1.26 results in an error. The fix is listing specific package version numbers, including NumPy version 1.24.2. I also had to remove the --force-reinstall option because even though we installed NumPy 1.24, XArray reinstalled using 1.26, which also errored out.

3. Build functional v3.1 image locally, manually push to DockerHub, and retest with GHA for this PR.

[jvigh]

@JohnHalleyGotway

Regarding differences of output:

Here is the screenshot of output from CIRA's TCDiag tool:

This file can be found in: seneca:/d1/projects/TCDiag/data_output/DIAGNOSTICS/TCDIAG/sample_diagnostics_data_for_IAN_for_HMON_GFS_HWRF/sal092022_avno_doper_2022092400_diag.dat

vs. MET's TCDiag tool:

1. CIRA uses 'AVNO' in the header while we have "GFSO".
2. I presume the reason that times beyond 24 h are mostly missing is just because the test data do not include the full set of GFS forecast data. Is that correct?
3. I'm seeing modest but substantial differences in the values that are present (e.g., LAND, RMW).
4. I note spacing differences between the first row and units for the sounding section.
5. The units for 850VORT and 200DVRG should be (/S).
6. The number of levels used in the sounding section differs (not sure if it's important, but it makes comparison more difficult).
7. There is no TGRD data in the custom section to compare.

[JohnHalleyGotway]

@jvigh thanks for clearly describing the differences.

1. CIRA uses 'AVNO' in the header while we have "GFSO".

I asked about this at the last project meeting and @musgrave-kate indicated we should use "GFSO".

2. I presume the reason that times beyond 24 h are mostly missing is just because the test data do not include the full set of GFS forecast data. Is that correct?

Correct, the existing unit test only uses GFS data out to 24 h. The need to enhance the unit tests is noted in issue #2729.

3. I'm seeing modest but substantial differences in the values that are present (e.g., LAND, RMW).

Agreed, the computed values do not match well enough yet. In particular, there is an issue with winds that @robertdemariacira has been tasked with investigating. These differences are noted in issue #2729.

4. I note spacing differences between the first row and units for the sounding section.

Yes, I see that. This is the difference between fixed-width FORTRAN output versus putting 1 space between columns. Let's ask @musgrave-kate if this matters, and if so, I can define a minimum width for the first column in issue #2729.

5. The units for 850VORT and 200DVRG should be (/S).

I assume that when the Python code returns all bad data it doesn't return a units string. Hopefully once the wind issues are resolved, so too will this units mismatch.

6. The number of levels used in the sounding section differs (not sure if it's important, but it makes comparison more difficult).

Yes, I configured MET with the levels @musgrave-kate indicated we should use by default. The CIRA-computed diagnostics include more levels. Let's follow up with @musgrave-kate about this.

7. There is no TGRD data in the custom section to compare.

Agreed, the CIRA Python code isn't returning it.

I'll update #2729 to further investigate and address these discrepancies.

[jvigh]

I note that the NetCDF output file is not CF-compliant. Not sure what NOAA's requirements are, but guessing this is okay. If CF compliance was desired, here's the compliance report.
IOOS Compliance Checker.pdf

The main omission is lack of standard_name attributes, which I don't think exist for these diagnostics.

Some file-level metadata might be good to include though.

Minor point: storm_id and model could be provided as global attributes of the file rather than variables. But I'm not sure there's any convention. If they are going to remain as variables, they should have long_name and/or description attributes.

[JohnHalleyGotway]

Agreed. That's why I want to review/revise the output formats before documenting them ;)

Please note any specific requests on issue #2729.

[jvigh]

Sounds good, thanks.

[jvigh]

I have left comments in the code review pane and the PR discussion. If these are all handled or identified as to-do items for later, I approve this PR. Great work!