[develop] Add restart capability to forecast task

[chan-hoo]

DESCRIPTION OF CHANGES:

• Add a flag DO_FCST_RESTART turning on/off the restart option of the forecast task.
• Add a python script update_restart_input_nml_file.py replacing the six parameters related to the restart option in the FV3 input.nml file.
• Add a parameter fhrot to model_configure.
• Update the configuration files model_configure and nems.configure with the latest format.
• This capability is the NCO's requirement.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• In the following example, the restart run starts from fhr=003.
• The sample run uses 'FV3_GFS_v15_thompson_mynn_lam3km' to match with the regression test of the ufs weather model.

1. Build the app without -a argument:

./devbuild.sh -p=hera

2. Run the following configuration file:

metadata:
  description: This script is to test the restart capability.
user:
  RUN_ENVIR: community
  MACHINE: HERA
  ACCOUNT: naqfc
platform:
  BUILD_MOD_FN: build_hera_intel
workflow:
  USE_CRON_TO_RELAUNCH: true
  CRON_RELAUNCH_INTVL_MNTS: 3
  EXPT_SUBDIR: test_restart_lam3km
  CCPP_PHYS_SUITE: FV3_GFS_v15_thompson_mynn_lam3km
  PREDEF_GRID_NAME: RRFS_CONUS_3km
  DATE_FIRST_CYCL: '2022070100'
  DATE_LAST_CYCL: '2022070100'
  FCST_LEN_HRS: 6
  PREEXISTING_DIR_METHOD: rename
  VERBOSE: false
  DEBUG: false
  COMPILER: intel
task_get_extrn_ics:
  EXTRN_MDL_NAME_ICS: FV3GFS
  FV3GFS_FILE_FMT_ICS: netcdf
task_get_extrn_lbcs:
  EXTRN_MDL_NAME_LBCS: FV3GFS
  FV3GFS_FILE_FMT_LBCS: netcdf
  LBC_SPEC_INTVL_HRS: 3
task_run_fcst:
  RESTART_INTERVAL: 3
  DO_FCST_RESTART: true

3. Run the rocotorewind for the forecast task:

rocotorewind -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -c 202207010000 -t run_fcst_mem000

4. Move the result files to another directory:

cd 2022070100
mkdir RES_ORG
mv dynf* RES_ORG
mv phyf* RES_ORG

5. Re-run the forecast task:

./launch_FV3LAM_wflow.sh

6. Check that phyf004-006.nc and dynf004-006.nc files are produced again.

• hera.intel
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

ISSUE:

Fixes issue mentioned in #681

DEPENDENCIES:

PR #663

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

CONTRIBUTORS:

RatkoVasic-NOAA

[chan-hoo]

@JeffBeck-NOAA, it works well !!! Thank you for your suggestion and I am sorry again for my late understanding! I am running a couple of WE2E tests now.

[JeffBeck-NOAA]

@JeffBeck-NOAA, it works well !!! Thank you for your suggestion and I am sorry again for my late understanding! I am running a couple of WE2E tests now.

@chan-hoo, excellent! Glad to hear things are running well and no problem!

[chan-hoo]

@MichaelLueken, the latest change passed three WE2E tests for FV3_GFS_v16, FV3_GFS_v17_p8, and FV3_GFS_v15p2 (w/ GSMGFS). Could you check the results of the Jenkins test?

[MichaelLueken]

@chan-hoo Thank you for working with @JeffBeck-NOAA with respect to nstf_name and NSST spin-up issues! I have resubmitted the Jenkins tests for this PR. There are still issues with the Jenkins tests following Friday's migration to a new site, so I will let you know if there are any new failures.

[MichaelLueken]

@chan-hoo The Jenkins tests have passed. I'm in the middle of manually rerunning the tests on Jet now. I'll update the PR once the Jet tests are complete.

[MichaelLueken]

@chan-hoo The manual WE2E fundamental tests on Jet have successfully passed.

I have also spoken with @christinaholtNOAA. We are hoping to get her PR in tomorrow or early on Friday. If there are issues with the Jenkins testing or manual tests on Jet, then I will be able to move forward with these changes. Thank you!

[chan-hoo]

@MichaelLueken, can you merge this PR first? I'd like to create a production branch for AQM from this PR without @christinaholtNOAA's PR.

[MichaelLueken]

@chan-hoo Unfortunately, this PR will only be able to be merged before @christinaholtNOAA's PR if there are still issues with her PR. If you don't want to include @christinaholtNOAA's PR in the production branch for aqmv7, then I would suggest creating the production/aqmv7 branch from this branch, or waiting and see if there are still issues with the testing following her latest updates, at which time I can move forward with this work.

[chan-hoo]

@MichaelLueken, no problem. Will do.

[chan-hoo]

@MichaelLueken, you can merge this PR although I found some bugs on WCOSS2 and AQM configuration caused by PR #676 (Issues #709 and #710). I hope @christinaholtNOAA will resolve them soon. This PR itself doesn't show any issues on Hera.

[MichaelLueken]

@chan-hoo From @christinaholtNOAA's discussion in issue #709, it sounds like she has a fix for several issues. I will hold off on merging this work until both of you have decided the best path to get the updates in (either through this PR or in a separate PR). Having said that, I have rerun the Jenkins tests after you updated your branch to the latest develop and all tests successfully passed. So, these changes are ready to go either now, or after merging Christina's changes in.

[chan-hoo]

@MichaelLueken, Can you merge this PR? There are so many issues to be resolved for AQM now, so I think it would be better to open a separate PR for them. I'll have to review and merge the PR #708 for the AQM implementation first.