Create 'coupler.res' log file in write grid comp. Explicitly specify chunk sizes in restart files

[DusanJovic-NOAA]

PR Author Checklist:

• I have linked PR's from all sub-components involved in section below.

• I am confirming reviews are completed in ALL sub-component PR's.

• I have run the full RT suite on either Hera/Cheyenne AND have attached the log to this PR below this line:

  • LOG: RegressionTests_hera.log

• I have added the list of all failed regression tests to "Anticipated changes" section.

• I have filled out all sections of the template.

Description

When the restart files are written by the write grid component, the log file (coupler.res) must also be written by write grid comp to ensure that it is written after all other restart files have already be created.

The chunk sizes are now explicitly specified in restart file when quilting_restart is used, and are equal to the Nx x Ny in horizontal and 1 in all other dimensions.

Linked Issues and Pull Requests

Associated UFSWM Issue to close

Closes #2020

Subcomponent Pull Requests

NOAA-EMC/fv3atm/pull/726

Blocking Dependencies

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Anticipated Changes

Input data

• No changes are expected to input data.
• Changes are expected to input data:
  • New input data.
  • Updated input data.

Regression Tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

Tests effected by changes in this PR:

Libraries

• Not Needed
• Needed
  • Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
  • Add issue link from JCSDA/spack-stack following this item

Code Managers Log

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.
  • N/A

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[jkbk2004]

@DusanJovic-NOAA rrfs team requested to commit this pr first for operational application. Can you sync up the branch?

[epic-cicd-jenkins]

Jenkins-ci ORTs passed

[BrianCurtis-NOAA]

Acorn is sitting on it's last item and it's queued still, it's been there for at least a few hours. There are no current fail-tests in the working directory. I plan on killing the last task on Acorn so we don't need to wait. The queue on Acorn is being used by the SA's, so I don't think this long wait is about the resources needed for this test being too high.

[zach1221]

Acorn is sitting on it's last item and it's queued still, it's been there for at least a few hours. There are no current fail-tests in the working directory. I plan on killing the last task on Acorn so we don't need to wait. The queue on Acorn is being used by the SA's, so I don't think this long wait is about the resources needed for this test being too high.

@BrianCurtis-NOAA ok I think we're good to go then.

[zach1221]

@DusanJovic-NOAA fv3atm pr is merged. Please revert change in .gitmodules and update the submodule hash.
NOAA-EMC/fv3atm@a82381c

[BrianCurtis-NOAA]

Did you find any consistency in the run taking longer than 30 or a history of higher run times?

[jkbk2004]

Did you find any consistency in the run taking longer than 30 or a history of higher run times?

Sorry for confusing commit remarks.. Actually the case is cpld_debug_pdlib_p8 on jet. The baseline was originated from #1967. Somehow the test log wasn't reported in the report for a few PRs. But 1882 sec is not bad on jet in a sense jet is small machine. On Orion and hercules, the case is taking about 1500 sec. I am trying to confirm and recover missing baseline case on jet At least, I was catching up the case on jet from last PR and this PR. So, we can let this pr move on to be merged. I will update in tomorrow's tag up.

[BrianCurtis-NOAA]

@jkbk2004 Im confused. We have changes in this PR for cpld_debug_p8 but you mention cpld_debug_pdlib_p8 and missing baselines? Do we need the cpld_debug_p8 changes?

[jkbk2004]

@BrianCurtis-NOAA Oh! my mistake. it's cpld_debug_pdlib_p8 issue as the jet log show at the end. I am fixing now.

[jkbk2004]

All set now.

[DeniseWorthen]

@jkbk2004 The pdlib tests were restricted to only hera, orion, wcoss2 and cheyenne, but they got turned on everywhere in #1967. So the last two weeks worth of pdlib tests should show up in the jet logs but the test does not show up in the log for that PR's commit. Why is the test not running on jet?

You can see here that the test (25) is not reported:

ufs-weather-model/tests/logs/RegressionTests_jet.log

Lines 1568 to 1578 in f431fbe

	Comparing 20210323.060000.out_grd.ww3 .........OK
	0: The total amount of wall time = 1516.470165
	0: The maximum resident set size (KB) = 1637252
	Test 024 cpld_mpi_pdlib_p8_intel PASS
	baseline dir = /mnt/lfs4/HFIP/hfv3gfs/role.epic/RT/NEMSfv3gfs/develop-20231117/control_flake_intel
	working dir = /lfs4/HFIP/h-nems/Zachary.Shrader/RT_RUNDIRS/Zachary.Shrader/FV3_RT/rt_242977/control_flake_intel
	Checking test 026 control_flake_intel results ....

[jkbk2004]

As I said, somehow the case was missing for the commits on Nov 22/27/29 and Dec but reported on Dec 4 and this pr. I am recovering now. cases are running and shows around 1880 sec. Anyway, no need to hold this pr. We can merge this pr. I will report the result of recovering missing jet cases in previous baseline dates tomorrow.

[jkbk2004]

cpld_debug_pdlib_p8 is all recovered on jet for baselines: develop-20231117 and develop-20231122. timing is around 1800 ~ 1900 sec. So bumping up wlclk makes a sense. @zach1221 it's ok to proceed to merge this pr.

[zach1221]

cpld_debug_pdlib_p8 is all recovered on jet for baselines: develop-20231117 and develop-20231122. timing is around 1800 ~ 1900 sec. So bumping up wlclk makes a sense. @zach1221 it's ok to proceed to merge this pr.

Ok, I'll need one more approval. Waiting on that.

[BrianCurtis-NOAA]

Please wait until after the CM meeting to merge.