indicates numerical instability

[zhanglikate]

Description

When @GonzaloFerrada-NOAA git clone the recent G-W and tried a testing run. We (@ShanSunNOAA ) found that the C384 run crash with the following numerical instability issue , the failed run with APP=ATM and DELTIM=300 . If we change the DELTIM=240, it works well. The ICs file is from /scratch1/NCEPDEV/global/glopara/data/ICSDIR/prototype_ICs/GEFS-NoahMP-aerosols-p8c_refactored/2016070100

We did not see the issue in the previous 2024 April version of commit f234a3e (HEAD)

0: aft fcst run output time= 300 FBcount= 11 na= 1
720:
720: FATAL from PE 360: NaN in input field of mpp_reproducing_sum(_2d), this indicates numerical instability
720:
720: Image PC Routine Line Source
720: ufs_model.x 0000000006461957 Unknown Unknown Unknown
720: ufs_model.x 00000000056A4C99 mpp_mod_mp_mpp_er 72 mpp_util_mpi.inc
720: ufs_model.x 000000000596CE72 mpp_efp_mod_mp_mp 195 mpp_efp.F90
720: ufs_model.x 00000000058B3741 mpp_domains_mod_m 143 mpp_global_sum.fh
720: ufs_model.x 0000000002EE4F70 fv_grid_utils_mod 3077 fv_grid_utils.F90
720: ufs_model.x 0000000002F1E4A9 fv_mapz_mod_mp_la 794 fv_mapz.F90
720: libiomp5.so 0000146F28604BB3 __kmp_invoke_micr Unknown Unknown
720: libiomp5.so 0000146F28580903 Unknown Unknown Unknown
720: libiomp5.so 0000146F285823DE __kmp_fork_call Unknown Unknown
720: libiomp5.so 0000146F28542CB5 _kmpc_fork_call Unknown Unknown
720: ufs_model.x 0000000002F1998C fv_mapz_mod_mp_la 683 fv_mapz.F90
720: ufs_model.x 0000000002EC0A90 fv_dynamics_mod_m 771 fv_dynamics.F90
720: ufs_model.x 0000000002E10D6C atmosphere_mod_mp 683 atmosphere.F90
720: ufs_model.x 0000000002CFA818 atmos_model_mod_m 880 atmos_model.F90
720: ufs_model.x 0000000002B9F4C7 module_fcst_grid 1330 module_fcst_grid_comp.F90
720: ufs_model.x 0000000000A9D5D4 Unknown Unknown Unknown
720: ufs_model.x 0000000000AA117F Unknown Unknown Unknown
720: ufs_model.x 0000000000948B7A Unknown Unknown Unknown
720: ufs_model.x 0000000001219E3F Unknown Unknown Unknown
720: ufs_model.x 0000000000A9EA1A Unknown Unknown Unknown
720: ufs_model.x 000000000096BCE0 Unknown Unknown Unknown
720: ufs_model.x 0000000000CA02E1 Unknown Unknown Unknown
720: ufs_model.x 0000000002B8D612 fv3atm_cap_mod_mp 1077 fv3_cap.F90
720: ufs_model.x 0000000002B8D06B fv3atm_cap_mod_mp 1026 fv3_cap.F90
720: ufs_model.x 00000000006A69E8 Unknown Unknown Unknown
720: ufs_model.x 00000000006A694A Unknown Unknown Unknown

To Reproduce:

version
Global workflow: commit 1b18f2f97eec24a8b3f817b7ef1b4caa2be24937 (HEAD -> develop, origin/develop, origin/HEAD)

ufs-weather-model: commit c127601 (HEAD)

Additional context

My GW directory:
/scratch2/BMC/gsd-fv3-dev/Gonzalo.Ferrada/work/global-workflow

abd expdir:
/scratch2/BMC/gsd-fv3-dev/Gonzalo.Ferrada/work/expdir/test3

Output

The log of the failed run with APP=ATM and DELTIM=300 is

/scratch2/BMC/gsd-fv3-dev/Gonzalo.Ferrada/work/output/test3/logs/2016070100/gfsfcst_seg0.log.0