Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dtc/hwrf-physics: HWRF RRTMG (based on #30) #36

Merged
merged 12 commits into from
Apr 9, 2020

Conversation

climbfuji
Copy link
Collaborator

@climbfuji climbfuji commented Apr 8, 2020

This PR is based on #30 by @mzhangw, but for branch dtc/hwrf-physics instead of dtc/develop. It also contains several bug fixes and updates to the original PR.

The following commits were cherry-picked from #30 (all from @mzhangw):
5ab446f
1cbc515
c6f51f5
2a43ab1
73f5dcb

The additional commits apply changes necessary for switching the target branch (e.g., add the new regression tests to the correct rt.sh configuration file rt.conf). In addition, several files that were added as part of the original PR #30 for testing purposes only are removed.

Associated PRs:
NCAR/ccpp-physics#430
NCAR/fv3atm#38
#36

For regression testing information, see below (in progress).

@climbfuji
Copy link
Collaborator Author

Attached is a screenshot of the 2m temperature differences after three hours of integration using the tests fv3_ccpp_regional_c768_FA_update_moist_repro (left) and fv3_ccpp_regional_c768_FA_HWRF_RRTMG_repro (right). The middle panel shows the difference left minus right. The plots were made using ncview fv3_history2d.nc.

This is by no means a full evaluation of the newly added code, just a poor man's attempt to show that the implementation of HWRF RRTMG produces reasonably looking results.

Screen Shot 2020-04-08 at 6 34 59 AM

@climbfuji
Copy link
Collaborator Author

Regression testing on hera.intel against the original/official EMC baseline located in
/scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20200317/INTEL: all tests pass, except the newly added tests for HWRF physics. For each of these, the model run completes and the failure is caused by not being able to find a baseline.

rt_full_against_existing_baseline.log

bash-4.2$ grep -le PASS log_hera.intel/rt_*
log_hera.intel/rt_001_fv3_control.log
log_hera.intel/rt_002_fv3_decomp.log
log_hera.intel/rt_003_fv3_2threads.log
log_hera.intel/rt_004_fv3_restart.log
log_hera.intel/rt_005_fv3_read_inc.log
log_hera.intel/rt_006_fv3_gfdlmp.log
log_hera.intel/rt_007_fv3_gfdlmprad_gwd.log
log_hera.intel/rt_008_fv3_gfdlmprad_noahmp.log
log_hera.intel/rt_009_fv3_wsm6.log
log_hera.intel/rt_010_fv3_wrtGauss_netcdf_esmf.log
log_hera.intel/rt_011_fv3_wrtGauss_netcdf.log
log_hera.intel/rt_012_fv3_wrtGauss_nemsio.log
log_hera.intel/rt_013_fv3_wrtGauss_nemsio_c192.log
log_hera.intel/rt_014_fv3_stochy.log
log_hera.intel/rt_015_fv3_iau.log
log_hera.intel/rt_016_fv3_csawmgshoc.log
log_hera.intel/rt_017_fv3_csawmg.log
log_hera.intel/rt_018_fv3_rasmgshoc.log
log_hera.intel/rt_019_fv3_csawmg3shoc127.log
log_hera.intel/rt_020_fv3_satmedmf.log
log_hera.intel/rt_021_fv3_lheatstrg.log
log_hera.intel/rt_022_fv3_gfdlmprad.log
log_hera.intel/rt_023_fv3_gfdlmprad_atmwav.log
log_hera.intel/rt_024_fv3_wrtGauss_nemsio_c768.log
log_hera.intel/rt_025_fv3_appbuild.log
log_hera.intel/rt_026_fv3_control_32bit.log
log_hera.intel/rt_027_fv3_gfdlmprad_32bit_post.log
log_hera.intel/rt_028_fv3_stretched.log
log_hera.intel/rt_029_fv3_stretched_nest.log
log_hera.intel/rt_030_fv3_stretched_nest_quilt.log
log_hera.intel/rt_031_fv3_regional_control.log
log_hera.intel/rt_032_fv3_regional_restart.log
log_hera.intel/rt_033_fv3_regional_quilt.log
log_hera.intel/rt_034_fv3_regional_c768.log
log_hera.intel/rt_035_fv3_control_debug.log
log_hera.intel/rt_036_fv3_stretched_nest_debug.log
log_hera.intel/rt_037_fv3_control_repro.log
log_hera.intel/rt_038_fv3_decomp_repro.log
log_hera.intel/rt_039_fv3_2threads_repro.log
log_hera.intel/rt_040_fv3_restart_repro.log
log_hera.intel/rt_041_fv3_read_inc_repro.log
log_hera.intel/rt_042_fv3_wrtGauss_netcdf_esmf_repro.log
log_hera.intel/rt_043_fv3_wrtGauss_netcdf_repro.log
log_hera.intel/rt_044_fv3_wrtGauss_nemsio_repro.log
log_hera.intel/rt_045_fv3_wrtGauss_nemsio_c192_repro.log
log_hera.intel/rt_046_fv3_stochy_repro.log
log_hera.intel/rt_047_fv3_iau_repro.log
log_hera.intel/rt_048_fv3_gfdlmp_repro.log
log_hera.intel/rt_049_fv3_gfdlmprad_gwd_repro.log
log_hera.intel/rt_050_fv3_gfdlmprad_noahmp_repro.log
log_hera.intel/rt_051_fv3_csawmgshoc_repro.log
log_hera.intel/rt_052_fv3_csawmg3shoc127_repro.log
log_hera.intel/rt_053_fv3_csawmg_repro.log
log_hera.intel/rt_054_fv3_satmedmf_repro.log
log_hera.intel/rt_055_fv3_gfdlmprad_repro.log
log_hera.intel/rt_056_fv3_gfdlmprad_atmwav_repro.log
log_hera.intel/rt_057_fv3_wrtGauss_nemsio_c768_repro.log
log_hera.intel/rt_058_fv3_appbuild_repro.log
log_hera.intel/rt_059_fv3_control_32bit_repro.log
log_hera.intel/rt_060_fv3_gfdlmprad_32bit_post_repro.log
log_hera.intel/rt_061_fv3_stretched_repro.log
log_hera.intel/rt_062_fv3_stretched_nest_repro.log
log_hera.intel/rt_063_fv3_regional_control_repro.log
log_hera.intel/rt_064_fv3_regional_restart_repro.log
log_hera.intel/rt_065_fv3_regional_quilt_repro.log
log_hera.intel/rt_066_fv3_regional_c768_repro.log
log_hera.intel/rt_067_fv3_gfdlmp_32bit_repro.log
log_hera.intel/rt_068_fv3_cpt_repro.log
log_hera.intel/rt_069_fv3_ccpp_control_repro.log
log_hera.intel/rt_070_fv3_ccpp_decomp_repro.log
log_hera.intel/rt_071_fv3_ccpp_2threads_repro.log
log_hera.intel/rt_072_fv3_ccpp_restart_repro.log
log_hera.intel/rt_073_fv3_ccpp_read_inc_repro.log
log_hera.intel/rt_074_fv3_ccpp_wrtGauss_netcdf_esmf_repro.log
log_hera.intel/rt_075_fv3_ccpp_wrtGauss_netcdf_repro.log
log_hera.intel/rt_076_fv3_ccpp_wrtGauss_nemsio_repro.log
log_hera.intel/rt_077_fv3_ccpp_wrtGauss_nemsio_c192_repro.log
log_hera.intel/rt_078_fv3_ccpp_stochy_repro.log
log_hera.intel/rt_079_fv3_ccpp_iau_repro.log
log_hera.intel/rt_080_fv3_ccpp_gfdlmprad_repro.log
log_hera.intel/rt_081_fv3_ccpp_gfdlmprad_atmwav_repro.log
log_hera.intel/rt_082_fv3_ccpp_wrtGauss_nemsio_c768_repro.log
log_hera.intel/rt_083_fv3_ccpp_appbuild_repro.log
log_hera.intel/rt_084_fv3_ccpp_control_32bit_repro.log
log_hera.intel/rt_085_fv3_ccpp_stretched_repro.log
log_hera.intel/rt_086_fv3_ccpp_stretched_nest_repro.log
log_hera.intel/rt_087_fv3_ccpp_regional_control_repro.log
log_hera.intel/rt_088_fv3_ccpp_regional_restart_repro.log
log_hera.intel/rt_089_fv3_ccpp_regional_quilt_repro.log
log_hera.intel/rt_090_fv3_ccpp_regional_c768_repro.log
log_hera.intel/rt_091_fv3_ccpp_gfdlmp_repro.log
log_hera.intel/rt_092_fv3_ccpp_gfdlmprad_gwd_repro.log
log_hera.intel/rt_093_fv3_ccpp_gfdlmprad_noahmp_repro.log
log_hera.intel/rt_094_fv3_ccpp_csawmg_repro.log
log_hera.intel/rt_095_fv3_ccpp_satmedmf_repro.log
log_hera.intel/rt_096_fv3_ccpp_gfdlmp_32bit_repro.log
log_hera.intel/rt_097_fv3_ccpp_gfdlmprad_32bit_post_repro.log
log_hera.intel/rt_098_fv3_ccpp_cpt_repro.log
log_hera.intel/rt_099_fv3_ccpp_gsd_repro.log
log_hera.intel/rt_103_fv3_ccpp_control_prod.log
log_hera.intel/rt_104_fv3_ccpp_decomp_prod.log
log_hera.intel/rt_105_fv3_ccpp_2threads_prod.log
log_hera.intel/rt_106_fv3_ccpp_restart_prod.log
log_hera.intel/rt_107_fv3_ccpp_read_inc_prod.log
log_hera.intel/rt_108_fv3_ccpp_wrtGauss_netcdf_esmf_prod.log
log_hera.intel/rt_109_fv3_ccpp_wrtGauss_netcdf_prod.log
log_hera.intel/rt_110_fv3_ccpp_wrtGauss_nemsio_prod.log
log_hera.intel/rt_111_fv3_ccpp_wrtGauss_nemsio_c192_prod.log
log_hera.intel/rt_112_fv3_ccpp_stochy_prod.log
log_hera.intel/rt_113_fv3_ccpp_iau_prod.log
log_hera.intel/rt_114_fv3_ccpp_gfdlmprad_prod.log
log_hera.intel/rt_115_fv3_ccpp_gfdlmprad_atmwav_prod.log
log_hera.intel/rt_116_fv3_ccpp_wrtGauss_nemsio_c768_prod.log
log_hera.intel/rt_117_fv3_ccpp_appbuild_prod.log
log_hera.intel/rt_118_fv3_ccpp_control_32bit_prod.log
log_hera.intel/rt_119_fv3_ccpp_stretched_prod.log
log_hera.intel/rt_120_fv3_ccpp_stretched_nest_prod.log
log_hera.intel/rt_121_fv3_ccpp_regional_control_prod.log
log_hera.intel/rt_122_fv3_ccpp_regional_restart_prod.log
log_hera.intel/rt_123_fv3_ccpp_regional_quilt_prod.log
log_hera.intel/rt_124_fv3_ccpp_regional_c768_prod.log
log_hera.intel/rt_125_fv3_ccpp_control_debug_prod.log
log_hera.intel/rt_126_fv3_ccpp_stretched_nest_debug_prod.log
log_hera.intel/rt_127_fv3_ccpp_gfdlmp_prod.log
log_hera.intel/rt_128_fv3_ccpp_gfdlmprad_gwd_prod.log
log_hera.intel/rt_129_fv3_ccpp_gfdlmprad_noahmp_prod.log
log_hera.intel/rt_130_fv3_ccpp_csawmg_prod.log
log_hera.intel/rt_131_fv3_ccpp_satmedmf_prod.log
log_hera.intel/rt_132_fv3_ccpp_gfdlmp_32bit_prod.log
log_hera.intel/rt_133_fv3_ccpp_gfdlmprad_32bit_post_prod.log
log_hera.intel/rt_134_fv3_ccpp_cpt_prod.log
log_hera.intel/rt_135_fv3_ccpp_gsd_prod.log
log_hera.intel/rt_136_fv3_ccpp_thompson_prod.log
log_hera.intel/rt_137_fv3_ccpp_thompson_no_aero_prod.log
log_hera.intel/rt_141_fv3_ccpp_gfs_v15p2_prod.log
log_hera.intel/rt_142_fv3_ccpp_gfs_v16beta_prod.log
log_hera.intel/rt_143_fv3_ccpp_gfs_v15p2_debug_prod.log
log_hera.intel/rt_144_fv3_ccpp_gfs_v16beta_debug_prod.log
log_hera.intel/rt_145_fv3_ccpp_gsd_debug_prod.log
log_hera.intel/rt_146_fv3_ccpp_thompson_debug_prod.log
log_hera.intel/rt_147_fv3_ccpp_thompson_no_aero_debug_prod.log
log_hera.intel/rt_148_fv3_ccpp_gfs_v15p2_repro.log
log_hera.intel/rt_149_fv3_ccpp_gfs_v16beta_repro.log
log_hera.intel/rt_150_fv3_gfs_v15p2_repro.log
log_hera.intel/rt_151_fv3_gfs_v16beta_repro.log

bash-4.2$ grep -le FAIL log_hera.intel/rt_*
log_hera.intel/rt_100_fv3_ccpp_regional_c768_FA_update_moist_repro.log
log_hera.intel/rt_101_fv3_ccpp_regional_c768_HWRF_PBL_repro.log
log_hera.intel/rt_102_fv3_ccpp_regional_c768_FA_HWRF_RRTMG_repro.log
log_hera.intel/rt_138_fv3_ccpp_regional_c768_FA_update_moist_prod.log
log_hera.intel/rt_139_fv3_ccpp_regional_c768_HWRF_PBL_prod.log
log_hera.intel/rt_140_fv3_ccpp_regional_c768_FA_HWRF_RRTMG_prod.log

bash-4.2$ for file in `grep -le FAIL log_hera.intel/rt_*`; do echo $file; cat $file; done
log_hera.intel/rt_100_fv3_ccpp_regional_c768_FA_update_moist_repro.log

baseline dir = /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20200317/INTEL/fv3_regional_c768_FA_update_moist_repro
working dir  = /scratch1/NCEPDEV/stmp2/Dom.Heinzeller/FV3_RT/rt_45718/fv3_ccpp_regional_c768_FA_update_moist_repro
Checking test 100 fv3_ccpp_regional_c768_FA_update_moist results ....
 Comparing atmos_4xdaily.nc ............MISSING baseline
 Comparing fv3_history2d.nc ............MISSING baseline
 Comparing fv3_history.nc ............MISSING baseline
Test 100 fv3_ccpp_regional_c768_FA_update_moist FAIL

log_hera.intel/rt_101_fv3_ccpp_regional_c768_HWRF_PBL_repro.log

baseline dir = /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20200317/INTEL/fv3_regional_c768_HWRF_PBL_repro
working dir  = /scratch1/NCEPDEV/stmp2/Dom.Heinzeller/FV3_RT/rt_45718/fv3_ccpp_regional_c768_HWRF_PBL_repro
Checking test 101 fv3_ccpp_regional_c768_HWRF_PBL results ....
 Comparing atmos_4xdaily.nc ............MISSING baseline
 Comparing fv3_history2d.nc ............MISSING baseline
 Comparing fv3_history.nc ............MISSING baseline
Test 101 fv3_ccpp_regional_c768_HWRF_PBL FAIL

log_hera.intel/rt_102_fv3_ccpp_regional_c768_FA_HWRF_RRTMG_repro.log

baseline dir = /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20200317/INTEL/fv3_regional_c768_FA_HWRF_RRTMG_repro
working dir  = /scratch1/NCEPDEV/stmp2/Dom.Heinzeller/FV3_RT/rt_45718/fv3_ccpp_regional_c768_FA_HWRF_RRTMG_repro
Checking test 102 fv3_ccpp_regional_c768_FA_HWRF_RRTMG results ....
 Comparing atmos_4xdaily.nc ............MISSING baseline
 Comparing fv3_history2d.nc ............MISSING baseline
 Comparing fv3_history.nc ............MISSING baseline
Test 102 fv3_ccpp_regional_c768_FA_HWRF_RRTMG FAIL

log_hera.intel/rt_138_fv3_ccpp_regional_c768_FA_update_moist_prod.log

baseline dir = /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20200317/INTEL/fv3_regional_c768_FA_update_moist_ccpp
working dir  = /scratch1/NCEPDEV/stmp2/Dom.Heinzeller/FV3_RT/rt_45718/fv3_ccpp_regional_c768_FA_update_moist_prod
Checking test 138 fv3_ccpp_regional_c768_FA_update_moist results ....
 Comparing atmos_4xdaily.nc ............MISSING baseline
 Comparing fv3_history2d.nc ............MISSING baseline
 Comparing fv3_history.nc ............MISSING baseline
Test 138 fv3_ccpp_regional_c768_FA_update_moist FAIL

log_hera.intel/rt_139_fv3_ccpp_regional_c768_HWRF_PBL_prod.log

baseline dir = /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20200317/INTEL/fv3_regional_c768_HWRF_PBL_ccpp
working dir  = /scratch1/NCEPDEV/stmp2/Dom.Heinzeller/FV3_RT/rt_45718/fv3_ccpp_regional_c768_HWRF_PBL_prod
Checking test 139 fv3_ccpp_regional_c768_HWRF_PBL results ....
 Comparing atmos_4xdaily.nc ............MISSING baseline
 Comparing fv3_history2d.nc ............MISSING baseline
 Comparing fv3_history.nc ............MISSING baseline
Test 139 fv3_ccpp_regional_c768_HWRF_PBL FAIL

log_hera.intel/rt_140_fv3_ccpp_regional_c768_FA_HWRF_RRTMG_prod.log

baseline dir = /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20200317/INTEL/fv3_regional_c768_FA_HWRF_RRTMG_ccpp
working dir  = /scratch1/NCEPDEV/stmp2/Dom.Heinzeller/FV3_RT/rt_45718/fv3_ccpp_regional_c768_FA_HWRF_RRTMG_prod
Checking test 140 fv3_ccpp_regional_c768_FA_HWRF_RRTMG results ....
 Comparing atmos_4xdaily.nc ............MISSING baseline
 Comparing fv3_history2d.nc ............MISSING baseline
 Comparing fv3_history.nc ............MISSING baseline
Test 140 fv3_ccpp_regional_c768_FA_HWRF_RRTMG FAIL

@climbfuji
Copy link
Collaborator Author

climbfuji commented Apr 8, 2020

Description of final regression testing

Based on the results above (#36 (comment)), a new baseline is generated and the location of the regression test baseline (RTPWD in rt.sh) is updated to /scratch1/BMC/gmtb/ufs-weather-model/RT/NEMSfv3gfs/hwrf-physics-20200317/INTEL on hera.

The contents of /scratch1/BMC/gmtb/ufs-weather-model/RT/NEMSfv3gfs/FV3_input_data_temporary_hafs is copied to a new subdirectory FV3_input_data_HWRF in the regression test baseline, and the run scripts for the HWRF tests are updated.


Individual steps

Creating new HWRF physics baseline in /scratch1/NCEPDEV/stmp4/Dom.Heinzeller/FV3_RT/REGRESSION_TEST_INTEL using ./rt.sh -f -c -e 2>&1 | tee rt_full_create.log: completed

rt_full_create.log

Syncing /scratch1/NCEPDEV/stmp4/Dom.Heinzeller/FV3_RT/REGRESSION_TEST_INTEL to /scratch1/BMC/gmtb/ufs-weather-model/RT/NEMSfv3gfs/hwrf-physics-20200317/INTEL: completed

Adding a subdirectory FV3_input_data_HWRF with data from /scratch1/BMC/gmtb/ufs-weather-model/RT/NEMSfv3gfs/FV3_input_data_temporary_hafs and updating the HWRF regression test run scripts: completed

Verifying against new HWRF physics baseline using ./rt.sh -f -e 2>&1 | tee rt_full_create.log after copying the newly generated baseline to the correct location listed above: completed, all tests pass.

rt_full_verify.log

…est namelist templates and run scripts, use own HWRF regression test baseline
@climbfuji
Copy link
Collaborator Author

All completed, PR is ready to merge!

Copy link
Collaborator

@ligiabernardet ligiabernardet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend we get this PR approved and onto dtc/hwrf-physics so work can proceed with the work. After that we can ask Mike Iacono to take a look.

@climbfuji
Copy link
Collaborator Author

I recommend we get this PR approved and onto dtc/hwrf-physics so work can proceed with the work. After that we can ask Mike Iacono to take a look.

Ok, thank you. I will wait for approval from one of the CODEOWNERs to start the merge process. Once this is in, I will look at the Noah LSM PRs from Grant.

@@ -129,6 +129,9 @@
pre_rad = .false.
ncld = 5
imp_physics = 15
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been doing the same thing as @mzhangw, editing this namelist for my HWRF physics schemes. Should we consider changing the name since it will be valid for many (all?) HWRF schemes (and not just the FA scheme)?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can consolidate all HWRF namelist options once all schemes are ready.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@grantfirl if you like, please do that as part of the Noah LSM PR (when you pull in the updates from these PRs before the final testing). Thanks!

Copy link
Collaborator

@grantfirl grantfirl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Approved.

@climbfuji climbfuji merged commit 61b9443 into NCAR:dtc/hwrf-physics Apr 9, 2020
@climbfuji climbfuji mentioned this pull request Apr 9, 2020
panll pushed a commit to panll/ufs-weather-model that referenced this pull request Apr 21, 2020
* Updated fv3atm (see fv3atm PR NCAR#48 NOAA-EMC/fv3atm#48)
* Updated WW3
SamuelTrahanNOAA pushed a commit to SamuelTrahanNOAA/ufs-weather-model that referenced this pull request Sep 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants