Updates after repo merge #1
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## DESCRIPTION OF CHANGES: Part of the effort described in ufs-community#71 to update the master branch of ufs-srweather-app to a functioning state. Updates include: - Cherry-picking 5a4b90e7 from release branch in order to include the RRFS_v1alpha CCPP suite, as well as fixing some updated namelist settings in the ufs-weather-model - Fixing other namelist options that have been updated in ufs-weather-model This PR must be merged prior to ufs-community#71, then that PR updated to point to this PR's hash. ## TESTS CONDUCTED: The following end-to-end tests are now all successful on Hera: - DOT_OR_USCORE - GST_release_public_v1 - get_extrn_files_from_hpss_FV3GFS_FV3GFS - get_extrn_files_from_hpss_GSMGFS_GSMGFS - get_extrn_files_from_hpss_HRRR_RAP - get_extrn_files_from_hpss_RAP_RAP - grid_GSD_HRRR_AK_50km - grid_GSD_RAP13km - grid_RRFS_AK_13km_FV3GFS_FV3GFS - grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 - grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16beta - grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_GSD_SAR - grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta - grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 - grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16beta - grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_GSD_SAR - grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta - new_ESGgrid - new_GFDLgrid - new_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE - new_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_TRUE - regional_001 - regional_002 - suite_FV3_GFS_v15p2 - suite_FV3_GFS_v16beta - suite_FV3_GSD_SAR - suite_FV3_GSD_v0 - suite_FV3_RRFS_v1beta The following end-to-end tests were successful on Jet: - DOT_OR_USCORE - get_extrn_files_from_hpss_FV3GFS_FV3GFS - get_extrn_files_from_hpss_GSMGFS_GSMGFS - get_extrn_files_from_hpss_HRRR_RAP - get_extrn_files_from_hpss_RAP_RAP - grid_RRFS_AK_13km_FV3GFS_FV3GFS - grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 - grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16beta - grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 - grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16beta - new_ESGgrid - new_GFDLgrid - new_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE - regional_001 - regional_002 - suite_FV3_GFS_v16beta A few tests on Hera and Jet are still failing, but they are known errors and will be fixed at a later date. Cheyenne tests are failing but this is fixed by ufs-community#378 ## ISSUE (optional): In conjunction with ufs-community#378 and ufs-community#71, will solve ufs-community#57, fixing the master branch of ufs-srweather-app so everyone will be happy and nothing will be broken and we will all live happily ever after.
… crontab on Cheyenne, as well as fix workflow generation on Cheyenne (ufs-community#378) ## DESCRIPTION OF CHANGES: A fix was applied to the release branch for the Cheyenne workflow and automated testing in ufs-community#343, this PR applies the same to develop. ## TESTS CONDUCTED: Workflow built and ran on Cheyenne; the following tests were run and all passed: - grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 - grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 - grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 ## ISSUE (optional): Fully resolves ufs-community#326
…NUS domain as possible (ufs-community#360) ## DESCRIPTION OF CHANGES: ### Main changes: * Change grid parameters for the RRFS_COUNS_25km, RRFS_CONUS_13km, and RRFS_CONUS_3km grids so that they span as much of the HRRR CONUS domain as possible. Note that this is done without regard to whether NX and NY are evenly divisible by LAYOUT_X and LAYOUT_Y because these restrictions were recently removed from the forecast model. * Remove several checks on the grid parameters. These are: (1) The check on whether NX and NY are evenly divisible by LAYOUT_X and LAYOUT_Y, respectively. This restriction has been removed from the forecast model. (2) The check on BLOCKSIZE, i.e. whether the quantity (NX/LAYOUT_X)*(NY/LAYOUT_Y) is evenly divisible by BLOCKSIZE. This already depends on restriction (1) being satisfied, so it cannot be satisfied if (1) is not satisfied. (3) The check on whether NY is evenly divisible by the number of write-component tasks per write-component group. Tests were run with parameters that violated this test, and the model did not crash, so we assume this is no longer a restriction. * Modify NCL plotting scripts to make them work with the latest version of the workflow. ### Bug fixes: * Fix bug in set_predef_grid_params.sh involving an extra space after the equal sign in the lines that set GRID_GEN_METHOD. * Remove the FV3_RRFS_v1alpha suite from the list of WE2E tests because that suite is not (yet?) supported by the version of ufs-weather-model this PR is using. * Fix bug in run_experiment.sh where the variable FV3GFS_FILE_FMT_ICS should be replaced with FV3GFS_FILE_FMT_LBCS. ### Improvements: * Forecast model executable: (1) Remove section of code that copies the FV3 executable from its location in the ufs-weather-model directory to the bin directory for all other executables. This is no longer necessary since the new build system already places this executable in the bin directory. (2) Change the name of the forecast model executable to the latest one that is being used in ufs-weather-model. * Remove commented-out code that was previously used to create module files for the various workflow tasks. * Change the default physics suite to GFS_v15p2 to be consistent with the release/public-v1 branch. * Alphabetize namelist variable entries in FV3.input.yml and input.nml.FV3. * Edit comments. ## TESTS CONDUCTED: Ran all 49 WE2E tests on Hera. All except 5 tests succeeded. The results are as follows: ``` (1) DOT_OR_USCORE SUCCESS (2) GST_release_public_v1 SUCCESS (3) community_ensemble_008mems SUCCESS (4) community_ensemble_2mems SUCCESS (5) get_extrn_files_from_hpss_FV3GFS_FV3GFS SUCCESS (6) get_extrn_files_from_hpss_GSMGFS_GSMGFS SUCCESS (7) get_extrn_files_from_hpss_HRRR_RAP SUCCESS (8) get_extrn_files_from_hpss_RAP_RAP SUCCESS (9) grid_CONUS_25km_GFDLgrid_FV3GFS_FV3GFS SUCCESS (10) grid_CONUS_3km_GFDLgrid_FV3GFS_FV3GFS SUCCESS (11) grid_GSD_HRRR_AK_50km SUCCESS (12) grid_GSD_RAP13km FAILURE (13) grid_RRFS_AK_13km_FV3GFS_FV3GFS SUCCESS (14) grid_RRFS_AK_13km_RAP_RAP FAILURE (15) grid_RRFS_AK_3km_FV3GFS_FV3GFS SUCCESS (16) grid_RRFS_AK_3km_RAP_RAP FAILURE (17) grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 SUCCESS (18) grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16beta SUCCESS (19) grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_GSD_SAR SUCCESS (20) grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta SUCCESS (21) grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 SUCCESS (22) grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16beta SUCCESS (23) grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_GSD_SAR SUCCESS (24) grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta SUCCESS (25) grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 SUCCESS (26) grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16beta SUCCESS (27) grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_GSD_SAR SUCCESS (28) grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta SUCCESS (29) grid_RRFS_SUBCONUS_3km_FV3GFS_FV3GFS SUCCESS (30) grid_RRFS_SUBCONUS_3km_HRRR_RAP FAILURE (31) nco_CONUS_25km_GFDLgrid SUCCESS (32) nco_RRFS_CONUS_25km_HRRR_RAP SUCCESS (33) nco_RRFS_CONUS_3km_FV3GFS_FV3GFS SUCCESS (34) nco_RRFS_CONUS_3km_HRRR_RAP SUCCESS (35) nco_RRFS_SUBCONUS_3km_HRRR_RAP SUCCESS (36) nco_ensemble SUCCESS (37) new_ESGgrid SUCCESS (38) new_GFDLgrid SUCCESS (39) new_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE SUCCESS (40) new_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_TRUE SUCCESS (41) pregen_grid_orog_sfc_climo SUCCESS (42) regional_001 SUCCESS (43) regional_002 SUCCESS (44) suite_FV3_CPT_v0 FAILURE (45) suite_FV3_GFS_v15p2 SUCCESS (46) suite_FV3_GFS_v16beta SUCCESS (47) suite_FV3_GSD_SAR SUCCESS (48) suite_FV3_GSD_v0 SUCCESS (49) suite_FV3_RRFS_v1beta SUCCESS ``` The 5 failed tests (with information on the failures) are as follows: ``` (12) grid_GSD_RAP13km Failed in most of the run_post tasks. (14) grid_RRFS_AK_13km_RAP_RAP Failed in the run_fcst task with "AVOST IN VILKA" messages. (16) grid_RRFS_AK_3km_RAP_RAP Failed in the run_fcst task with "AVOST IN VILKA" messages. (30) grid_RRFS_SUBCONUS_3km_HRRR_RAP Failed in the run_fcst task without a clear failure message. (44) suite_FV3_CPT_v0 Failed in the run_fcst task with the following namelist read error: forrtl: severe (19): invalid reference to variable in NAMELIST input, unit -5, file Internal Formatted NML Read, line -1, position 13 ``` Except for FV3_CPT_v0, the failed tests do not use the new RRFS_CONUS grids. Thus, these failures are not related to the CONUS grid modifications. The failure of FV3_CPT_v0 is likely due to a bad namelist variable specification and thus also unrelated to the CONUS grid modifications. Note that all tests using grids for which NX and NY are not evenly divisible by LAYOUT_X and LAYOUT_Y, respectively (which include all 3 new versions of the RRFS_CONUS grids), and thus the quantity (NX/LAYOUT_X)*(NY/LAYOUT_Y) is not evenly divisible by BLOCKSIZE gave warnings of this kind in the log file for the run_fcst task: `WARNING from PE 478: atmos_modeldefine_blocks_packed: domain ( 59 65) is not an even divisor with definition ( 32) - blocks will not be uniform with a remainder of 27 ` This warning should either be removed (since this restriction on BLOCKSIZE has been removed from the code) or, if having this remainder makes the model run less efficiently, be replaced by a different caution message.
… orography, and surface climatology files) in NCO mode; treat fixed files in NCO mode more in line with that in community mode
## DESCRIPTION OF CHANGES:
### Main changes:
* In NCO mode, remove requirement that there be "fix_am" and "fix_lam" symlinks under a directory named "fix" under "regional_workflow". Instead, move "fix_am" and "fix_lam" to under the experiment directory, and have "fix_am" be a symlink like before but (for consistency with what is done in community mode) have "fix_lam" be an actual directory (not a symlink) that contains symlinks to files in some other specified directory.
* For clarity, move creation of the "fix_am" symlink in NCO mode from setup.sh to generate_FV3LAM_wflow.sh, and make it part of the same if-statement in generate_FV3LAM_wflow.sh that creates the "fix_am" directory in community mode.
* Assume that in NCO mode, the pregenerated grid, orography, and surface climatology files are located in the directory ${FIXLAM_NCO_BASEDIR}/${PREDEF_GRID_NAME}, where FIXLAM_NCO_BASEDIR is a new workflow variable that specifies the base directory in which these files are located. Thus, the files must be located in a subdirectory having the name of the predefined grid under ${FIXLAM_NCO_BASEDIR}. Note that FIXLAM_NCO_BASEDIR is only used in NCO mode.
* With the above changes, the creation of symlinks in the "fix_lam" subdirectory under the experiment directory and the setting of the variable RES_IN_FIXLAM_FILENAMES can now be handled in the same way in NCO mode as in community mode. Thus, remove section of code in setup.sh that sets RES_IN_FIXLAM_FILENAMES specifically in NCO mode, and use the section of code that was previously only for community mode to create the symlinks and set RES_IN_FIXLAM_FILENAMES in both NCO and community modes.
* Remove check near the top of setup.sh on existence of SFC_CLIMO_DIR when RUN_TASK_MAKE_SFC_CLIMO is set to "FALSE" because this test is in effect now performed later in the script, both for NCO mode and community mode.
* For simplicity, remove setting of SFC_CLIMO_DIR to a null string when it is not already a null string and RUN_TASK_MAKE_SFC_CLIMO is set to "TRUE" because this is not done for GRID_DIR and OROG_DIR.
* For the purposes of running the WE2E tests, assume that when using pregenerated grid, orography, and surface climatology files in community mode, the files are located directly under the ${PREDEF_GRID_NAME} subdirectory of some specified base directory just as we assume is the case in NCO mode. Thus, in run_experiments.sh, remove the "grid", "orog", and "sfc_climo" subdirectories in the settings of GRID_DIR, OROG_DIR, and SFC_CLIMO_DIR. Note that this is only for the WE2E testing system. Users can still specify (in config.sh) independent directories for GRID_DIR, OROG_DIR, and SFC_CLIMO_DIR when using pregenerated files in community mode.
* In the sample NCO mode workflow configuration file config.nco.sh, added the variable FIXLAM_NCO_BASEDIR (for a valid value on Hera only; values are platform and user dependent).
### Improvements:
* Add missing code that checks that RUN_TASK_MAKE_OROG is set to a valid value (just like there is code to check that RUN_TASK_MAKE_GRID and RUN_TASK_MAKE_SFC_CLIMO are set to valid values).
* Edit error messages in setup.sh to make them more accurate and clearer.
* Clean up comments.
## TESTS CONDUCTED:
Ran the following WE2E tests on Hera:
* nco_CONUS_25km_GFDLgrid
* nco_RRFS_CONUS_25km_HRRR_RAP
* pregen_grid_orog_sfc_climo
* regional_002
These are ones that test the workflow in NCO mode (nco_CONUS_25km_GFDLgrid, nco_RRFS_CONUS_25km_HRRR_RAP), in community mode with pregenerated files (pregen_grid_orog_sfc_climo), and in community mode without pregenerated files (regional_002). All tests were successful.
…ll as before) for both nemsio and grib2 formats (ufs-community#391) ## DESCRIPTION OF CHANGES: * Modify workflow script get_extrn_mdl_dir_file_info.sh to enable fetching of FV3GFS files from NOAA HPSS on and after 202002600, which was broken due to a change in file naming convention on HPSS starting on this date. * Files before this date (and after 20190612, which is the start of operations for FV3GFS) can still be fetched as usual. * New code works for both nemsio-formatted and grib2-formatted FV3GFS files on HPSS. * Add 10 new WE2E test configurations to test this capability (both for nemsio and of grib2 formatted files) for five different dates. Dates chosen as follows: * 2019061200: Date on which FV3GFS becomes operational. Appears in if-statement in get_extrn_mdl_dir_file_info.sh. * 2020022600: Date on which there is a change in the naming convention of FV3GFS files on HPSS. Appears in if-statement in get_extrn_mdl_dir_file_info.sh. * 2020022518: Date right before the one on which there is a change in the naming convention of FV3GFS files on HPSS (i.e. 2020022600). * 2019101800: Date about halfway between the date on which FV3GFS becomes operational (2019061200) and the one on which there is the change in file naming convention (2020022600). * 2021010100: Date after the one on which there is a change in file naming convention (2020022600) and is very recent. * Rename the preexisting WE2E tests that get files from NOAA HPSS so that they use the same naming convention as the five new WE2E tests. ## TESTS CONDUCTED: On Hera, ran the 10 new WE2E tests for fetching FV3GFS files from HPSS for various dates. All succeeded.
…community#387) * Fix stochastic physics block to get it working. Change tm${cyc} to tm00 in post script. * Add tmmark description.
…idation meeting. (ufs-community#394) * Namelist changes following GSL, EMC, and DTC namelist consolidation meeting * Change ttendlim from 50 to -999
* Updates to plot_allvars.py * Updated Instructions and Forced Forecast Interval to be INT - Updated instructions to include start fhr, end fhr, and interval. - Changed np.linspace to include "dtype='int'". This option requires NumPy v1.9.0 to work. - Added int() to the num calculation (~line 250) to also make it work with np.linspace. * Enhancements to plot_allvars_diff.py * Update plot_allvars.py * Update plot_allvars_diff.py Co-authored-by: David Wright <[email protected]>
…predefined grids (ufs-community#396) ## DESCRIPTION OF CHANGES: * When using a predefined grid, allow the default values of DT_ATMOS, LAYOUT_X, LAYOUT_Y, and BLOCKSIZE for that grid to be overwritten with values specified by the user in the experiment configuration file. * Add a WE2E test (config.grid_RRFS_CONUS_25km_modify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE.sh) that uses the RRFS_CONUS_25km predefined grid but changes DT_ATMOS, LAYOUT_X, LAYOUT_Y, and BLOCKSIZE from the default values for that grid. * Update comments. ## TESTS CONDUCTED: * Ran the new WE2E test (grid_RRFS_CONUS_25km_modify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE) on Hera. It was successful.
Fix to run workflow tasks on the WCOSS dell and cray
…onal parameters; bugfix for setting FXILAM_NCO_BASEDIR in WE2E tests; other improvements. (ufs-community#403)
…community#412) * Add submission scripts for python scripts for hera specifically * Updated scripts to remove hard coded paths and incorporate better practices and helpful usage statements * Typo in usage statement - there are six command line arguements
…community#414) The main modification that this PR makes is that the `exregional_make_ics.sh` and `exregional_make_lbcs.sh` scripts no longer use long (and error-prone) if-statements over physics suites to set the `chgres_cube` namelist variable `thomp_mp_climo_file`; instead, the function `set_thompson_mp_fix_files.sh` that already checks whether Thompson MP is included in the SDF now also returns the new workflow variable `THOMPSON_MP_USED` that indicates whether or not Thompson MP is called by the physics suite, and the `exregional_make_ics.sh` and `exregional_make_lbcs.sh` scripts then use this flag to set the `chgres_cube` namelist variable `thomp_mp_climo_file` appropriately. The improvement being made here is that the developer no longer needs to remember which suites use Thompson MP and which don't, which must be known in order to maintain the long and repeated if-statements over physics suites currently in `exregional_make_ics.sh` and `exregional_make_lbcs.sh`. ## DESCRIPTION OF CHANGES: * In set_thompson_mp_fix_files.sh, set and output via a new function argument the variable thompson_mp_used. If the Thompson microphysics parameterization is being used by the physics suite, this will be set to "TRUE", otherwise it will be set to "FALSE". * In setup.sh: * Set the new workflow variables THOMPSON_MP_CLIMO_FN and THOMPSON_MP_CLIMO_FP. The first contains the name (but not full path) of the aerosol climatology file that may be needed by Thompson MP, and the second contains the full path of this aerosol climatology file (regardless of whether or not Thompson MP is used in the physics suite and regardless of whether or not the external model files contain aerosol data). * Set the new workflow variable THOMPSON_MP_USED (via the call to the set_thompson_mp_fix_files.sh function) that indicates whether or not Thompson MP is called by the physics suite. * Write all three new workflow variables to the experiment variable defintions file (var_defns.sh). * In exregional_make_ics.sh and exregional_make_lbcs.sh, remove suite-dependent if-statements for setting thomp_mp_climo_file. Instead, use a much simpler if-statement that sets this variable to THOMPSON_MP_CLIMO_FP only if Thompson MP is being used AND the external model does not already provide the aerosol fields needed by Thompson MP (which is all external models except the HRRR and the RAP) and otherwise leaves it set to a null string. Note that thomp_mp_climo_file is passed to the namelist file for chgres_cube. * Add a WE2E test (grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GSD_SAR) that uses the FV3GFS (which does not provide aerosol fields) as the external model for ICs and LBCs and uses the GSD_SAR physics suite (which includes Thompson MP). Thus, the aerosol climatology file Thompson_MP_MONTHLY_CLIMO.nc must be used to provide aerosol data to Thompson MP. This test exercises use of this aerosol climatology file in the workflow. ### Improvements: * Add a script (tests/get_expts_status.sh) that obtains the workflow status of all experiments in a specified experiments base directory. * Takes one argument -- the experiments base directory. This is the directory under which all the experiment subdirectories are located. * Assumes that all subdirectories under the base directory are experiment directories. * For each subdirectory/experiment, calls the workflow launch script to update the status of the workflow for that experiment. It then places the result from the launch script in a summary file named expts_status_${YYYYMMDDHHMN}.txt, where ${YYYYMMDDHHMN} is the 4-digit year, 2-digit month, 2-digit day-of-month, 2-digit hour-of-day, and 2-digit minute of the time at which the script was called. The summary file is placed in the experiments base directory. * For each experiment, it prints to screen the name of the experiment and the status of the corresponding workflow ("IN PROGRESS", "SUCCESS", or "FAILURE"). * Edit error messages for clarity. * Clean up comments. ## TESTS CONDUCTED: The following WE2E tests were run successfully on Hera: * grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 * grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta * grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 * **grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GSD_SAR** * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta * grid_RRFS_CONUS_25km_modify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE * grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta * nco_CONUS_25km_GFDLgrid * nco_RRFS_CONUS_25km_HRRR_RAP The one that directly tests the change in the way the Thompson climatology file is handled is grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GSD_SAR (in bold in the list above). The remaining tests ensure pre-existing functionality is not broken.
## DESCRIPTION OF CHANGES: ### Main Changes: * Add the new function check_ruc_lsm.sh that reads in the SDF to check whether it uses the RUC LSM. If so, it sets the local variable sdf_uses_ruc_lsm to "TRUE", otherwise to "FALSE". It then sets the environment variable whose name is specified by the input argument output_varname_sdf_uses_ruc_lsm to the value of sdf_uses_ruc_lsm. * In setup.sh, introduce new workflow variable SDF_USES_RUC_LSM by passing its name as an argument to the new function check_ruc_lsm.sh. This gets set to "TRUE" if the physics suite uses the RUC LSM and "FALSE" otherwise. * In exregional_make_ics.sh, set the chgres_cube namelist variable nsoill_out (which is the number of soil levels to have in the NetCDF file that chgres_cube generates) by uisng an if-statement that checks the new workflow variable SDF_USES_RUC_LSM as well as the name of the external model used for ICs. This if-statemnt replaces the settings of nsoill_out in the external-model-dependent case-statement and, depending on model, the physics-suite-dependent if-statements within that case statement. This simplifies the script because we no longer need to remember which suites use the RUC LSM and which don't. * In generate_FV3LAM_wflow.sh, set the number of ice levels in the FV3 input file according to whether the RUC_LSM is used in the physics suite (i.e. by checking the new variable SDF_USES_RUC_LSM). ### Improvements: * Change the name of the workflow variable THOMPSON_MP_USED to SDF_USES_THOMPSON_MP to be consistent with the new varable SDF_USES_RUC_LSM. * In set_thompson_mp_fix_files.sh, change the local variable thompson_mp_used to sdf_uses_thompson_mp (to be consistent with the change in the corresponding workflow variable SDF_USES_THOMPSON_MP). ## TESTS CONDUCTED: Ran the following WE2E tests on Hera using the HEAD (hash #ea8a7aa) of the authoritative ufs-community/ufs-weather-model repo: * grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 * grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_GSD_SAR_2020020800 * grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta * grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 * grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GSD_SAR * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_GSD_SAR_2020020800 * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta * grid_RRFS_CONUS_25km_modify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE * grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_GSD_SAR_2020020800 * grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta * nco_CONUS_25km_GFDLgrid * nco_RRFS_CONUS_25km_HRRR_RAP * suite_FV3_GSD_v0 All tests were successful. Note that the three tests whose names end with "_2020020800" are not in the workflow, but they are generated by taking the respective WE2E tests in the workflow with names without the trailing "_2020020800" and simply changing DATE_FIRST_CYCL and DATE_LAST_CYCL from 20200801 to 20200208 and LBC_SPEC_INTVL_HRS from 3 to 1. The change in date was to try out a winter case, which was the situation under which failures were common (the setting of kice to 9 for suites that use the RUC LSM seems to have fixed this winter case problem).
## DESCRIPTION OF CHANGES:
### Main changes:
* Add the FV3_HRRR suite to the workflow. This consists of:
* Adding "FV3_HRRR" to the list of valid values of CCPP_PHYS_SUITE in valid_param_vals.sh.
* Adding the diag_table.FV3_HRRR and field_table.FV3_HRRR files to the ush/templates directory.
* Adding the FV3_HRRR suite to the case-statements in exregional_make_ics.sh and exregional_make_lbcs.sh.
* Adding a stanza to FV3.input.yml to customize the FV3 namelist file for the FV3_HRRR suite.
* Add the following two WE2E tests for the FV3_HRRR suite on the RRFS_CONUS_25km grid. The first is initialized for the 2020 derecho case (2020081000) and updates the LBCs every hour while the second is initialized on 2019052000 and updates the LBCs every 6 hours:
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
* suite_FV3_HRRR
* Add an additional three WE2E tests for the FV3_HRRR suite on the 25km, 13km, and 3km RRFS_CONUS grids. All three are initialized for the 2020 derecho case (2020081000). They are named as follows:
* grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR
* grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_HRRR
* grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR
* Modify the following three WE2E tests that use the FV3_GSD_SAR suite so that they are initialized for the 2020 derecho case (2020081000) instead of 2020080100 (so that they are consistent with the 3 new WE2E tests added above):
* grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_GSD_SAR
* grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_GSD_SAR
* grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_GSD_SAR
* Enable the workflow to find and copy to the experiment directory the pregenerated (for a given grid) orography statistics files `${CRES}_oro_data_ls.tile7.halo0.nc` and `${CRES}_oro_data_ss.tile7.halo0.nc` needed by the gravity wave drag parameterization in FV3_HRRR. (Note that this is a temporary fix. Eventually, a program will be made available that the make_orog task can call to generate these files as part of the workflow for any grid.) Modifications for this purpose consist of:
* Adding to config_defaults.sh the new workflow variable GWD_HRRRsuite_BASEDIR that specifies the base directory in which the aformentioned "ls" and "ss" orography statistics files are located for the predefined grid being used. This workflow variable will be removed once a program is available to generate the "ls" and "ss" files on-the-fly for any grid.
* Adding code to setup.sh that sets the new derived/secondary workflow variable GWD_HRRRsuite_DIR that specifies the directory in which the "ls" and "ss" files are located (as opposed to just the base directory, which is what GWD_HRRRsuite_BASEDIR represents). This is set using GWD_HRRRsuite_BASEDIR and PREDEF_GRID_NAME. Like GWD_HRRRsuite_BASEDIR, this variable will be removed once a program is available to generate the "ls" and "ss" files on-the-fly for any grid.
* Adding code to link_fix.sh that creates symlinks in the fv3_lam directory that point to the "ls" and "ss" files in OROG_DIR.
* Adding code to exregional_make_orog.sh that copies the "ls" and "ss" files from the directory specified by GWD_HRRRsuite_DIR to the orography directory (OROG_DIR). This code will evenutally be replaced by a call to a program that will generate these files and place them in OROG_DIR (instead of copying them).
* Adding code to exregional_run_fcst.sh that creates links in the run directory to the"ls" and "ss" orography files that the FV3 model looks for.
* Adding code to run_experiments.sh that sets GWD_HRRRsuite_BASEDIR in the workflow configuration file (config.sh) to the base directory in which the pregenerated "ls" and "ss" orography files are located. Note that this setting is machine-dependent.
### Improvements:
* Edit error messages in exregional_make_ics.sh and exregional_make_lbcs.sh to make them clearer.
* Correct code indentation and remove blank lines.
* Alphabetize FV3_GSD_SAR stanza in FV3.input.yml.
* In tests/run_experiments.sh, change the location FIXLAM_NCO_BASEDIR on Jet to be a more "official" directory (instead of a personal directory).
* Add the WE2E test grid_RRFS_CONUS_25km_ics_HRRR_lbcs_HRRR_suite_GSD_SAR that uses the HRRRX external model for both the ICs and the LBCs (and uses the FV3_GSD_SAR suite).
* Clean up comments.
## TESTS CONDUCTED:
Ran the following WE2E tests on Hera:
* grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
* grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_GSD_SAR
* grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_GSD_SAR_2020020800
* grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_HRRR
* grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_HRRR_2020020800
* grid_RRFS_CONUS_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GSD_SAR
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
* grid_RRFS_CONUS_25km_ics_HRRR_lbcs_HRRR_suite_GSD_SAR
* grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_GSD_SAR
* grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_GSD_SAR_2020020800
* grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR
* grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR_2020020800
* grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
* grid_RRFS_CONUS_25km_modify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE
* grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_GSD_SAR
* grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_GSD_SAR_2020020800
* grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR
* grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR_2020020800
* grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
* nco_CONUS_25km_GFDLgrid
* nco_RRFS_CONUS_25km_HRRR_RAP
* suite_FV3_GSD_v0
* suite_FV3_HRRR
All tests were successful. Note that the six tests whose names end with "_2020020800" are not in the workflow. They are generated by taking the respective WE2E tests in the workflow with names without the trailing "_2020020800" and simply changing DATE_FIRST_CYCL and DATE_LAST_CYCL from 20200801 to 20200208 and LBC_SPEC_INTVL_HRS from 3 to 1. The change in date was to try out a winter case.
## CONTRIBUTORS (optional):
@mdtoy
* Add default namelist with SPP entries. * Changes necessary to run with SPP * Typo fix in generate script. * Changes to implement SPP. * Add comment regarding use of SPP. * Pass N_VAR_SPP to the var_defns.sh file.
…ther-model. (ufs-community#420) * Remove SPP lines from the input.nml.FV3 template until code is ready in ufs-weather-model * Remote references to SPP in the generate script. * Remove references to SPP in the generate script.
## DESCRIPTION OF CHANGES: * In set_extrn_mdl_params.sh, add stanzas for the "NAM" external model as well as for the "CHEYENNE" platform. * In run_experiments.sh, add stanzas for "NAM". * WE2E tests: * Add 3 new WE2E tests for the NAM on the RRFS_CONUS_25km grid. These use NAM for both the ICs and LBCs, run 24hr forecasts, and use an LBC update interval of 3 hours. They all run the 2015060212 case, which is currently the only date for which NAM data is available (provided by Bill Gallus and Jonathan Thielen). The difference between the three tests is that they run the FV3_GSD_SAR, FV3_HRRR, and FV3_RRFS_v1beta suites, respectively. * Modify the four WE2E tests that run on the RRFS_CONUS_25km grid with HRRR/RAP ICs/LBCs so that they all use the 2020 Derecho case (2020081000) and go out to 24 hours (with a boundary update every 3 hours). This is because the Derecho case is of more interest to the community than the (randomly chosen) previous case 2020080100. The four tests run using the GSD_SAR, HRRR, RRFS_v1alpha, and RRFS_v1beta suites, respectively. * For comparison purposes between the ICs/LBCs combinations of NAM/NAM, HRRR/RAP, and HRRR/HRRR, add 2 WE2E tests on the RRFS_COUNS_25km grid that use HRRR for both ICs and LBCs (and run the 2020 Derecho case for 24 hrs with a 3-hour boundary update). * Modify script that gets experiments' workflow status so that it ignores all non-experiment directories as well as all "inactive" (i.e. renamed) experiment directories under the specified experiments base directory. * Modify the default value of DT_ATMOS on the RRFS_CONUS_25km grid to be 40sec (due to test results below). ## TESTS CONDUCTED: On Cheyenne, ran the following tests on the RRFS_CONUS_25km grid with both DT_ATMOS=300sec and DT_ATMOS=40sec. The 40sec tests all succeeded (although some with error messages) while some of the 300sec tests succeeded while others failed. Full results are as follows: * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_HRRR_suite_GSD_SAR * DT_ATMOS = 300sec: succeeded * DT_ATMOS = 40sec: succeeded * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_HRRR_suite_HRRR * DT_ATMOS = 300sec: succeeded * DT_ATMOS = 40sec: succeeded * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta * DT_ATMOS = 300sec: succeeded * DT_ATMOS = 40sec: succeeded * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_GSD_SAR * DT_ATMOS = 300sec: succeeded * DT_ATMOS = 40sec: succeeded * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR * DT_ATMOS = 300sec: succeeded * DT_ATMOS = 40sec: succeeded * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta * DT_ATMOS = 300sec: succeeded * DT_ATMOS = 40sec: succeeded * grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GSD_SAR (CDATE = 2015060212) * DT_ATMOS = 300sec: failed * DT_ATMOS = 40sec: succeeded * grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_HRRR (CDATE = 2015060212) * DT_ATMOS = 300sec: failed * DT_ATMOS = 40sec: succeeded * grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta (CDATE = 2015060212) * DT_ATMOS = 300sec: failed * DT_ATMOS = 40sec: succeeded It is not yet clear what causes the failures with the NAM experiments, but since these succeed with DT_ATMOS=40sec, we change the default value of DT_ATMOS to 40 sec for the RRFS_CONUS_25km grid. ## CONTRIBUTORS (optional): Bill Gallus
…uite in NCO mode (ufs-community#421) @chunhuazhou (Chunhua Zhou) reported that GWD_HRRRsuite_BASEDIR is not getting set correctly when running the FV3_HRRR suite in NCO mode. This PR is to fix that bug. ## DESCRIPTION OF CHANGES: ### Main changes: * In setup.sh, ensure that GWD_HRRRsuite_BASEDIR is set properly when running the FV3_HRRR suite in NCO mode, and add checks to make sure directories exist, etc. * Add 2 new WE2E tests for running the FV3_HRRR suite in NCO mode on the RRFS_CONUS_25km and RRFS_CONUS_3km grids. The test names are nco_grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR and nco_grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR. These run the 2020020800 case, which is a winter case that has a history of crashing. ### Improvements: * Remove the regional_workflow/env directory since it is unused. * Change the names of 5 of the existing NCO-mode WE2E tests to make it clearer what configurations they run (and to be consistent with the naming convention of community-mode tests). ## TESTS CONDUCTED: Ran the following WE2E tests (including the two new ones) on Hera. All were successful. * grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR * nco_grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR * nco_grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR (both on 2020020800 and 2020081006)
…ity#424) * Changes required to process global NetCDF files. * Add netcdf option to valid_param_vals.sh * Remove nemsio from variable name. * Update comments for fn_grib2, fn_atm, and fn_sfc variables. * Remove netcdf FV3GFS filename and path syntax from HPSS code, add comments to explain to user.
Co-authored-by: Chan-Hoo Jeon <[email protected]>
* Add options for wcoss to we2e tests * Make cron work on wcoss_dell_p3 * Modify the baseline list file
Remove all fdiag entries from FV3.input.yml
…s-community#439) ## DESCRIPTION OF CHANGES: Modified the Jinja-formatted FV3LAM_wflow.xml template workflow to accommodate sub-hourly post-processing tasks that rely on sub-hourly FV3 output as a dependency. All changes are _additions_ to existing code and include the addition of a few keyword variables in the config.sh script. These new flags include... - SUB_HOURLY_POST: a logical flag indicating whether nor not sub-hourly post-processing is to be used - DT_SUBHOURLY_POST_MNTS: the increment in minutes to sub-divide the hour Additional post-processing tasks were added to FV3LAM_wflow.xml to account for the different FV3 output file names depending on whether sub-hourly FV3 output is used (the first FV3 output file has a different naming structure than the remaining output files). setup.sh was updated to check whether valid entries were used for these two variables and also check that DT_ATMOS divides evenly into DT_SUBHOURLY_POST_MNTS so that the FV3 output is consistent with the requested frequency of UPP output. config_defaults.sh and valid_param_vals.sh were also updated accordingly. ## TESTS CONDUCTED: Have run generate_FV3LAM_wflow.sh on a large variety of settings of SUB_HOURLY_POST and DT_SUBHOURLY_POST_MNTS. Note that setup.sh is configured such that DT_SUBHOURLY_POST_MNTS = 0 will cause SUB_HOURLY_POST to be ignored. I have successfully tested cases in which DT_ATMOS _does not_ divide evenly into DT_SUBHOURLY_POST_MNTS and when DT_SUBHOURLY_POST_MNTS is specified as anything other than a two-digit value (strings vs. open integers both work). The resulting workflows run successfully with rocotorun and output no error messages. ## ISSUE: Resolves issue ufs-community#434 ## CONTRIBUTORS: @gsketefian. Contributions: 1) Fixed bug in setup.sh in the test that checks whether DT_SUBHOURLY_POST_MNTS is set to 0: should use the -eq operator instead of ==. 2) Fixed bug in the jinja XML template for rocoto (FV3LAM_wflow.xml) as follows: rearranged the post-processing tasks so that the post task is run for only the first minute of the last hour (e.g. if the forecast is 3 hours long, post is run for 3:00 but not for 3:15, 3:30, etc). 3) Ran the following 3 WE2E tests [note that tests (b) and (c) are not yet in the regional_workflow repo and will be included in a future PR]: a) **grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta**. This is without subhourly post-processing, i.e. SUB_HOURLY_POST is set to "FALSE". b) **subhourly_post**. This is with subhourly post-processing, i.e. SUB_HOURLY_POST set is set to "TRUE" (with DT_SUBHOURLY_POST_MNTS set to "12" minutes). c) **subhourly_post_ensemble_2mems**. This is with subhourly post-processing and with ensemble forecasts enabled, i.e. SUB_HOURLY_POST and DO_ENSEMBLES are both set to "TRUE" (with DT_SUBHOURLY_POST_MNTS is set to "12" minutes and NUM_ENS_MEMBERS set to "2" members). This test is run because the changes in the jinja XML template FV3LAM_wflow.xml needed to add subhourly post invovle code that executes ensemble forecasts. **All three tests were successful.** Note that this PR changes the names of the grib2 files that UPP generates such that they now always include the minutes -- regardless of whether SUB_HOURLY_POST is set to "TRUE" or "FALSE" (if set to "FALSE", the minutes are always "00"). For example, previously, the grib2 file for forecast hour 1 was named `rrfs.t00z.bgdawpf001.tm00.grib2`; henceforth, it will be named `rrfs.t00z.bgdawpf00100.tm00.grib2`.
* Fix to post flat file. * Create MET and METplus config files under ush/templates/parm * Added script to pull and reorg ccpa data. Added a script to run gridstat with METplus. Updated MET and METplus config files. * Added new jjob for running grid-stat vx. Updated setup.sh to include grid-stat vx. Updated run_gridstatvx script. * Fixed typo on script name from ksh to sh * Moved some hard coded items out from the script to the XML * Updates to get METplus to run with fewer hard-coded paths. * Updates to add grid-stat task to XML generation. * Bug fixes for adding grid-stat to XML generation * Updates to remove hard-coded paths in config files * Change log dir to put master_metplus log file with other logs under log/, rather than default logs/. * Updates to generate xml without hard-coded paths for MET * Add hera gridstat module file * Add METplus point-stat task for both sfc and upper air * Small tweaks to remove hard coded paths and add some flexibility * Updates for adding point-stat into auto-generated xml * Add in function to set point-stat task to FALSE * Final tweaks to get it to generate the xml correctly * Minor updates to run ensure 0,6,12,18 * Tweaks to var list for Point-Stat * Add METplus settings to config_defaults * Move quote for end of settings and fix extra comment. * Fix typos to populate templates correctly * Updated to include SCRIPTSDIR and other MET specific settings along with updates to FHR syntax * Update module loads on hera * Fixed comment for BOTH_VARn_THRESH to avoid syntax issues * Added files to run grid_stat for a variety of accumulation intervals, including 3, 6, and 24h * Added module load hpss * Remove module load informatino from these scripts * Updated the method of turning on/off vx tasks using jinja template if statement * Remove commented out lines of code. Fixed typo. Removed gen_wflow.out file. * Updated pull scripts to have file names dependent on date to pull from HPSS. Updated to export a few more local variables that METplus conf needed in scripts. Updated workflow to use service queue (for now) to for 1h grid_stat and point_stat run and default for 3+h accumulation grid_stat runs) * moved common_hera.conf to common.conf - no platform specific information included that needs to be handled. * Remove common_hera.conf * Add scripts to pull and process MRMS data from NOAA HPSS * Updates for REFC vx tasks * updates to obs pull scripts * Update for adding in reflectivity verification using MRMS analyses and updating name of model output to RRFS rather than HRRR * Updates to account for CCPA issues on HPSS - day off for 00-05 UTC directories * Verification mods to feature/add metplus (#1) * Remove unused/outdated code (ufs-community#313) ## DESCRIPTION OF CHANGES: * In setup.sh and generate_FV3LAM_wflow.sh, remove temporary codes that fix bugs in the FV3_GFS_2017_gfdlmp_regional suite definition file because those bugs have been fixed (in the ufs-weather-model repo). * In setup.sh, remove block of code that is no longer necessary because chgres_cube can now initialize from external model data with either 4 or 9 soil levels, and run with LSMs of either 4 or 9 soil levels. * Remove modifications to LD_LIBRARY_PATH in exregional_run_fcst.sh. * For the make_ics and make_lbcs tasks, move the setting of APRUN and other machine-specific actions from the J-job to the ex-script in order to be consistent with the other workflow tasks. * Fix indentation and edit comments. * Remove unused file load_fv3gfs_modules.sh. ## TESTS CONDUCTED: Ran two WE2E tests on hera, new_ESGgrid and new_GFDLgrid: * new_ESGgrid uses the FV3_GFS_2017_gfdlmp_regional suite. The test was successful. * new_GFDLgrid uses the FV3_GFS_2017_gfdlmp suite. The test was successful. ## ISSUE (optional): This resolves issue ufs-community#198. * Add and call a function that checks for use of Thompson microphysics parameterization in the SDF and if so, adjusts certain workflow arrays to contain the names and other associated values of the fixed files needed by this parameterization so that those files are automatically copied and/or linked to. (ufs-community#319) ## DESCRIPTION OF CHANGES: Add and call a function that checks for use of Thompson microphysics parameterization in the suite definition file (SDF). If not, do nothing. If so, add to the appropriate workflow arrays the names and other associated values of the fixed files needed by this parameterization so that they are automatically copied and/or linked to instead of being regenerated from scratch in the run_fcst task. ## TESTS CONDUCTED: On hera, ran two WE2E tests, one in NCO mode (nco_RRFS_CONUS_25km_HRRRX_RAPX) and the other in community mode (suite_FV3_GSD_v0). These use suites FV3_GSD_SAR and FV3_GSD_v0, respectively, and both of these call Thompson microphysics. Both succeeded. ## ISSUE (optional): This PR resolves issue ufs-community#297. * RRFS_v1beta SDF changes after reverting from GSL to GFS GWD suite (ufs-community#322) (ufs-community#327) ## DESCRIPTION OF CHANGES: Removed checks on the RRFS_v1beta SDF implemented for use with the GSL GWD suite (now uses the GFS GWD suite). No longer copies staged orography files necessary for the GSL GWD suite. ## TESTS CONDUCTED: Runs to completion on Hera. End-to-end runs DOT_OR_USCORE and suite_FV3_RRFS_v1beta succeeded on Cheyenne. Co-authored-by: JeffBeck-NOAA <[email protected]> * Update FV3.input.nml for fhzero = 1.0 * Updated conf files for file name conventions. * Updated MET scripts and MRMS pull scripts. * Adjust RRFS_CONUS_... grids (ufs-community#294) ## DESCRIPTION OF CHANGES: * Adjust RRFS_CONUS_25km, RRFS_CONUS_13km, and RRFS_CONUS_3km grid parameters so that: * All grids, including their 4-cell-wide halos, lie completely within the HRRRX domain. * All grids have dimensions nx and ny that factor "nicely", i.e. they don't have factors greather than 7. * The write-component grids corresponding to these three native grids cover as much of the native grids as possible without going outside of the native grid boundaries. The updated NCL scripts (see below) were used to generate the write-component grid parameters. * For the RRFS_CONUS_13km grid, reduce the time step (DT_ATMOS) from 180sec to 45sec. This is necessary to get a successful forecast with the GSD_SAR suite, and thus likely also the RRFS_v1beta suite. * Modify WE2E testing system as follows: * Add new tests with the RRFS_CONUS_25km, RRFS_CONUS_13km, and RRFS_CONUS_3km grids that use the GFS_v15p2 and RRFS_v1beta suites (which are now the ones officially supported in the first release of the short-range weather app) instead of the GFS_v16beta and GSD_SAR suites, respectively. * For clarity, rename the test configuration files that use the GFS_v16beta and GSD_SAR suites so they include the suite name. * Update list of WE2E tests (baselines_list.txt). * Update the NCL plotting scripts to be able to plot grids with the latest version of the workflow. ## TESTS CONDUCTED: On hera, ran tests with all three grids with the GFS_v15p2 and RRFS_v1beta suites (a total of 6 tests). All were successful. * Remove redundant model_configure.${CCPP_PHYS_SUITE} template files; use Jinja2 to create model_configure (ufs-community#321) ## DESCRIPTION OF CHANGES: * Remove model_configure template files whose names depend on the physics suite, i.e. files with names of the form model_configure.${CCPP_PHYS_SUITE}. Only a single template file is needed because the contents of the model_configure file are not suite dependent. This leaves just one template file (named model_configure). * Change the function create_model_configure_file.sh and the template file model_configure so they use jinja2 instead of sed to replace placeholder values. * Absorb the contents of the write-component template files wrtcmp_lambert_conformal, wrtcmp_regional_latlon, and wrtcmp_rotated_latlon into the new jinja2-compliant model_configure file. We can do this because Jinja2 allows use of if-statements in the template file. * In the new model_configure jinja2 template file, include comments to explain the various write-component parameters. ## TESTS CONDUCTED: On Hera, ran the two WE2E tests new_ESGgrid and new_GFDLgrid. The first uses a "lambert_conformal" type of write-component grid, and the second uses a "rotated_latlon" type of write-component grid. (The write-component also allows "regional_latlon" type grids, which is just the usual earth-relative latlon coordinate system, but we do not have any cases that use that.) Both tests succeeded. ## ISSUE (optional): This PR resolves issue ufs-community#281. * Add Thompson ice- and water-friendly aerosol climo file support (ufs-community#332) * Add if statement in set_thompson_mp_fix_files.sh to source Thompson climo file when using a combination of a Thompson-based SDF and non-RAP/HRRR external model data * Modify if statement based on external models for Thompson climo file * Remove workflow variable EMC_GRID_NAME (ufs-community#333) ## DESCRIPTION OF CHANGES: * Remove the workflow variable EMC_GRID_NAME. Henceforth, PREDEF_GRID_NAME is the only variable that can be used to set the name of the predefined grid to use. * Make appropriate change of variable name (EMC_GRID_NAME --> PREDEF_GRID_NAME) in the WE2E test configuration files. * Change anywhere the "conus" and "conus_c96" grids are specified to "EMC_CONUS_3km" and "EMC_CONUS_coarse", respectively. * Rename WE2E test configuration files with names containing the strings "conus" and "conus_c96" by replacing these strings with "EMC_CONUS_3km" and "EMC_CONUS_coarse", respectively. * Update the list of WE2E test names (tests/baselines_list.txt). * Bug fixes not directly related to grids: * In config.nco.sh, remove settings of QUEUE_DEFAULT, QUEUE_HPSS, and QUEUE_FCST since these are now set automatically (due to another PR). * In the template file FV3LAM_wflow.xml, add the ensemble member name after RUN_FCST_TN in the dependency of the run_post metatask. ## TESTS CONDUCTED: Since this change only affects runs in NCO mode, the following NCO-mode WE2E tests were rerun on hera, all successfully: ``` nco_EMC_CONUS_3km SUCCESS nco_EMC_CONUS_coarse SUCCESS nco_EMC_CONUS_coarse__suite_FV3_GFS_2017_gfdlmp SUCCESS nco_RRFS_CONUS_25km_HRRRX_RAPX SUCCESS nco_RRFS_CONUS_3km_FV3GFS_FV3GFS SUCCESS nco_RRFS_CONUS_3km_HRRRX_RAPX SUCCESS nco_ensemble SUCCESS ``` * Port workflow to Orion (ufs-community#309) ## DESCRIPTION OF CHANGES: * Add stanzas for Orion where necessary. * Add new module files for Orion. * On Orion, both the slurm partition and the slurm QOS need to be specified in the rocoto XML in order to be able to have wall times longer than 30 mins (the partition needs to be specified because it is by default "debug", which has a limit of 30 mins). Thus, introduce modifications to more easily specify slurm partitions: * Remove the workflow variables QUEUE_DEFAULT_TAG, QUEUE_HPSS_TAG, and QUEUE_FCST_TAG that are currently used to determine whether QUEUE_DEFAULT, QUEUE_HPSS, and QUEUE_FCST specify the names of queue/QOS's or slurm partitions. * Add the workflow variables PARTITION_DEFAULT_TAG, PARTITION_HPSS_TAG, and PARTITION_FCST_TAG. These will be used to specify slurm partitions only, and the variables QUEUE_DEFAULT, QUEUE_HPSS, and QUEUE_FCST will be used to specify queues/QOS's only. IMPORTANT NOTE: On Orion, in order to load the regional_workflow environment needed for generating an experiment, the user must first issue the following commands: ``` module use -a /apps/contrib/miniconda3-noaa-gsl/modulefiles module load miniconda3 conda activate regional_workflow ``` ## TESTS CONDUCTED: Ran 11 WE2E tests on Orion, Hera, and Cheyenne. Results on Orion: ``` community_ensemble_2mems SUCCESS DOT_OR_USCORE SUCCESS grid_GSD_HRRR_AK_50km FAILURE - In the run_fcst task. * Error message: !!! (1) Error in subr radiation_aerosols: unrealistic surface pressure = 1 NaN new_ESGgrid SUCCESS new_GFDLgrid SUCCESS regional_001 SUCCESS regional_002 SUCCESS suite_FV3_GFS_v15p2 SUCCESS suite_FV3_GFS_v16beta SUCCESS suite_FV3_GSD_SAR SUCCESS suite_FV3_GSD_v0 SUCCESS ``` Results on Hera: ``` community_ensemble_2mems SUCCESS DOT_OR_USCORE SUCCESS grid_GSD_HRRR_AK_50km SUCCESS new_ESGgrid SUCCESS new_GFDLgrid SUCCESS regional_001 SUCCESS regional_002 SUCCESS suite_FV3_GFS_v15p2 SUCCESS suite_FV3_GFS_v16beta SUCCESS suite_FV3_GSD_SAR SUCCESS suite_FV3_GSD_v0 SUCCESS ``` Results on Cheyenne: ``` community_ensemble_2mems SUCCESS DOT_OR_USCORE SUCCESS grid_GSD_HRRR_AK_50km FAILURE - In run_fcst task. * Error message: !!! (1) Error in subr radiation_aerosols: unrealistic surface pressure = 1 NaN new_ESGgrid SUCCESS new_GFDLgrid SUCCESS regional_001 SUCCESS regional_002 SUCCESS suite_FV3_GFS_v15p2 SUCCESS suite_FV3_GFS_v16beta SUCCESS suite_FV3_GSD_SAR SUCCESS suite_FV3_GSD_v0 SUCCESS ``` All succeed except GSD_HRRR_AK_50km on Orion and Cheyenne. It is not clear why grid_GSD_HRRR_AK_50km fails on Orion and Cheyenne but not Hera. Seems to point to a bug in the forecast model. These two failures are not so important since this grid will soon be deprecated. Also tested successfully on Jet by @JeffBeck-NOAA and on Odin and Stampede by @ywangwof. ## ISSUE: This resolves Issue ufs-community#152. ## CONTRIBUTORS: @JeffBeck-NOAA @ywangwof @christinaholtNOAA * Removed comments from exregional_get_mrms_files.sh and removed fhzero from FV3.input.yml * Update FV3.input.nml for fhzero = 1.0 * Updated conf files for file name conventions. * Updated MET scripts and MRMS pull scripts. * Removed comments from exregional_get_mrms_files.sh and removed fhzero from FV3.input.yml Co-authored-by: gsketefian <[email protected]> Co-authored-by: Michael Kavulich <[email protected]> Co-authored-by: JeffBeck-NOAA <[email protected]> Co-authored-by: Jamie Wolff <[email protected]> * Change cov_thresh for REFL to be a true max in nbrhood as SPC does. * Separated Pull Data Scripts from Run Vx Scripts: Feature/add_metplus (ufs-community#2) * Job script for get_obs_ccpa * Jobs script for get_obs_mrms * Jobs script for get_obs_ndas * Added external variables necessary to get_ccpa script * Updated workflow template with separate get obs tasks * Separated pull scripts from run scripts * Added necessary defaults/values for defining pull tasks * Added module files, default config.sh options, and changed dependencies for vx tasks * Changed name of new workflow to FV3LAM_wflow.xml * Added task get_obs_tn, removed config.sh, updated config_defaults and config.community.sh * Adjusted the community and default config files based on comments * Updated FV3LAM workflow * Fixed discrepancies in config.community.sh * Fixed discrepancies in config_defaults.sh * Fixed discrepancies in config_defaults.sh round 2 * Fixed discrepancies in config_defaults.sh round 3 * Fixed discrepancies in config_defaults.sh round 4 * Fixed discrepancies in config.community.sh round 2 * Fixed discrepancies in config.community.sh round 3 * Fixed discrepancies in generate_FV3LAM_wflow.sh * Fixed discrepancies in generate_FV3LAM_wflow.sh round 2 * Fixed discrepancies in generate_FV3LAM_wflow.sh round 3 * Updated FV3LAM_wflow template * Fixed Vx Task Dependencies in Workflow: Feature/add metplus (ufs-community#3) * Job script for get_obs_ccpa * Jobs script for get_obs_mrms * Jobs script for get_obs_ndas * Added external variables necessary to get_ccpa script * Updated workflow template with separate get obs tasks * Separated pull scripts from run scripts * Added necessary defaults/values for defining pull tasks * Added module files, default config.sh options, and changed dependencies for vx tasks * Changed name of new workflow to FV3LAM_wflow.xml * Added task get_obs_tn, removed config.sh, updated config_defaults and config.community.sh * Adjusted the community and default config files based on comments * Updated FV3LAM workflow * Fixed discrepancies in config.community.sh * Fixed discrepancies in config_defaults.sh * Fixed discrepancies in config_defaults.sh round 2 * Fixed discrepancies in config_defaults.sh round 3 * Fixed discrepancies in config_defaults.sh round 4 * Fixed discrepancies in config.community.sh round 2 * Fixed discrepancies in config.community.sh round 3 * Fixed discrepancies in generate_FV3LAM_wflow.sh * Fixed discrepancies in generate_FV3LAM_wflow.sh round 2 * Fixed discrepancies in generate_FV3LAM_wflow.sh round 3 * Updated FV3LAM_wflow template * Fixed the dependencies of the vx tasks * Manual merge with develop that didn't seem to work before. Trying to get feature branch updated so it will run again! * Add local module files * Add environment variable for SCRIPTSDIR * Remove echo statement * Remove old module files * Update to config_default for walltime for ndas pull. Update to metplus parm for obs file template. Update to FV3LAM xml to not include 00 hour for verification * Update template to remove full path * Verification channges for obs. (ufs-community#4) * Verification channges for obs. * Update config_defaults.sh for vx description * Update config_defaults.sh to remove extraneous MET info. Co-authored-by: Michelle Harrold <[email protected]> * Pull in updates from develop that were not merging properly. Small change to config.community to turn off vx tasks by default. * Did manual merge of these files because it was not handled properly automatically * Adding additional variables to METplus for regional workflow (ufs-community#5) * Updates to address comments in PR review * Changed name of get_obs to remove _tn * Missed removal of on get_obs Co-authored-by: michelleharrold <[email protected]> Co-authored-by: gsketefian <[email protected]> Co-authored-by: Michael Kavulich <[email protected]> Co-authored-by: JeffBeck-NOAA <[email protected]> Co-authored-by: Lindsay <[email protected]> Co-authored-by: Michelle Harrold <[email protected]> Co-authored-by: PerryShafran-NOAA <[email protected]>
…munity#457) * Updating develop branch with updates from the release branch mostly focused on python plotting but also including the license. * Changed named of post file read in to account for additional 00 (min placeholder) * Removed 00 to account for subhourly data as that will be handled separately
…unity#458) ## DESCRIPTION OF CHANGES: * Rename GSD_RAP13km grid to RRFS_NA_13km. * In ush/set_predef_grid_params.sh: * Change parameters of the (newly renamed) RRFS_NA_13km grid to make its write-component grid be completely within the native grid. * For the (newly renamed) RRFS_NA_13km grid, automate calculation of WRTCMP_dlon and WRTCMP_dlat from ESGgrid_DELX and ESGgrid_DELY, respectively. * NCL scripts: * Make modifications to make scripts more general and have the find_wrtcmp_grid_params.ncl script work for a write-component grid of type "rotated_latlon". * Remove unused code in NCL scripts. * Fix up comments. ## TESTS CONDUCTED: Ran the (renamed) WE2E test grid_RRFS_NA_13km on Hera. Was successful. ## CONTRIBUTORS (optional): This grid was also tested by @jaymes-kenyon on Jet.
…"--cpus-per-task" value. (ufs-community#459) * Create user-defined OMP variables. * Add OMP variables to make_sfc_climo task * Move OMP stanza in exregional_run_fcst.sh file * Add "OpenMP" to comment. * Make --cpus-per-task a user-defined variable. * Revert to "srun" for the RUN_FCST task on Jet * Remove export commands from ex-scripts, remove OMP options from make_grid and make_orog tasks. * Add OMP options to make_orog ex-script. * Remove KMP_AFFINITY setting from WCOSS_CRAY block. * Remove exports from all commands in exregional_make_orog.sh and update comments in all relevant ex-scripts.
* Modify path to external GFS data * Modify path to external GFS data for WE2E
Co-authored-by: Natalie Perlin <[email protected]>
…2E test info (ufs-community#776) ## DESCRIPTION OF CHANGES: This PR modifies the script `get_WE2Etest_names_subdirs_descs.sh` that creates the CSV file containing test information the columns for relative cost and time step (`DT_ATMOS`). This allows users to have a better idea of what the cost of each test is (the relative cost is defined such that 1 unit represents the cost of running a single 6-hour forecast on the `RRFS_CONUS_25km` grid). Other related changes: * Add the flag `generate_csv_file` to the `run_WE2E_tests.sh` script to allow users to skip the CSV file generation step (since it can take a while). * Use arguments for the function `set_predef_grid_params()` so that it can be called not just from the experiment generation scripts but also others (in this case from `get_WE2Etest_names_subdirs_descs.sh`). Necessary for this PR. * Move some constants (`NH0`, `NH3`, `NH4`) from `config_defaults.sh` to `constants.sh` to make it easier to access them. * Place the contents of `constants.sh` at the start of each experiment's `var_defns.sh` so that these constants are available to the j-jobs and ex-scripts when `var_defns.sh` is sourced. * Use variable `valid_vals_BOOLEAN` in many more places (which allows many other `valid_vals_...` variables in `valid_param_vals.sh` and `setup.sh` to be removed). * Several other bug fixes. ## TESTS CONDUCTED: The following tests (with results) were conducted on Hera. There were only three (previously existing) failures. ``` * grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16: SUCCESS * grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp: SUCCESS * grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional: SUCCESS * grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2: SUCCESS * grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16: SUCCESS * grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR: SUCCESS * grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_2017_gfdlmp: SUCCESS * grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2: SUCCESS * grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16: FAILURE * grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_HRRR: SUCCESS * grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta: SUCCESS * grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16: SUCCESS * grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR: SUCCESS * grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta: SUCCESS * grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR: SUCCESS * grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha: SUCCESS * grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta: SUCCESS * grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16: SUCCESS * grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR: SUCCESS * grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta: SUCCESS * GST_release_public_v1: SUCCESS * MET_ensemble_verification: SUCCESS * MET_verification: SUCCESS * community_ensemble_008mems: SUCCESS * community_ensemble_2mems: SUCCESS * custom_ESGgrid: SUCCESS * custom_GFDLgrid: SUCCESS * custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE: SUCCESS * custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_TRUE: SUCCESS * deactivate_tasks: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019101818: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022518: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022600: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2021010100: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019101818: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022518: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022600: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021010100: SUCCESS * get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2021062000: SUCCESS * get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS: SUCCESS * get_from_HPSS_ics_HRRR_lbcs_RAP: SUCCESS * get_from_HPSS_ics_RAP_lbcs_RAP: SUCCESS * get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio: SUCCESS * inline_post: SUCCESS * nco_ensemble: SUCCESS * nco_inline_post: SUCCESS * pregen_grid_orog_sfc_climo: SUCCESS * specify_DOT_OR_USCORE: SUCCESS * specify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE: SUCCESS * specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS: SUCCESS * specify_RESTART_INTERVAL: SUCCESS * specify_template_filenames: SUCCESS * subhourly_post: FAILURE * subhourly_post_ensemble_2mems: FAILURE ``` ## DOCUMENTATION: Documentation is updated via PR #[278](ufs-community#278) in the ``ufs-srweather-app`` repo.
ufs-community#782) ## DESCRIPTION OF CHANGES: This PR fixes a bug in the way the number of time steps (`num_time_steps` in `get_WE2Etest_names_subdirs_descs.sh`) is calculated for each WE2E test when creating the CSV file containing information about the available WE2E tests. ## TESTS CONDUCTED: Reran the `run_WE2E_tests.sh` script (which in turn calls `get_WE2Etest_names_subdirs_descs.sh`), and a CSV file containing the adjusted numbers was successfully created. ## DOCUMENTATION: None needed.
## DESCRIPTION OF CHANGES: PR #[763](ufs-community/regional_workflow#763) introduced the experiment variable `POST_OUTPUT_DOMAIN_NAME` and used it to modify the names of the UPP output files. This PR adds `POST_OUTPUT_DOMAIN_NAME` as an argument to the python plotting scripts to enable those scripts to form the proper UPP output file names to read in. It also modifies the batch scripts (`qsub_job.sh`, `qsub_job_diff.sh`, `sq_job.sh`, and `sq_job_diff.sh`) to add the new argument to the calls to the python plotting scripts that are called. ## TESTS CONDUCTED: Both python plotting scripts `plot_allvars.py` and `plot_allvars_diff.py` were run on Hera on preexisting UPP output files and completed successfully. The batch scripts `sq_job.sh` and `sq_job_diff.sh` were also run successfully on Hera (`qsub_job.sh` and `qsub_job_diff.sh` were not run on e.g. Cheyenne). ## DEPENDENCIES: PR # in `ufs-srweather-app`. ## DOCUMENTATION: Modifications to the documentation are in PR #. ## ISSUE: Partially resolves issue ufs-community#781 (still need to introduce tests in `regional_workflow` that run the plotting scripts). ## CONTRIBUTORS: @mark-a-potts pointed out the problem and first made the necessary fixes in the `NOAA-EPIC` fork of `regional_workflow`; @gsketefian introduced those changes to this PR and made edits to the comments and documentation.
…fs-community#775) * update input namelist of chgres_cube * update diag_table templates * update scripts * back to original * specify miniconda version on Jet
…odel (ufs-community#796) * update new variables introduced in uwm * update python script * update template for HRRR suite
…mmunity#779) * remove unused module loads for MET * update met version in machine files * changes to pointstat/pb2nc bufr mapping * add NA to retain bufr variables without qc markers * updated bufr variable list * change bufr var list * Change versions to MET 10.1.1 and METplus 4.1.1
…munity#803) Co-authored-by: Natalie Perlin <[email protected]>
…ity#787) ## DESCRIPTION OF CHANGES: For clarity, rename variables related to GFDLgrid-type grids: 1) Rename `GFDLgrid_RES` to `GFDLgrid_NUM_CELLS`. 2) Rename `GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES` to `GFDLgrid_USE_NUM_CELLS_IN_FILENAMES`. 3) Rename WE2E tests on GFDLgrid-type custom grids according to variable renaming above. Related improvements: * In `link_fix.sh`, remove creation of unnecessary symlink when using a GFDLgrid-type grid with `GFDLgrid_USE_NUM_CELLS_IN_FILENAMES` is set to `"FALSE"`. * In WE2E test configuration files that use custom grids, move the definition of the variable `POST_OUTPUT_DOMAIN_NAME` to the beginning of the custom grid definition because it is really the name of the custom native grid, and it's most appropriate to place that at the beginning of the grid definition section. * Reduce `WTIME_MAKE_OROG` from 1 hour to 20 minutes since on Hera, 1 hour causes the task to wait quite a while in the queue. This used to be 20 minutes but was changed to 1 hour because 20 minutes was not sufficient to complete the task on `WCOSS_DELL_P3`. Thus, reset it to 1 hour only in the machine file for `WCOSS_DELL_P3`. * Minor fixes to comments in python plotting scripts. * Edits to other comments. ## TESTS CONDUCTED: Ran the four WE2E tests that use custom grids (listed below) on Hera. All were successful. * `custom_ESGgrid` * `custom_GFDLgrid` * `custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE` (previously `custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE`) * `custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_TRUE` (previously `custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_TRUE`) ## DEPENDENCIES: PR #[310](ufs-community#310) in `ufs-srweather-app`. ## DOCUMENTATION: Updates to documentation are in PR #[310](ufs-community#310) in `ufs-srweather-app`.
This is a bug fix for running the WE2E tests on Hera. The non-existence of this path is causing failures after the path has become unreadable.
* add metplus paths * add run_vx.local for jet Co-authored-by: Edward Snyder <[email protected]>
* Remove RRFS_v1alpha suite
* Bug fix with FIELD_TABLE_FN * Modify crontab management, use config_defaults.sh. * Add status badge. * Update cheyenne crontab management. * source lmod-setup * Add main to set_predef_grid * Bug fix in predef_grid * Don't import dead params. * Fix bug in resetting VERBOSE * Minor fix in INI config. * Construct var_defns components from dictionary. * Allow also lower case variables to be exported. * Updates to python workflow due to PR ufs-community#776 * Use python versions of link_fix and set_FV3_sfc in job script. * Use python versions of create_diag/model. * Some fixes addressing Christina's suggestions. * Delete shell workflow * Append pid to temp files. * Update scripts to work with the latest hashes of UFS_UTILS and UPP (ufs-community#775) * update input namelist of chgres_cube * update diag_table templates * update scripts * back to original * specify miniconda version on Jet * Remove -S option from link_fix call. * Fixes due to merge * Cosmoetic changes. Co-authored-by: Chan-Hoo.Jeon-NOAA <[email protected]>
* remove wcoss_dell_p3 * remove block for tide and gyre
…mmunity#814) * activate b file on hpss for >40h * add a new we2e test for fcst_len_hrs>40 * reduce fcst time for we2e
…or stochastic physics. (ufs-community#819) * Add path_to_defns argument to set_FV3nml_ens_stoch_seeds. * Keep hour in directory name of ensemble members. * Fix SPP/T flag * Fix indentation of error/info message containing settings * Bugfix for DO_SHUM/SKEB which came from shell workflow generation * Fix nam_spp/p/erts confusion. * Make create_diag_table info/error message more informative. * Minor fix in info message. * Add a WE2E test case for testing stochastic physics. * Modify stoch phys test case.
Adds functionality to ush/retrieve_data.py that supports ingesting global ensemble members in the ways that have been used for RRFS. It also adds GEFS and GDAS known data locations to ush/templates/data_locations.yml. This means that it will pull data from AWS and/or HPSS for: - GEFS ICS and LBCS grib2 a & b files (it does not merge them, as that is outside the scope of this tool's functionality). - GDAS ICS and LBCS netcdf files. For ICS, it needs atm and sfc files. For LBCS, it only needs atm files. The tool is set up to pull all members as a single job, or it can be configured to do a subset of members if the user prefers to pull subsets of members. Just like for other datasets, the ICS and LBCS should be handled in separate calls to the tool, and the tool only handles a single start time since that is consistent with the behavior of the workflow. Additional functionality now includes the ability to template the output directory to include the ensemble member so each member can be stored in its own separate directory when the workflow runs. I have added functional tests to pull data from AWS and HPSS, and have hooked the AWS tests up to GitHub Actions to run when the relevant files have changed. No user intervention is needed to have those tests run -- they should run automatically upon opening a PR.
* Mods to METplus conf files: TCDC specifications, correction to level specificatons in point-stat mean and prob files, and added functionality to make METplus output dirs in ex scripts. * Updated comments in MET ex-scripts for creating output directories. * Fixed minor formatting issue in exregional_run_gridstatvx.sh
* add paths to recent MET/METplus installations on Gaea * change data staging directory to ncep_shared
- Replace all instances of "homerrfs" with "SR_WX_APP_TOP_DIR", and update location appropriately - Remove references to "regional_workflow" and update paths accordingly
mkavulich
added a commit
that referenced
this pull request
Sep 9, 2022
* Fix to post flat file. * Create MET and METplus config files under ush/templates/parm * Added script to pull and reorg ccpa data. Added a script to run gridstat with METplus. Updated MET and METplus config files. * Added new jjob for running grid-stat vx. Updated setup.sh to include grid-stat vx. Updated run_gridstatvx script. * Fixed typo on script name from ksh to sh * Moved some hard coded items out from the script to the XML * Updates to get METplus to run with fewer hard-coded paths. * Updates to add grid-stat task to XML generation. * Bug fixes for adding grid-stat to XML generation * Updates to remove hard-coded paths in config files * Change log dir to put master_metplus log file with other logs under log/, rather than default logs/. * Updates to generate xml without hard-coded paths for MET * Add hera gridstat module file * Add METplus point-stat task for both sfc and upper air * Small tweaks to remove hard coded paths and add some flexibility * Updates for adding point-stat into auto-generated xml * Add in function to set point-stat task to FALSE * Final tweaks to get it to generate the xml correctly * Minor updates to run ensure 0,6,12,18 * Tweaks to var list for Point-Stat * Add METplus settings to config_defaults * Move quote for end of settings and fix extra comment. * Fix typos to populate templates correctly * Updated to include SCRIPTSDIR and other MET specific settings along with updates to FHR syntax * Update module loads on hera * Fixed comment for BOTH_VARn_THRESH to avoid syntax issues * Added files to run grid_stat for a variety of accumulation intervals, including 3, 6, and 24h * Added module load hpss * Remove module load informatino from these scripts * Updated the method of turning on/off vx tasks using jinja template if statement * Remove commented out lines of code. Fixed typo. Removed gen_wflow.out file. * Updated pull scripts to have file names dependent on date to pull from HPSS. Updated to export a few more local variables that METplus conf needed in scripts. Updated workflow to use service queue (for now) to for 1h grid_stat and point_stat run and default for 3+h accumulation grid_stat runs) * moved common_hera.conf to common.conf - no platform specific information included that needs to be handled. * Remove common_hera.conf * Add scripts to pull and process MRMS data from NOAA HPSS * Updates for REFC vx tasks * updates to obs pull scripts * Update for adding in reflectivity verification using MRMS analyses and updating name of model output to RRFS rather than HRRR * Updates to account for CCPA issues on HPSS - day off for 00-05 UTC directories * Verification mods to feature/add metplus (#1) * Remove unused/outdated code (ufs-community#313) ## DESCRIPTION OF CHANGES: * In setup.sh and generate_FV3LAM_wflow.sh, remove temporary codes that fix bugs in the FV3_GFS_2017_gfdlmp_regional suite definition file because those bugs have been fixed (in the ufs-weather-model repo). * In setup.sh, remove block of code that is no longer necessary because chgres_cube can now initialize from external model data with either 4 or 9 soil levels, and run with LSMs of either 4 or 9 soil levels. * Remove modifications to LD_LIBRARY_PATH in exregional_run_fcst.sh. * For the make_ics and make_lbcs tasks, move the setting of APRUN and other machine-specific actions from the J-job to the ex-script in order to be consistent with the other workflow tasks. * Fix indentation and edit comments. * Remove unused file load_fv3gfs_modules.sh. ## TESTS CONDUCTED: Ran two WE2E tests on hera, new_ESGgrid and new_GFDLgrid: * new_ESGgrid uses the FV3_GFS_2017_gfdlmp_regional suite. The test was successful. * new_GFDLgrid uses the FV3_GFS_2017_gfdlmp suite. The test was successful. ## ISSUE (optional): This resolves issue ufs-community#198. * Add and call a function that checks for use of Thompson microphysics parameterization in the SDF and if so, adjusts certain workflow arrays to contain the names and other associated values of the fixed files needed by this parameterization so that those files are automatically copied and/or linked to. (ufs-community#319) ## DESCRIPTION OF CHANGES: Add and call a function that checks for use of Thompson microphysics parameterization in the suite definition file (SDF). If not, do nothing. If so, add to the appropriate workflow arrays the names and other associated values of the fixed files needed by this parameterization so that they are automatically copied and/or linked to instead of being regenerated from scratch in the run_fcst task. ## TESTS CONDUCTED: On hera, ran two WE2E tests, one in NCO mode (nco_RRFS_CONUS_25km_HRRRX_RAPX) and the other in community mode (suite_FV3_GSD_v0). These use suites FV3_GSD_SAR and FV3_GSD_v0, respectively, and both of these call Thompson microphysics. Both succeeded. ## ISSUE (optional): This PR resolves issue ufs-community#297. * RRFS_v1beta SDF changes after reverting from GSL to GFS GWD suite (ufs-community#322) (ufs-community#327) ## DESCRIPTION OF CHANGES: Removed checks on the RRFS_v1beta SDF implemented for use with the GSL GWD suite (now uses the GFS GWD suite). No longer copies staged orography files necessary for the GSL GWD suite. ## TESTS CONDUCTED: Runs to completion on Hera. End-to-end runs DOT_OR_USCORE and suite_FV3_RRFS_v1beta succeeded on Cheyenne. Co-authored-by: JeffBeck-NOAA <[email protected]> * Update FV3.input.nml for fhzero = 1.0 * Updated conf files for file name conventions. * Updated MET scripts and MRMS pull scripts. * Adjust RRFS_CONUS_... grids (ufs-community#294) ## DESCRIPTION OF CHANGES: * Adjust RRFS_CONUS_25km, RRFS_CONUS_13km, and RRFS_CONUS_3km grid parameters so that: * All grids, including their 4-cell-wide halos, lie completely within the HRRRX domain. * All grids have dimensions nx and ny that factor "nicely", i.e. they don't have factors greather than 7. * The write-component grids corresponding to these three native grids cover as much of the native grids as possible without going outside of the native grid boundaries. The updated NCL scripts (see below) were used to generate the write-component grid parameters. * For the RRFS_CONUS_13km grid, reduce the time step (DT_ATMOS) from 180sec to 45sec. This is necessary to get a successful forecast with the GSD_SAR suite, and thus likely also the RRFS_v1beta suite. * Modify WE2E testing system as follows: * Add new tests with the RRFS_CONUS_25km, RRFS_CONUS_13km, and RRFS_CONUS_3km grids that use the GFS_v15p2 and RRFS_v1beta suites (which are now the ones officially supported in the first release of the short-range weather app) instead of the GFS_v16beta and GSD_SAR suites, respectively. * For clarity, rename the test configuration files that use the GFS_v16beta and GSD_SAR suites so they include the suite name. * Update list of WE2E tests (baselines_list.txt). * Update the NCL plotting scripts to be able to plot grids with the latest version of the workflow. ## TESTS CONDUCTED: On hera, ran tests with all three grids with the GFS_v15p2 and RRFS_v1beta suites (a total of 6 tests). All were successful. * Remove redundant model_configure.${CCPP_PHYS_SUITE} template files; use Jinja2 to create model_configure (ufs-community#321) ## DESCRIPTION OF CHANGES: * Remove model_configure template files whose names depend on the physics suite, i.e. files with names of the form model_configure.${CCPP_PHYS_SUITE}. Only a single template file is needed because the contents of the model_configure file are not suite dependent. This leaves just one template file (named model_configure). * Change the function create_model_configure_file.sh and the template file model_configure so they use jinja2 instead of sed to replace placeholder values. * Absorb the contents of the write-component template files wrtcmp_lambert_conformal, wrtcmp_regional_latlon, and wrtcmp_rotated_latlon into the new jinja2-compliant model_configure file. We can do this because Jinja2 allows use of if-statements in the template file. * In the new model_configure jinja2 template file, include comments to explain the various write-component parameters. ## TESTS CONDUCTED: On Hera, ran the two WE2E tests new_ESGgrid and new_GFDLgrid. The first uses a "lambert_conformal" type of write-component grid, and the second uses a "rotated_latlon" type of write-component grid. (The write-component also allows "regional_latlon" type grids, which is just the usual earth-relative latlon coordinate system, but we do not have any cases that use that.) Both tests succeeded. ## ISSUE (optional): This PR resolves issue ufs-community#281. * Add Thompson ice- and water-friendly aerosol climo file support (ufs-community#332) * Add if statement in set_thompson_mp_fix_files.sh to source Thompson climo file when using a combination of a Thompson-based SDF and non-RAP/HRRR external model data * Modify if statement based on external models for Thompson climo file * Remove workflow variable EMC_GRID_NAME (ufs-community#333) ## DESCRIPTION OF CHANGES: * Remove the workflow variable EMC_GRID_NAME. Henceforth, PREDEF_GRID_NAME is the only variable that can be used to set the name of the predefined grid to use. * Make appropriate change of variable name (EMC_GRID_NAME --> PREDEF_GRID_NAME) in the WE2E test configuration files. * Change anywhere the "conus" and "conus_c96" grids are specified to "EMC_CONUS_3km" and "EMC_CONUS_coarse", respectively. * Rename WE2E test configuration files with names containing the strings "conus" and "conus_c96" by replacing these strings with "EMC_CONUS_3km" and "EMC_CONUS_coarse", respectively. * Update the list of WE2E test names (tests/baselines_list.txt). * Bug fixes not directly related to grids: * In config.nco.sh, remove settings of QUEUE_DEFAULT, QUEUE_HPSS, and QUEUE_FCST since these are now set automatically (due to another PR). * In the template file FV3LAM_wflow.xml, add the ensemble member name after RUN_FCST_TN in the dependency of the run_post metatask. ## TESTS CONDUCTED: Since this change only affects runs in NCO mode, the following NCO-mode WE2E tests were rerun on hera, all successfully: ``` nco_EMC_CONUS_3km SUCCESS nco_EMC_CONUS_coarse SUCCESS nco_EMC_CONUS_coarse__suite_FV3_GFS_2017_gfdlmp SUCCESS nco_RRFS_CONUS_25km_HRRRX_RAPX SUCCESS nco_RRFS_CONUS_3km_FV3GFS_FV3GFS SUCCESS nco_RRFS_CONUS_3km_HRRRX_RAPX SUCCESS nco_ensemble SUCCESS ``` * Port workflow to Orion (ufs-community#309) ## DESCRIPTION OF CHANGES: * Add stanzas for Orion where necessary. * Add new module files for Orion. * On Orion, both the slurm partition and the slurm QOS need to be specified in the rocoto XML in order to be able to have wall times longer than 30 mins (the partition needs to be specified because it is by default "debug", which has a limit of 30 mins). Thus, introduce modifications to more easily specify slurm partitions: * Remove the workflow variables QUEUE_DEFAULT_TAG, QUEUE_HPSS_TAG, and QUEUE_FCST_TAG that are currently used to determine whether QUEUE_DEFAULT, QUEUE_HPSS, and QUEUE_FCST specify the names of queue/QOS's or slurm partitions. * Add the workflow variables PARTITION_DEFAULT_TAG, PARTITION_HPSS_TAG, and PARTITION_FCST_TAG. These will be used to specify slurm partitions only, and the variables QUEUE_DEFAULT, QUEUE_HPSS, and QUEUE_FCST will be used to specify queues/QOS's only. IMPORTANT NOTE: On Orion, in order to load the regional_workflow environment needed for generating an experiment, the user must first issue the following commands: ``` module use -a /apps/contrib/miniconda3-noaa-gsl/modulefiles module load miniconda3 conda activate regional_workflow ``` ## TESTS CONDUCTED: Ran 11 WE2E tests on Orion, Hera, and Cheyenne. Results on Orion: ``` community_ensemble_2mems SUCCESS DOT_OR_USCORE SUCCESS grid_GSD_HRRR_AK_50km FAILURE - In the run_fcst task. * Error message: !!! (1) Error in subr radiation_aerosols: unrealistic surface pressure = 1 NaN new_ESGgrid SUCCESS new_GFDLgrid SUCCESS regional_001 SUCCESS regional_002 SUCCESS suite_FV3_GFS_v15p2 SUCCESS suite_FV3_GFS_v16beta SUCCESS suite_FV3_GSD_SAR SUCCESS suite_FV3_GSD_v0 SUCCESS ``` Results on Hera: ``` community_ensemble_2mems SUCCESS DOT_OR_USCORE SUCCESS grid_GSD_HRRR_AK_50km SUCCESS new_ESGgrid SUCCESS new_GFDLgrid SUCCESS regional_001 SUCCESS regional_002 SUCCESS suite_FV3_GFS_v15p2 SUCCESS suite_FV3_GFS_v16beta SUCCESS suite_FV3_GSD_SAR SUCCESS suite_FV3_GSD_v0 SUCCESS ``` Results on Cheyenne: ``` community_ensemble_2mems SUCCESS DOT_OR_USCORE SUCCESS grid_GSD_HRRR_AK_50km FAILURE - In run_fcst task. * Error message: !!! (1) Error in subr radiation_aerosols: unrealistic surface pressure = 1 NaN new_ESGgrid SUCCESS new_GFDLgrid SUCCESS regional_001 SUCCESS regional_002 SUCCESS suite_FV3_GFS_v15p2 SUCCESS suite_FV3_GFS_v16beta SUCCESS suite_FV3_GSD_SAR SUCCESS suite_FV3_GSD_v0 SUCCESS ``` All succeed except GSD_HRRR_AK_50km on Orion and Cheyenne. It is not clear why grid_GSD_HRRR_AK_50km fails on Orion and Cheyenne but not Hera. Seems to point to a bug in the forecast model. These two failures are not so important since this grid will soon be deprecated. Also tested successfully on Jet by @JeffBeck-NOAA and on Odin and Stampede by @ywangwof. ## ISSUE: This resolves Issue ufs-community#152. ## CONTRIBUTORS: @JeffBeck-NOAA @ywangwof @christinaholtNOAA * Removed comments from exregional_get_mrms_files.sh and removed fhzero from FV3.input.yml * Update FV3.input.nml for fhzero = 1.0 * Updated conf files for file name conventions. * Updated MET scripts and MRMS pull scripts. * Removed comments from exregional_get_mrms_files.sh and removed fhzero from FV3.input.yml Co-authored-by: gsketefian <[email protected]> Co-authored-by: Michael Kavulich <[email protected]> Co-authored-by: JeffBeck-NOAA <[email protected]> Co-authored-by: Jamie Wolff <[email protected]> * Change cov_thresh for REFL to be a true max in nbrhood as SPC does. * Job script for get_obs_ccpa * Jobs script for get_obs_mrms * Jobs script for get_obs_ndas * Added external variables necessary to get_ccpa script * Updated workflow template with separate get obs tasks * Separated pull scripts from run scripts * Added necessary defaults/values for defining pull tasks * Added module files, default config.sh options, and changed dependencies for vx tasks * Changed name of new workflow to FV3LAM_wflow.xml * Added task get_obs_tn, removed config.sh, updated config_defaults and config.community.sh * Adjusted the community and default config files based on comments * Updated FV3LAM workflow * Fixed discrepancies in config.community.sh * Fixed discrepancies in config_defaults.sh * Fixed discrepancies in config_defaults.sh round 2 * Fixed discrepancies in config_defaults.sh round 3 * Fixed discrepancies in config_defaults.sh round 4 * Fixed discrepancies in config.community.sh round 2 * Fixed discrepancies in config.community.sh round 3 * Fixed discrepancies in generate_FV3LAM_wflow.sh * Fixed discrepancies in generate_FV3LAM_wflow.sh round 2 * Fixed discrepancies in generate_FV3LAM_wflow.sh round 3 * Updated FV3LAM_wflow template * Separated Pull Data Scripts from Run Vx Scripts: Feature/add_metplus (ufs-community#2) * Job script for get_obs_ccpa * Jobs script for get_obs_mrms * Jobs script for get_obs_ndas * Added external variables necessary to get_ccpa script * Updated workflow template with separate get obs tasks * Separated pull scripts from run scripts * Added necessary defaults/values for defining pull tasks * Added module files, default config.sh options, and changed dependencies for vx tasks * Changed name of new workflow to FV3LAM_wflow.xml * Added task get_obs_tn, removed config.sh, updated config_defaults and config.community.sh * Adjusted the community and default config files based on comments * Updated FV3LAM workflow * Fixed discrepancies in config.community.sh * Fixed discrepancies in config_defaults.sh * Fixed discrepancies in config_defaults.sh round 2 * Fixed discrepancies in config_defaults.sh round 3 * Fixed discrepancies in config_defaults.sh round 4 * Fixed discrepancies in config.community.sh round 2 * Fixed discrepancies in config.community.sh round 3 * Fixed discrepancies in generate_FV3LAM_wflow.sh * Fixed discrepancies in generate_FV3LAM_wflow.sh round 2 * Fixed discrepancies in generate_FV3LAM_wflow.sh round 3 * Updated FV3LAM_wflow template * Fixed the dependencies of the vx tasks * Fixed Vx Task Dependencies in Workflow: Feature/add metplus (ufs-community#3) * Job script for get_obs_ccpa * Jobs script for get_obs_mrms * Jobs script for get_obs_ndas * Added external variables necessary to get_ccpa script * Updated workflow template with separate get obs tasks * Separated pull scripts from run scripts * Added necessary defaults/values for defining pull tasks * Added module files, default config.sh options, and changed dependencies for vx tasks * Changed name of new workflow to FV3LAM_wflow.xml * Added task get_obs_tn, removed config.sh, updated config_defaults and config.community.sh * Adjusted the community and default config files based on comments * Updated FV3LAM workflow * Fixed discrepancies in config.community.sh * Fixed discrepancies in config_defaults.sh * Fixed discrepancies in config_defaults.sh round 2 * Fixed discrepancies in config_defaults.sh round 3 * Fixed discrepancies in config_defaults.sh round 4 * Fixed discrepancies in config.community.sh round 2 * Fixed discrepancies in config.community.sh round 3 * Fixed discrepancies in generate_FV3LAM_wflow.sh * Fixed discrepancies in generate_FV3LAM_wflow.sh round 2 * Fixed discrepancies in generate_FV3LAM_wflow.sh round 3 * Updated FV3LAM_wflow template * Fixed the dependencies of the vx tasks * Manual merge with develop that didn't seem to work before. Trying to get feature branch updated so it will run again! * Add local module files * Add environment variable for SCRIPTSDIR * Remove echo statement * Remove old module files * Update to config_default for walltime for ndas pull. Update to metplus parm for obs file template. Update to FV3LAM xml to not include 00 hour for verification * Update template to remove full path * Verification channges for obs. (ufs-community#4) * Verification channges for obs. * Update config_defaults.sh for vx description * Update config_defaults.sh to remove extraneous MET info. Co-authored-by: Michelle Harrold <[email protected]> * Initial METplus .confs and MET config files for EnsembleStat APCP * J-Job script for running ensemble stat * Exregional script for ensemble-stat * Added EnsembleStat.conf for A6 and A24. Added PCPCombine to A3, A6, and A24. * Added EnsembleStatConfig files for 6 and 24h * Copy of workflow template with precipitation ensemble tasks added. Will become main template when testing is complete * Added export statement for number of ensemble members * Added necessary task definitions in ush * Updated workflow to included ENTITY definitions for ensstat * Fixed typo * Added ens vx configs * Pull in updates from develop that were not merging properly. Small change to config.community to turn off vx tasks by default. * Added/mod files for point ens vx. * Updated metplus conf files for ens point vx * Did manual merge of these files because it was not handled properly automatically * Adding additional variables to METplus for regional workflow (ufs-community#5) * Changes made based on meeting with Michelle and Jamie * Updating fork * Cleanup after merge * Added additional ens vx * Ensemble point vx mods * Additional updates for ens and det vx * ensgrid_mean and ensgrid_prob .conf files for APCP * Updates for ensemble vx. * Added mean and prob point-stat configs * Updates to ensgrid_vx * Updates for mean/prob vx. * Updates to FV3LAM_wflow.xml * Deterministic and ensembel vx updates. * Ensgrid mean * Update setup.sh * Changed workflow template title * Updates to deterministic and ensemble verification * Created EnsembleStat METplus conf and MET config files for REFC * Added reflectivity mean and prob METplus and MET config files. Updated APCP mean and prob METplus and MET config files. * Added all J-job scripts, exregional scripts, and necessary definitons for workflow generation for all ensgrid_mean and ensgrid_prob tasks * Updates to workflow to add ensgrid_vx * Changes I made to account for runtime errors. * Made changes to directory structures * Made changes to directory structures and variables * Changed log files and stage dir. * Changes for grid- and point-vx. * Updated METplus ensemble precip conf files. * Mods for ensemble and deterministic vx. * Change to GridStatConfig_REFC_mean * Updated EnsembleStat_REFC.conf * Updated to METv10.0.0 * Updated conf files for paths. * Updated FV3LAM_wflow.xml template. * Mods for vx dependencies * Updated for censor thresh in METplus conf files; changes to FV3LAM_wflow.xml after sync with develop. * Updated exregional_run_fcst.sh generate_FV3LAM_wflow.sh to address merge with develop. * Mods for ensemble precip vx, handling padded/non-padded ensemble member names, fixes for python environment for obs pull. * Changes to RETOP (units) and REFC (naming and level) verification. * Fix OUTPUT_BASE for deterministic vx. * Changes to some verification ex-scripts for syntax and path fixes. Included start end dates of incorrect 01-h CCPA data. Removed some extra lines in FV3LAM_wflow.xml template. * Changed comp. ref. variable name in GridStat_REFC_prob.conf * Changed comp. ref. level in GridStat_REFC_prob.conf * Updated logic for number padding in the directory name when running in ensemble mode. * Added MET ensemble vx WE2E test. * Modified location of obs to live outside cycle dir, allowing for obs to be shared across cycles. * Mods to address comments on PR575. * Updated ensemble METPlus conf files for changes to post output name. * Addessed comments in PR and mods for 10-m WIND. * Addressing final comments in PR. Co-authored-by: Jamie Wolff <[email protected]> Co-authored-by: gsketefian <[email protected]> Co-authored-by: Michael Kavulich <[email protected]> Co-authored-by: JeffBeck-NOAA <[email protected]> Co-authored-by: lindsayrblank <[email protected]> Co-authored-by: Michelle Harrold <[email protected]> Co-authored-by: PerryShafran-NOAA <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a demonstration PR for informational purposes regarding the regional_workflow --> ufs-srweather-app merger. The
developbranch of this fork has had the merge manually applied as described in the procedure document. This branch (updates_after_repo_merge)shows what the necessary post-merge changes will look like.Click the "Files changed" tab to see the changes in this example; the final changes may be slightly different depending on PRs that are merged into ufs-srweather-app prior to the merger event.