-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix bug for nco we2e tests on Orion; re-organize we2e input data and nco we2e tests #713
Conversation
… into bugfix/orion_we2e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have access to Orion to test this but visual it looks good to me.
@chan-hoo Can you put a bit more detail about these changes to the WE2E tests? I understand the test deletions for reducing the number of NCO tests, but in others you have changed physics options and boundary condition settings and I'm interested in why those were changed. |
@mkavulich, we'd like to test three domains [RRFS_CONUS_25km (for test), RRFS_CONUS_13km (for RRFS-CMAQ), RRFS_CONUS_3km (for parallel)], and three ccpp suites [FV3_GFS_v15_thompson_mynn_lam3km (major interest at EMC), FV3_GFS_v15p2 and _v16 (officially supported suites in the UFS SRW App as well as in RRFS-CMAQ)] in the 'nco' mode. Since the '*_thompson_mynn_lam3km' should be used with the 3km resolution domain, I matched it with the RRFS_CONUS_3km domain. In addition, the 'inline post' option is widely used at EMC. As you know, the difference between the 'community' and 'nco' modes is the structure of the input and output data. I think the above tests are good enough for the 'nco' mode at this moment. For IC/BCs, I've set the same date, 20190615, for all the tests for convenience. In the staged data set, the specific date 20190615 provides grib2 files every 6 hours. This is the reason why I changed 'LBC_SPEC_INTVL_HRS' from '3' to '6'. Finally, I'd like to test one case for two BC intervals (12 hours). I set this to the lowest resolution domain (25km) to save its computing time. |
|
||
DATE_FIRST_CYCL="20190615" | ||
DATE_LAST_CYCL="20190615" | ||
CYCL_HRS=( "00" ) | ||
|
||
FCST_LEN_HRS="6" | ||
LBC_SPEC_INTVL_HRS="3" | ||
FCST_LEN_HRS="12" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chan-hoo Actually, the data is available every 3 hours in the original directory on Hera at /scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/staged_extrn_mdl_files/FV3GFS/grib2/2019061500
. I think @mkavulich copied it over to the official location at /scratch2/BMC/det/UFS_SRW_app/develop/model_data/FV3GFS/grib2/2019061500
. Not sure why only 6-hour output was copied.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gerard and I talked offline, but for the benefit of everyone else: We are trying to minimize the size of static data being maintained, so I only copied over data that was being used in tests actively. The test as it is on Hera right now does not use static data, it pulls it from HPSS.
.../grids_extrn_mdls_suites_nco/config.nco_grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
Outdated
Show resolved
Hide resolved
tests/WE2E/test_configs/grids_extrn_mdls_suites_nco/config.nco_inline_post.sh
Outdated
Show resolved
Hide resolved
@@ -1,4 +1,4 @@ | |||
# | |||
# |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chan-hoo The purpose of this test is to check whether the inline post works in NCO mode, so it is a test of a workflow feature (or a combination of two features being enabled, inline post and NCO mode). That's why it's in the wflow_features
directory. And that's why it's a short forecast (3 hours), since if it works for the first 4 outputs, it's probably going to work for all hours. The directory grids_extrn_mdls_suites_nco
is more for scientific tests, i.e. to check that the important combinations of grids, external models, and suites work.
If you want to reduce the number of tests, one thing you can do is make one of the other tests in grids_extrn_mdls_suites_nco
use WRITE_POST="TRUE"
and replace this file with a symlink that points to that test. The symlink can have the same name as this file, i.e.
wflow_features/config.nco_inline_post.sh --> ../grids_extrn_mdls_suites_nco/config.whichever_test_you_want_to_enable_inline_post_in.sh
The testing script (run_WE2E_tests.sh
) knows what to do with symlinks like this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gsketefian, agreed. I'll remove this test and put the inline post option into one of the existing test cases.
@mkavulich, @gsketefian, Can you add the missing 3 hour data to the central location on Hera (/scratch2/BMC/det/UFS_SRW_app/develop/model_data/FV3GFS/grib2/2019061500)? I don't have permission for this. |
@gsketefian, I copied the missing data from your original directory on Hera to Orion and WCOSS_dell_p3. The new nco tests worked well on Orion and WCOSS_dell_p3. They should work if the missing data is copied to the designated directory on Hera. If you have any other comments, let me know. |
@chan-hoo the 3-hour data is now available on Hera at /scratch2/BMC/det/UFS_SRW_app/develop/model_data/FV3GFS/grib2/2019061500 |
@@ -1,28 +0,0 @@ | |||
# |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Which under grids_extrn_mdls_suites_nco
is now the test for inline post?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh sorry, I just saw it's config.nco_grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
. Can you add a symlink here (named config.nco_inline_post.sh
) to that file so we know where that is?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
PREDEF_GRID_NAME="CONUS_25km_GFDLgrid" | ||
CCPP_PHYS_SUITE="FV3_GFS_2017_gfdlmp_regional" | ||
PREDEF_GRID_NAME="RRFS_CONUS_25km" | ||
CCPP_PHYS_SUITE="FV3_GFS_v15p2" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chan-hoo Just curious why you're using 15p2 here since you removed the other 15p2 tests above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gsketefian, there was no special intention. v15p2 was officially supported in release v1 and it will be used for RRFS-CMAQ along with v16. I thought v15p2 would be better than 2017_gfdlmp_regional. As you said, v15p2 was not tested in other nco we2e tests. So I applied it to this test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, sounds good.
@@ -17,9 +17,11 @@ EXTRN_MDL_NAME_ICS="HRRR" | |||
EXTRN_MDL_NAME_LBCS="RAP" | |||
USE_USER_STAGED_EXTRN_FILES="TRUE" | |||
|
|||
DATE_FIRST_CYCL="20200208" | |||
DATE_LAST_CYCL="20200208" | |||
DATE_FIRST_CYCL="20200810" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chan-hoo How are you deciding on new dates?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gsketefian, I matched the date with the same test in the 'community' mode (config.grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_HRRR.sh).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, makes sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good.
* Add missing user-defined stochastic physics options; fix stochastic physics seed generation script (ufs-community#704) ## DESCRIPTION OF CHANGES: Add missing user-defined options for tendency-based stochastic physics and fix the ensemble-based seed generation script to work regardless of whether stochastic physics is turned on or not. ## TESTS CONDUCTED: Tested on Hera using the following WE2E configurations with and without stochastic physics: config.grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh config.community_ensemble_2mems.sh ## ISSUE (optional): [Issue ufs-community#702](ufs-community#702) ## CONTRIBUTORS (optional): Thanks to @mkavulich and @chan-hoo for finding this problem. * Add namelist option for netCDF4 when running with the 3-km NA domain; update NAM HPSS settings and WE2E tests (ufs-community#707) * Change to netcdf4 when using the NA 3-km domain * Update HPSS paths for NAM data * Update NAM HPSS locations and dates for WE2E tests. * Remove lines from merge. * Tweaks to allow compiler and build_env_fn to be specified in the run_WE2E_test.sh script (ufs-community#711) * Changed 20200304 to 20200303 in ush/mrms_pull_topofhour.py (ufs-community#712) * Remove unused rocoto directory in ush (ufs-community#720) * Fix bug for nco we2e tests on Orion; re-organize we2e input data and nco we2e tests (ufs-community#713) * Update machine script for orion * Update machine script for wcoss_dell_p3 * Update we2e run script for wcoss and orion * Reorganize nco we2e tests * remove machine based logic * Add symlink for nco inline post test * Added stand-alone verification scripts (feature/issue_683_standaloneVX) (ufs-community#726) * Grid-stat and point-stat run scripts. * Stand-alone scripts for verification. * Added comments to gridvx scripts. * Added qsub_job.sh and added comments to provide context on running Vx. * remove machine base logic (ufs-community#727) * Allow user-defined file names for input template files (ufs-community#717) * Allow multiple template names * parameterize file_TMPL_FN and add a we2e test * Increase maxtries_task for make_grid/orog/sfc_climo * Modify file name and description * Changes to RRFS 3- and 13-km domains, setup.sh script bug fixes, make_ics task modification, and tweaks to stochastic physics namelist settings (ufs-community#721) * Modify RRFS North America 3- and 13-km domain configuration and WE2E test. * Change sotyp_from_climo to "true" based on operational RAP grib2 files. * Update for changes to stochastic physics namelist options. * Check for DO_ENSEMBLE="TRUE" when running ensemble verification and turn of VX when running in NCO mode. * Revert to 3-km domain. * Remove commented-out GFDL grid for the RRFS_NA_13km domain * Add RRFS_NA_13km WE2E test * Changes to comments. * Adding 25 km tests to Jet/Hera suites. (ufs-community#718) * Add a small 3km predefined grid over Indianapolis for testing (ufs-community#725) * Add 3km grid over Indianapolis. This is about 600km x 600km in extent (200 x 200 grid points). It is intended for use in the WE2E tests. * Edit comments. * Use Python tool for get_extrnl_mdl_file tasks (ufs-community#681) These changes hook in the Python-based data ingest tool, replacing the previous scripts that handled this work as part of the get_extrn_mdl_file task. No attempt was made in this PR to replace the NOMADS fetching script with the Python utility, but the NOMADS data location has been added to the data_locations.yml file. The functionality to write the data summary file has also been added to the Python tool to match the capabilities of the existing workflow tools. * Increase size of RRFS CONUS grid (ufs-community#724) Co-authored-by: Benjamin.Blake EMC <[email protected]> Co-authored-by: Benjamin.Blake EMC <[email protected]> Co-authored-by: Benjamin.Blake EMC <[email protected]> Co-authored-by: chan-hoo <[email protected]> * add include-style quality mark options in metplus confs (ufs-community#738) * Add Gaea as a supported platform for the regional_workflow (ufs-community#734) * Updates to port regional workflow to gaea * Temp change with -v as batch option * new fixes for gaea/slurm * Updated time for make lbcs * added TEST data directory path * Update gaea.sh * fixes for PR * Add more parameters to CSV file containing WE2E test info (ufs-community#740) ## DESCRIPTION OF CHANGES: The script/function `get_WE2Etest_names_subdirs_descs.sh` (which is called from `run_WE2E_tests.sh` if needed) creates a CSV (Comma-Separated Value) file named `WE2E_test_info.csv` that contains information about the WE2E tests. Currently, this CSV file contains only 3 columns: the test name, the names of any alternate names for the test, and the test description. In order to have a more complete summary of the WE2E tests, this PR modifies `get_WE2Etest_names_subdirs_descs.sh` so that additional information is included in the CSV file. This additional information consists of the values of the following experiment variables for each test: ``` PREDEF_GRID_NAME CCPP_PHYS_SUITE EXTRN_MDL_NAME_ICS EXTRN_MDL_NAME_LBCS DATE_FIRST_CYCL DATE_LAST_CYCL CYCL_HRS INCR_CYCL_FREQ FCST_LEN_HRS LBC_SPEC_INTVL_HRS NUM_ENS_MEMBERS ``` In addition, the script uses this information to calculate the number of times each test calls the forecast model (e.g. if the test uses 3 different cycle dates, then the forecast model will be called 3 times; if it is an ensemble test for a single cycle, the test will call the forecast model as many times as the number of ensemble members). ## TESTS CONDUCTED: The script `run_WE2E_tests.sh` was called that in turn calls `get_WE2Etest_names_subdirs_descs.sh`. This created a new CSV file that contained the new fields (columns). The CSV file was imported into Google Sheets (using "|" as the field/column separator) and looked correct. ## DOCUMENTATION: The documentation is for the most part already within the `get_WE2Etest_names_subdirs_descs.sh`. This PR slightly modifies that documentation to update it. * Update directory structure of NCO mode (ufs-community#743) * update vertical structure of NCO mode * update sample script for nco * Fix typo on write component of new RRFS CONUS * Default CCPP physics option is FV3_GFS_v16 (ufs-community#746) * Updated the default CCPP physics option to FV3_GFS_v16 * Updated the default CCPP physics option to FV3_GFS_v16 in config_defaults.sh Co-authored-by: Natalie Perlin <[email protected]> * Adds an alternative python workflow generation path (ufs-community#698) * Workflow in python starting to work. * Use new python_utils package structure. * Some bug fixes. * Use uppercase TRUE/FALSE in var_dfns * Use config.sh by default. * Minor bug fixes. * Remove config.yaml * Update to the latest develop * Remove quotes from numbers in predef grid. * Minor bug fix. * Move validity checker to the bottom of setup * Add more unit tests. * Update with python_utils changes. * Update to latest develop additions (Need to re-run regression test) * Use set_namelist and fill_jinja_template as python functions. * Replace sed regex searches with python re. * Use python realpath. * Construct settings as dictionary before passing to fill_jinja and set_namelist * Use yaml for setting predefined grid parameters. * Use xml parser for ccpp phys suite definition file. * Remove more run_command calls. * Simplify some func argument processing. * Move different config format parsers to same file. * Use os.path.join for the sake of macosx * Remove remaining func argument processing via os.environ. * Minor bug fix in set_extrn_mdl_params.sh * Add suite defn in test_data. * Minor fixes on unittest on jet. * Simplify boolean condition checks. * Include old in renaming of old directories * Fix conflicting yaml !join tag for paths and strings. * Bug fix with setting sfcperst dict. * Imitate "readlink -m" with os.path.realpath instead of os.readlink * Don't use /tmp as that is shared by multiple users. * Bug fix with cron line, maintain quotes around TRUE/FALSE. * Update to latest develop (untested) * Bug fix with existing cron line and quotes. * Bug fix with case-sensitive MACHINE name, and empty EXPT_DIR. * Update to latest develop * More updates. * Bug fix thanks to @willmayfield! Check both starting/ending characters are brackets for shell variable to be considered an array. * Make empty EXPT_BASEDIR workable. * Update to latest develop * Update in predef grid. * Check f90nml as well. Co-authored-by: Daniel Abdi <[email protected]> * Fix typo and crontab issue on wcoss dell in workflow python scripts (ufs-community#750) * Fix typo and failure on wcoss * fix new line issue on wcoss dell * remove capture_output * Get USER from environment Co-authored-by: Daniel Abdi <[email protected]> * Add new WE2E configs (ufs-community#748) ## DESCRIPTION OF CHANGES: Added two new WE2E config files for the Sub-CONUS Indianapolis domain to support the upcoming SRW release. In addition, modified the external data used in the `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` to match more common datasets used in the WE2E testing process. ## TESTS CONDUCTED: Successfully ran the new WE2E tests (`config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh`, `config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh`) and `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` on NOAA Parallel Works AWS instance. ## DEPENDENCIES: None. ## DOCUMENTATION: No documentation changes are required. * Added a fixed WoF grid and the python tool to determine the write component parameters (ufs-community#733) * Added a fixed WoF grid and the python tool to determine the write component parameters * Update set_predef_grid_params.sh * Renamed file as recommended and removed unused lines * Modified comment Co-authored-by: JeffBeck-NOAA <[email protected]> Co-authored-by: WYH@MBP <[email protected]> * Replace env with modulefiles in scripts (ufs-community#752) * change env to mod * update we2e script * WE2E script improvements for usability (ufs-community#745) ## DESCRIPTION OF CHANGES: * Modifications to `run_WE2E_tests.sh`: * Add examples to help/usage statement * Modifications to `check_expts_status.sh`: * Add arguments list that can be processed by `process_args` * Add new optional arguments: `num_log_lines`, `verbose` * Include a help/usage message ## TESTS CONDUCTED: * Ran `run_WE2E_tests.sh --help` from the command line and got the expected help message. * Ran `check_expts_status.sh --help` from the command line and got the expected help message. * Used `run_WE2E_tests.sh` to run a set of 2 WE2E tests -- works as expected. * Used `check_expts_status` to check on the status of the 2 tests run above and got the expected status message. ## DEPENDENCIES: PR #[241](ufs-community/ufs-srweather-app#241) ## DOCUMENTATION: A lot of this PR is documentation in the scripts. There is an accompanying documentation PR #[241](ufs-community/ufs-srweather-app#241) into ufs-srweather-app. * Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval (ufs-community#744) * Bug fixes (grid size + suppress screen output from module load) (ufs-community#756) ## DESCRIPTION OF CHANGES: 1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR ufs-community#725 ). 2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community/ufs-srweather-app#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context). ## TESTS CONDUCTED: 1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does). 2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is). * Update default SPP ISEED array in config_defaults.sh to use unique values (ufs-community#759) * Modify RRFS North America 3- and 13-km domain configuration and WE2E test. * Modify default ISEED values for SPP * Fix grid in WE2E test * Update workflow python scripts (ufs-community#760) * update python scripts * Change output file name of run_post to meet NCO standards (ufs-community#758) * change output file name * change variable name * update python script * remove duplicates * add a check for empty variables * move variable to common area * clean up unnecessary comments * update scripts * remove duplicate * update python scripts * fix user-staged dir path issue in python script * Add POST_OUTPUT_DOMAIN_NAME to WE2E tests for new grids (ufs-community#763) * Add new var to we2e tests for new grids * rename we2e tests for custom grid * remove unnecessary $ * Modifications to `CODEOWNERS` file (ufs-community#757) * Add @gspetro-NOAA, @natalie-perlin, and @EdwardSnyder-NOAA to CODEOWNERS so they are notified of all PRs and can review them. * Remove duplicates in CODEOWNERS; remove users who will no longer be working with the repo. * Adding a python utility for summarizing compute. (ufs-community#769) Adds a utility that summarizes Rocoto database computational usage information. * Add github actions for python unittests. (ufs-community#747) * Add github actions for python unittests. * Include all python script in ush * Skip defining QUILTING params when it is set to False * Update py_workflow * Update unittest for set_extrn_mdl_params. * Updates from develop. Co-authored-by: Daniel Shawul <[email protected]> * Update sample script for NCO mode (ufs-community#771) * update config.nco.sh * Add comment * Feature/noaacloud (ufs-community#767) * updates for noaacloud * working version * fixes for noaacloud * added extra modules for post * removed cheyenne-specific crontab editing section (ufs-community#773) * Pin down hera miniconda3 module file version. (ufs-community#770) Pin down the version of miniconda3 on Hera, and do not append to the module path. * update staged data dir (ufs-community#774) Co-authored-by: JeffBeck-NOAA <[email protected]> Co-authored-by: Mark Potts <[email protected]> Co-authored-by: michelleharrold <[email protected]> Co-authored-by: Chan-Hoo.Jeon-NOAA <[email protected]> Co-authored-by: gsketefian <[email protected]> Co-authored-by: BenjaminBlake-NOAA <[email protected]> Co-authored-by: Benjamin.Blake EMC <[email protected]> Co-authored-by: Benjamin.Blake EMC <[email protected]> Co-authored-by: Benjamin.Blake EMC <[email protected]> Co-authored-by: chan-hoo <[email protected]> Co-authored-by: Will Mayfield <[email protected]> Co-authored-by: Natalie Perlin <[email protected]> Co-authored-by: Natalie Perlin <[email protected]> Co-authored-by: danielabdi-noaa <[email protected]> Co-authored-by: Daniel Abdi <[email protected]> Co-authored-by: Daniel Abdi <[email protected]> Co-authored-by: EdwardSnyder-NOAA <[email protected]> Co-authored-by: Yunheng Wang <[email protected]> Co-authored-by: WYH@MBP <[email protected]> Co-authored-by: Michael Kavulich <[email protected]> Co-authored-by: Daniel Shawul <[email protected]>
DESCRIPTION OF CHANGES:
Orion
andWCOSS_dell_p3
to match up with those onHera
.Orion
in the 'ush/machine' directory to make the 'nco' WE2E tests work onOrion
.TESTS CONDUCTED:
Orion
,WCOSS_dell_p3
, andHera
:WCOSS_dell_p3
andHera
: