Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[develop] Modify retrieve_data.py to check a url instead of download. #776

Merged
merged 2 commits into from
May 15, 2023
Merged

[develop] Modify retrieve_data.py to check a url instead of download. #776

merged 2 commits into from
May 15, 2023

Conversation

christinaholtNOAA
Copy link
Collaborator

DESCRIPTION OF CHANGES:

Try to speed up tests by only checking that a url exists for the files that are very large.
Apply the new flag to the nemsio unittests to avoid a lengthy download.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

TESTS CONDUCTED:

  • hera.intel
  • orion.intel
  • cheyenne.intel
  • cheyenne.gnu
  • gaea.intel
  • jet.intel
  • wcoss2.intel
  • NOAA Cloud (indicate which platform)
  • Jenkins
  • fundamental test suite
  • comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

Attempts to fix Issue #753

CHECKLIST

  • My code follows the style guidelines in the Contributor's Guide
  • I have performed a self-review of my own code using the Code Reviewer's Guide
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes do not require updates to the documentation (explain).
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

  • Work In Progress
  • bug
  • enhancement
  • documentation
  • release
  • high priority
  • run_ci
  • run_we2e_fundamental_tests
  • run_we2e_comprehensive_tests
  • Needs Cheyenne test
  • Needs Jet test
  • Needs Hera test
  • Needs Orion test
  • help wanted

CONTRIBUTORS (optional):

MichaelLueken
MichaelLueken approved these changes May 5, 2023
View reviewed changes
Copy link
Collaborator

@MichaelLueken MichaelLueken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@christinaholtNOAA Thanks for making these changes! The GitHub Action running test_retrieve_data.py is now completing in 7 minutes. Approving the PR.

@clouden90 clouden90 mentioned this pull request May 5, 2023
Closed
@MichaelLueken MichaelLueken mentioned this pull request May 8, 2023
Merged
13 tasks
mkavulich added a commit to mkavulich/ufs-srweather-app that referenced this pull request May 8, 2023
@mkavulich
Remove UFS case study skip, will go with propsal from ufs-community#776
445c003
@christinaholtNOAA
Copy link
Collaborator Author

Can this one be merged?

@MichaelLueken MichaelLueken merged commit c60a738 into ufs-community:develop May 15, 2023
michelleharrold pushed a commit to michelleharrold/ufs-srweather-app that referenced this pull request Jun 7, 2023
@gsketefian
Add columns for relative test and time step to CSV file containing WE…
37c2dcb 
…2E test info (ufs-community#776)

## DESCRIPTION OF CHANGES: 
This PR modifies the script `get_WE2Etest_names_subdirs_descs.sh` that creates the CSV file containing test information the columns for relative cost and time step (`DT_ATMOS`).  This allows users to have a better idea of what the cost of each test is (the relative cost is defined such that 1 unit represents the cost of running a single 6-hour forecast on the `RRFS_CONUS_25km` grid).  

Other related changes:
* Add the flag `generate_csv_file` to the `run_WE2E_tests.sh` script to allow users to skip the CSV file generation step (since it can take a while).
* Use arguments for the function `set_predef_grid_params()` so that it can be called not just from the experiment generation scripts but also others (in this case from `get_WE2Etest_names_subdirs_descs.sh`).  Necessary for this PR.
* Move some constants (`NH0`, `NH3`, `NH4`) from `config_defaults.sh` to `constants.sh` to make it easier to access them.
* Place the contents of `constants.sh` at the start of each experiment's `var_defns.sh` so that these constants are available to the j-jobs and ex-scripts when `var_defns.sh` is sourced.
* Use variable `valid_vals_BOOLEAN` in many more places (which allows many other `valid_vals_...` variables in `valid_param_vals.sh` and `setup.sh` to be removed).
* Several other bug fixes.

## TESTS CONDUCTED: 
The following tests (with results) were conducted on Hera.  There were only three (previously existing) failures.
```
* grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16:               SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp:           SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional:  SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2:                 SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16:                   SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR:                         SUCCESS
* grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_2017_gfdlmp:           SUCCESS
* grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2:                 SUCCESS
* grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16:                   FAILURE
* grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_HRRR:                            SUCCESS
* grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta:                     SUCCESS
* grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16:            SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR:                   SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta:            SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR:                    SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha:            SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta:             SUCCESS
* grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16:                  SUCCESS
* grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR:                          SUCCESS
* grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta:                   SUCCESS
* GST_release_public_v1:                                                       SUCCESS
* MET_ensemble_verification:                                                   SUCCESS
* MET_verification:                                                            SUCCESS
* community_ensemble_008mems:                                                  SUCCESS
* community_ensemble_2mems:                                                    SUCCESS
* custom_ESGgrid:                                                              SUCCESS
* custom_GFDLgrid:                                                             SUCCESS
* custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE:            SUCCESS
* custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_TRUE:             SUCCESS
* deactivate_tasks:                                                            SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019101818:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022518:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022600:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2021010100:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio:                             SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019101818:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022518:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022600:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021010100:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2021062000:                  SUCCESS
* get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS:                                        SUCCESS
* get_from_HPSS_ics_HRRR_lbcs_RAP:                                             SUCCESS
* get_from_HPSS_ics_RAP_lbcs_RAP:                                              SUCCESS
* get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio:                           SUCCESS
* inline_post:                                                                 SUCCESS
* nco_ensemble:                                                                SUCCESS
* nco_inline_post:                                                             SUCCESS
* pregen_grid_orog_sfc_climo:                                                  SUCCESS
* specify_DOT_OR_USCORE:                                                       SUCCESS
* specify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE:                                        SUCCESS
* specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS:                                       SUCCESS
* specify_RESTART_INTERVAL:                                                    SUCCESS
* specify_template_filenames:                                                  SUCCESS
* subhourly_post:                                                              FAILURE
* subhourly_post_ensemble_2mems:                                               FAILURE
```

## DOCUMENTATION:
Documentation is updated via PR #[278](ufs-community#278) in the ``ufs-srweather-app`` repo.
michelleharrold pushed a commit to michelleharrold/ufs-srweather-app that referenced this pull request Jun 7, 2023
@danielabdi-noaa @chan-hoo
[develop]: Remove shell workflow. (ufs-community#764)
6bd798f 
* Bug fix with FIELD_TABLE_FN

* Modify crontab management, use config_defaults.sh.

* Add status badge.

* Update cheyenne crontab management.

* source lmod-setup

* Add main to set_predef_grid

* Bug fix in predef_grid

* Don't import dead params.

* Fix bug in resetting VERBOSE

* Minor fix in INI config.

* Construct var_defns components from dictionary.

* Allow also lower case variables to be exported.

* Updates to python workflow due to PR ufs-community#776

* Use python versions of link_fix and set_FV3_sfc in job script.

* Use python versions of create_diag/model.

* Some fixes addressing Christina's suggestions.

* Delete shell workflow

* Append pid to temp files.

* Update scripts to work with the latest hashes of UFS_UTILS and UPP (ufs-community#775)

* update input namelist of chgres_cube

* update diag_table templates

* update scripts

* back to original

* specify miniconda version on Jet

* Remove -S option from link_fix call.

* Fixes due to merge

* Cosmoetic changes.

Co-authored-by: Chan-Hoo.Jeon-NOAA <[email protected]>
@christinaholtNOAA christinaholtNOAA deleted the retrieve_nemsio_test branch December 8, 2023 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MichaelLueken MichaelLueken MichaelLueken approved these changes

@JeffBeck-NOAA JeffBeck-NOAA JeffBeck-NOAA approved these changes

@mkavulich mkavulich Awaiting requested review from mkavulich mkavulich is a code owner

@gsketefian gsketefian Awaiting requested review from gsketefian gsketefian is a code owner

@RatkoVasic-NOAA RatkoVasic-NOAA Awaiting requested review from RatkoVasic-NOAA RatkoVasic-NOAA is a code owner

@BenjaminBlake-NOAA BenjaminBlake-NOAA Awaiting requested review from BenjaminBlake-NOAA BenjaminBlake-NOAA is a code owner

@ywangwof ywangwof Awaiting requested review from ywangwof ywangwof is a code owner

@chan-hoo chan-hoo Awaiting requested review from chan-hoo chan-hoo is a code owner

@panll panll Awaiting requested review from panll panll is a code owner

@christopherwharrop-noaa christopherwharrop-noaa Awaiting requested review from christopherwharrop-noaa christopherwharrop-noaa is a code owner

@danielabdi-noaa danielabdi-noaa Awaiting requested review from danielabdi-noaa danielabdi-noaa is a code owner

@mark-a-potts mark-a-potts Awaiting requested review from mark-a-potts mark-a-potts is a code owner

@jkbk2004 jkbk2004 Awaiting requested review from jkbk2004 jkbk2004 is a code owner

@willmayfield willmayfield Awaiting requested review from willmayfield willmayfield is a code owner

@dmwright526 dmwright526 Awaiting requested review from dmwright526 dmwright526 is a code owner

@gspetro-NOAA gspetro-NOAA Awaiting requested review from gspetro-NOAA gspetro-NOAA is a code owner

@natalie-perlin natalie-perlin Awaiting requested review from natalie-perlin natalie-perlin is a code owner

@EdwardSnyder-NOAA EdwardSnyder-NOAA Awaiting requested review from EdwardSnyder-NOAA EdwardSnyder-NOAA is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@christinaholtNOAA @JeffBeck-NOAA @MichaelLueken