Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WE2E test grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta fails on Hera Intel in the run_fcst step #805

Closed
MichaelLueken opened this issue May 19, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@MichaelLueken
Copy link
Collaborator

Expected behavior

The WE2E coverage tests should successfully pass on any given machine.

Current behavior

The grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta test is failing in the run_fcst step with the following error message on Hera Intel (NCO-mode test):

FATAL from PE 25: NetCDF: Index exceeds dimension bound: netcdf_read_data_2d: file:INPUT/gfs_data.nc- variable:ps

Please note that, if this test is run by itself, it will successfully pass. Also, this test will pass on other machines.

Machines affected

Hera Intel WE2E coverage tests

Steps To Reproduce

  1. Clone the current develop branch on Hera
  2. Run the coverage.hera.intel.nco suite
  3. See that the grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta test fails in run_fcst step with the above error message.
@MichaelLueken MichaelLueken added the bug Something isn't working label May 19, 2023
michelleharrold pushed a commit to michelleharrold/ufs-srweather-app that referenced this issue Jun 7, 2023
Adds functionality to ush/retrieve_data.py that supports ingesting global ensemble members in the ways that have been used for RRFS. It also adds GEFS and GDAS known data locations to ush/templates/data_locations.yml. This means that it will pull data from AWS and/or HPSS for:

- GEFS ICS and LBCS grib2 a & b files (it does not merge them, as that is outside the scope of this tool's functionality).
- GDAS ICS and LBCS netcdf files. For ICS, it needs atm and sfc files. For LBCS, it only needs atm files.

The tool is set up to pull all members as a single job, or it can be configured to do a subset of members if the user prefers to pull subsets of members. Just like for other datasets, the ICS and LBCS should be handled in separate calls to the tool, and the tool only handles a single start time since that is consistent with the behavior of the workflow.

Additional functionality now includes the ability to template the output directory to include the ensemble member so each member can be stored in its own separate directory when the workflow runs.

I have added functional tests to pull data from AWS and HPSS, and have hooked the AWS tests up to GitHub Actions to run when the relevant files have changed. No user intervention is needed to have those tests run -- they should run automatically upon opening a PR.
@MichaelLueken
Copy link
Collaborator Author

This test successfully runs on Hera Intel as part of the comprehensive test suite and also on Hercules as part of the coverage tests for that machine. Closing obsolete issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant