The WE2E test "get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio" will never succeed due to bad dates

[mkavulich]

Description

The WE2E test "get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio" purports to test the ability to source data directly from NOMADS over the internet using wget. However, the internet site that the script sources (https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/) only keeps the last seven days of operational GFS

Steps to Reproduce

Please provide detailed steps for reproducing the issue.

1. Attempt to run WE2E test "get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio" on any platform
2. Observe failure due to trying to access nonexistent data.

Additional Context

The quickest way to fix this issue would be to modify the test to always use a current date (e.g. "yesterday"). This could have unforeseen problems, as currently all tests use static dates and input data, but it could also be a very good test to have as it could catch problems that may otherwise go unnoticed.

Output

This is the output if you attempt to run this test currently, after the experiment is set up:

Getting NOMADS online data
NOMADS_file_type= NEMSIO
Download files to /202008/20200826 ...
--2022-04-05 00:36:24--  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.atmanl.nemsio
Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Retrying.

--2022-04-05 00:38:32--  (try: 2)  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.atmanl.nemsio
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Giving up.

--2022-04-05 00:40:39--  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.sfcanl.nemsio
Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Retrying.

--2022-04-05 00:42:48--  (try: 2)  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.sfcanl.nemsio
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Giving up.

3
003
--2022-04-05 00:44:55--  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.atmf003.nemsio
Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Retrying.

--2022-04-05 00:47:04--  (try: 2)  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.atmf003.nemsio
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Giving up.

--2022-04-05 00:49:11--  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.sfcf003.nemsio
Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Retrying.

--2022-04-05 00:51:19--  (try: 2)  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.sfcf003.nemsio
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Giving up.

6
006
--2022-04-05 00:53:27--  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.atmf006.nemsio
Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Retrying.

--2022-04-05 00:55:35--  (try: 2)  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.atmf006.nemsio
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Giving up.

--2022-04-05 00:57:42--  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.sfcf006.nemsio
Resolving nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)... 140.172.138.18
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Retrying.

--2022-04-05 00:59:51--  (try: 2)  https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20200826/00/gfs.t00z.sfcf006.nemsio
Connecting to nomads.ncep.noaa.gov (nomads.ncep.noaa.gov)|140.172.138.18|:443... failed: Connection timed out.
Giving up.

For the above test, I reduced the wget number of tries from the default of 20 to a more reasonable 2 so that the failure would take less than several hours. This is another change that should be made.

[mkavulich]

Issue migrated to ufs-srweather-app repository: #728