GFSv16.3 upgrade

[KateFriedman-NOAA]

Description

The GFS will update in operations on WCOSS2 to include a data assimilation (GSI) upgrade and companion UPP and WAFS upgrades.

Spreadsheet: https://docs.google.com/spreadsheets/d/1fVa-yqGmxqwCrruq73agRKbXnpAPVzNOlvnu-pp1sn0/edit#gid=0
Timeline: https://docs.google.com/presentation/d/1y-KvQhYUzLlqfnfGKj6K0OO0WNPbeqUT_JAgIk9K2IE/edit#slide=id.g1207e8ee0a0_0_192
QUAD: https://docs.google.com/presentation/d/1Z1f7zpaSz3zjTOOG2doxaembIzxQYFSNPyBqpsbtxaA/edit#slide=id.p

The changes and objectives are :

1. Update standalone GFS Post Processing system and WAFS package to increase temporal/vertical resolutions, and forecast ranges of .25 deg WAFS products to meet 2023 IACO milestones for improving international flight planning and global aviation safety.
2. Improve overall global analysis and forecast through the addition of satellite winds, ozone and PAZ RO data, the enhanced NSST analysis, better use of ASCAT winds, GPS RO data, and surface sensitive IR radiances, and expanding the all-sky framework to include precipitation affected MW radiances and the all-sky assimilation of GPM GMI radiances.
3. Improve the decision making process of precipitation types in the forecast model to improve the accuracy of frozen precipitation forecasts.

To-do

• create release/gfs.v16.3.0 branch off of dev_v16 branch
• update GSI tag in checkout.sh and Externals.cfg ( @MichaelLueken-NOAA )
• update ufs-weather-model tag in checkout.sh and Externals.cfg ( @HelinWei-NOAA & @junwang-noaa )
• update UPP tag in checkout.sh and Externals.cfg ( @WenMeng-NOAA ) - upp_v8.2.0
• update GLDAS tag in checkout.sh and Externals.cfg ( @HelinWei-NOAA ) - gldas_gfsv16_release.v.2.1.0
• update WAFS tag in checkout.sh and Externals.cfg ( @YaliMao-NOAA )
• update crtm_ver in build.ver to 2.4.0
• update crtm_ver in run.ver to 2.4.0
• update obsproc_ver in run.ver to v1.1
• update obsproc_run_ver in target .ver files to 1.1.0
• install new obsproc tag under global space on supported platforms (@KateFriedman-NOAA)
• commit final resource adjustments into release/gfs.v16.3.0 (@lgannoaa )
• commit final workflow changes into release/gfs.v16.3.0
• update resources to address Cgroup mem limit exceeded warning messages
• update Fit2Obs version to newm.1.5 ( @KateFriedman-NOAA )
• update obsproc/prepobs versions as needed ( @KateFriedman-NOAA )
• pull in updated enkf_chgres_recenter_nc code from GDIT ( @GeorgeGayno-NOAA to submit PR)
• add release notes to docs folder in release/gfs.v16.3.0
• sync with operations branch via dev_v16.2.x branch (continuous)
• cut hand-off tag for CDF (EMC-v16.3.0)
• submit CDF to NCO (@emilyhcliu)
• check that GSI has been retagged for global_convinfo.txt flag change (@emilyhcliu)

Post-implementation tasks:

• merge release/gfs.v16.3.0 branch into dev/gfs.v16 branch after implementation
• cut gfs.v16.3.4 tag from dev/gfs.v16 branch
• announce implementation to glopara-announce listserv

Acceptance Criteria (Definition of Done)

System is assembled, built, and tested on WCOSS2 ahead of implementation.

Dependencies

• GSI update (new tag)
• ufs-weather-model (new tag)
• UPP (new tag)
• GLDAS (new tag)
• WAFS update (new tag)

[KateFriedman-NOAA]

Cut release/gfs.v16.3.0 branch from dev_v16 branch. Will sync updates from feature/ops-wcoss2 and dev_v16 branches into this new branch as needed.

[KateFriedman-NOAA]

Global Dump Archive on WCOSS2 is available here: /lfs/h2/emc/global/noscrub/emc.global/dump

[KateFriedman-NOAA]

Update from @WenMeng-NOAA :

To support GFS V16.3 implementation, the UPP branch release/gfs_v16 were updated the following changes:
1) aviation product upgrade
2) add continuous bucket precipitation product for wgne
A tag upp_v8.2.0 was cut for global workflow.

[KateFriedman-NOAA]

Updated UPP tag upp_v8.2.0 committed to release/gfs.v16.3.0 branch @ 3c2dd2c.

[WenMeng-NOAA]

@KateFriedman-NOAA Thanks!

[MichaelLueken]

There have been some minor renaming of DA component executables within release/gfsda.v16.3.0. These include:

global_gsi.x ---> gsi.x
global_enkf.x ---> enkf.x
ncdiag_cat.x ---> ncdiag_cat_serial.x

Should the scripts that utilize global_gsi.x, global_enkf.x, and ncdiag_cat.x be updated to point to gsi.x, enkf.x, and ncidag_cat_serial.x, respectively? Or, will the linking of these executables in link_fv3gfs.sh be updated?

I'm fine either way, I just want to make sure before I move forward with these changes.

Tagging @emilyhcliu @aerorahul @lgannoaa @KateFriedman-NOAA

[KateFriedman-NOAA]

Should the scripts that utilize global_gsi.x, global_enkf.x, and ncdiag_cat.x be updated to point to gsi.x, enkf.x, and ncidag_cat_serial.x, respectively?

My vote is to use the new exec names everywhere and update the scripts that use them. What do others think?

[emilyhcliu]

ncdiag_cat.x ---> ncdiag_cat_seri

Should the scripts that utilize global_gsi.x, global_enkf.x, and ncdiag_cat.x be updated to point to gsi.x, enkf.x, and ncidag_cat_serial.x, respectively?

My vote is to use the new exec names everywhere and update the scripts that use them. What do others think?

I agree with @KateFriedman-NOAA . Let's use the new names.

[KateFriedman-NOAA]

@emilyhcliu Can you confirm if the CRTM version is changing for this upgrade to 2.4.0? If so, we need to update that in the relevant version files in global-workflow. Thanks!

[WenMeng-NOAA]

@KateFriedman-NOAA I should clarify that the UPP has been using CRTM 2.3.0 to generate simulated satellite products in GFS V16.3.0.

[KateFriedman-NOAA]

I should clarify that the UPP has been using CRTM 2.3.0 to generate simulated satellite products in GFS V16.3.0.

Ok, thanks @WenMeng-NOAA ! Will need to confirm with @emilyhcliu if the GSI is updating the CRTM version and if UPP will need to test with it or not. The GFSv16.3.0 implementation spreadsheet lists CRTM 2.4.0.

[KateFriedman-NOAA]

Requested that Jun rename the new ufs-weather-model tag to reflect the GFS version it is for: GFS.v16.3.0

[KateFriedman-NOAA]

@lgannoaa @emilyhcliu FYI, PRs #853 and #855 have been merged into the release/gfs.v16.3.0 branch. The ufs-weather-model tag is now updated for the v16.3.0 upgrade. Update your testing clones when ready to get these updates. Thanks!

[KateFriedman-NOAA]

From October 7th RFC memo:

RFC 10095 - On WCOSS2, move the GFS/v16.3.2, obsproc/v1.1.2, and
bufr_dump/v1.1.2 packages to the production ecFlow server and turn on parallel
DBNet alerts in preparation for the GFS v16.3 30-day IT stability test. To be
implemented on October 12 at 1330Z.

UPPv.8.2.0 module installed into WCOSS2 production:

RFC 10103 - On WCOSS2, install the Unified Post Processing (UPP) and
Flexible Modelling System (FMS) module upgrades into production. The
UPP.v8.2.0 upgrade is needed for the GFS.v16.3 upgrade. FMS will be used in
the Unified Forecast System (UFS) as well as the Joint Effort for Data
assimilation Integration (JEDI). To be implemented on October 13, 1400Z to
1600Z.

[KateFriedman-NOAA]

From SDM log:

NOTIFICATION - GFS STABILITY TEST

13. 1355Z - SPA TL Steven sent notification that NCO has begun
the IT stability test for the Global Forecast System (GFS)
version 16.3.3

[KateFriedman-NOAA]

SCN: https://www.weather.gov/media/notification/pdf2/scn22-104_gfs.v16.3.0.pdf

[KateFriedman-NOAA]

Updated SCN: https://www.weather.gov/media/notification/pdf2/scn22-104_gfs.v16.3.0_aaa.pdf

[KateFriedman-NOAA]

Opened NOAA-EMC/hpc-stack#506 to get bufr/11.7.0 installed in hpc-stack-gfsv16 stacks on Hera and Orion.

[KateFriedman-NOAA]

November 18th 2022 RFC memo:

***RFC 10279 - On WCOSS2, implement GFS v16.3.3 and Obsproc v1.1.2 into
production by removing the evaluation flag and turning on production DBNet
alerts. Major highlights of this upgrade include:

● GFS Post Processing system and World Area Forecast System (WAFS)
upgrades
● FV3 model physics upgrade in the Noah Land Surface Model
● Grid-point Statistics Interpolation (GSI) Analysis - Enhance the use of
observations, add new data, improve the near sea surface temperature
(NSST) analysis, and bug fixes.
● Obsproc v1.1.2 upgrade is being done due to the addition of new
observation types and the removal of one other observation type.
The SCNs for this upgrade may be viewed at:
https://www.weather.gov/media/notification/pdf2/scn22-104_gfs.v16.3.0_aaa.pdf
and
https://www.weather.gov/media/notification/pdf2/scn22-103_obsproc.v1.1.0.pdf
To be implemented on November 29 and 30, 1100Z to 2000Z each day.