Updates for HR4 tag #2914

aerorahul · 2024-09-12T19:31:24Z

Description

This PR:

enables writing native grid model output when doing JEDI-atm DA
updates compression settings for C384 model output

Fixes #2891

Type of change

New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Should be tested on WCOSS2. Don't expect failures as the changes are not expected to break anything.

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

…odel output options in the write grid comp

emcbot · 2024-09-12T19:36:10Z

CI Update on Wcoss2 at 09/12/24 07:36:10 PM
============================================
Cloning and Building global-workflow PR: 2914
with PID: 7720 on host: dlogin03

DavidHuber-NOAA

Looks good. Thanks @aerorahul!

emcbot · 2024-09-12T20:23:40Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Thu Sep 12 19:39:20 UTC 2024 on dlogin03
---------------------------------------------------
Build: Completed at 09/12/24 08:23:19 PM
Case setup: Completed for experiment C48_ATM_1678a2f6
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_1678a2f6
Case setup: Skipped for experiment C48_S2SWA_gefs_1678a2f6
Case setup: Completed for experiment C48_S2SW_1678a2f6
Case setup: Completed for experiment C96_atm3DVar_extended_1678a2f6
Case setup: Skipped for experiment C96_atm3DVar_1678a2f6
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_1678a2f6
Case setup: Completed for experiment C96C48_hybatmDA_1678a2f6
Case setup: Completed for experiment C96C48_ufs_hybatmDA_1678a2f6

junwang-noaa

Looks good.

DavidNew-NOAA

Looks good, thanks

junwang-noaa · 2024-09-12T20:39:27Z

ush/parsing_model_configure_FV3.sh

+if [[ "${DO_JEDIATMVAR:-}" == "YES" ]]; then
+  local HISTORY_FILE_ON_NATIVE_GRID=".true."
+else
+  local HISTORY_FILE_ON_NATIVE_GRID=".false."


Quick question, when HISTORY_FILE_ON_NATIVE_GRID=".false.", will the softlinks cubed_sphere_grid_atmf???.nc/cubed_sphere_grid_sfcf???.nc still be created?

Good point @junwang-noaa . @aerorahul , logic should be added to forecast_postdet.sh where those soft links are created, depending on whether JEDI is being used.

Starting here:

global-workflow/ush/forecast_postdet.sh

Line 236 in 108db4d

${NLN} "${COMOUT_ATMOS_HISTORY}/${RUN}.t${cyc}z.cubed_sphere_grid_atmf${FH3}.nc" "cubed_sphere_grid_atmf${f_hhmmss}.nc"

Good catch @junwang-noaa
I'll remove the linking for when this is false.

@DavidNew-NOAA
Not sure how the cubed-sphere restarts got linked in the section where output_grid = Gaussian if-block.

@aerorahul Yeah, that was a mistake on my part

@aerorahul Well, maybe it was more sloppiness. That OUTPUT_GRID parameter is somewhat useless, because one cannot write only cubed-sphere histories the way UFS is configured. You either write Gaussian histories or both Gaussian and cubed-sphere histories, depending on how you set HISTORY_FILE_ON_NATIVE_GRID

DavidNew-NOAA · 2024-09-12T21:02:17Z

ush/parsing_model_configure_FV3.sh

@@ -31,7 +31,11 @@ local WRITE_GROUP=${WRITE_GROUP:-1}
 local WRTTASK_PER_GROUP=${WRTTASK_PER_GROUP:-24}
 local ITASKS=1
 local OUTPUT_HISTORY=${OUTPUT_HISTORY:-".true."}
-local HISTORY_FILE_ON_NATIVE_GRID=".true."
+if [[ "${DO_JEDIATMVAR:-}" == "YES" ]]; then


Should this be set in a config file, so that these lines in parsing_model_configure_FV3.sh become:

local HISTORY_FILE_ON_NATIVE_GRID=${history_file_on_native_grid:-".false."}

and history_file_on_native_grid is set in config.ufs

Maybe not, but I don't know what the design philosophy is and how all these scripts are supposed to fit together. It seems like some variables in this script are indeed coming from config.ufs.

emcbot · 2024-09-13T09:18:08Z

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_1678a2f6 *** SUCCESS *** at 09/12/24 10:00:20 PM
Experiment C48_S2SW_1678a2f6 *** SUCCESS *** at 09/12/24 10:18:18 PM
Experiment C96C48_hybatmDA_1678a2f6 *** SUCCESS *** at 09/12/24 10:57:32 PM
Experiment C96C48_hybatmaerosnowDA_1678a2f6 *** SUCCESS *** at 09/12/24 11:51:30 PM
Experiment C96C48_ufs_hybatmDA_1678a2f6 *** SUCCESS *** at 09/13/24 12:57:29 AM
Experiment C96_atm3DVar_extended_1678a2f6 *** SUCCESS *** at 09/13/24 09:15:35 AM

…=YES

aerorahul · 2024-09-13T13:30:42Z

@junwang-noaa @DavidNew-NOAA
I have enabled linking of cubed-sphere model-output only when JEDIATMVAR is turned on.
When doing replay, there is no DA, so it is unnecessary to write/save cubed sphere output.

CatherineThomas-NOAA

Thanks for acting quickly on this!

DavidHuber-NOAA

Looks good! Ran a quick test on WCOSS2 to verify that the cubed_sphere_grid_atmf###.nc links are not created when not running a JEDI experiment.

aerorahul · 2024-09-13T15:08:47Z

Thanks @DavidHuber-NOAA

DavidHuber-NOAA · 2024-09-13T15:11:13Z

Merging based on successful WCOSS2 tests.

aerorahul · 2024-09-13T15:28:03Z

@CatherineThomas-NOAA @RuiyuSun
A tag for HR4 has been released hr.v4

RuiyuSun · 2024-09-13T15:36:10Z

@CatherineThomas-NOAA @RuiyuSun A tag for HR4 has been released hr.v4

Great! Thanks!

emcbot · 2024-09-13T19:01:51Z

CI Passed on Hera in Build# 2
Built and ran in directory /scratch1/NCEPDEV/global/CI/2914


Experiment C48mx500_3DVarAOWCDA_6bd01783 Completed 2 Cycles: *SUCCESS* at Fri Sep 13 16:52:19 UTC 2024
Experiment C48_ATM_6bd01783 Completed 1 Cycles: *SUCCESS* at Fri Sep 13 16:58:24 UTC 2024
Experiment C96C48_hybatmDA_6bd01783 Completed 3 Cycles: *SUCCESS* at Fri Sep 13 17:59:44 UTC 2024
Experiment C96_atm3DVar_6bd01783 Completed 3 Cycles: *SUCCESS* at Fri Sep 13 18:12:09 UTC 2024
Experiment C48_S2SWA_gefs_6bd01783 Completed 1 Cycles: *SUCCESS* at Fri Sep 13 18:31:12 UTC 2024
Experiment C96C48_ufs_hybatmDA_6bd01783 Completed 2 Cycles: *SUCCESS* at Fri Sep 13 18:37:01 UTC 2024
Experiment C48_S2SW_6bd01783 Completed 1 Cycles: *SUCCESS* at Fri Sep 13 18:43:11 UTC 2024
Experiment C96C48_hybatmaerosnowDA_6bd01783 Completed 3 Cycles: *SUCCESS* at Fri Sep 13 19:01:43 UTC 2024

emcbot · 2024-09-13T19:36:17Z

CI Passed on Orion in Build# 1
Built and ran in directory /work2/noaa/stmp/CI/ORION/2914


Experiment C48_ATM_6bd01783 Completed 1 Cycles: *SUCCESS* at Fri Sep 13 12:51:47 PM CDT 2024
Experiment C48_S2SWA_gefs_6bd01783 Completed 1 Cycles: *SUCCESS* at Fri Sep 13 02:07:17 PM CDT 2024
Experiment C48_S2SW_6bd01783 Completed 1 Cycles: *SUCCESS* at Fri Sep 13 02:23:30 PM CDT 2024
Experiment C96_atm3DVar_6bd01783 Completed 3 Cycles: *SUCCESS* at Fri Sep 13 02:35:34 PM CDT 2024
Experiment C96C48_hybatmDA_6bd01783 Completed 3 Cycles: *SUCCESS* at Fri Sep 13 02:36:00 PM CDT 2024

* origin/develop: Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917) Cleanup job for GEFS (NOAA-EMC#2919) Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912) add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922) Support gefs C48 on Azure (NOAA-EMC#2881) Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)

* emc/develop: Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917) Cleanup job for GEFS (NOAA-EMC#2919) Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912) add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922) Support gefs C48 on Azure (NOAA-EMC#2881) Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914) Add new UPP links to .gitignore (NOAA-EMC#2904)

…al_obs_path * origin/develop: Update global atmos upp job to use COMIN/COMOUT (NOAA-EMC#2867) Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917) Cleanup job for GEFS (NOAA-EMC#2919) Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912) add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922) Support gefs C48 on Azure (NOAA-EMC#2881) Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)

disable native grid model output when not doing JEDI-atm DA. Update m…

1678a2f

…odel output options in the write grid comp

aerorahul requested review from DavidHuber-NOAA, DavidNew-NOAA, junwang-noaa, RuiyuSun and CatherineThomas-NOAA September 12, 2024 19:31

aerorahul added the CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS label Sep 12, 2024

emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Sep 12, 2024

WalterKolczynski-NOAA previously approved these changes Sep 12, 2024

View reviewed changes

aerorahul mentioned this pull request Sep 12, 2024

C768 gdasfcst runs too slow on WCOSS2 #2891

Closed

DavidHuber-NOAA previously approved these changes Sep 12, 2024

View reviewed changes

RuiyuSun approved these changes Sep 12, 2024

View reviewed changes

emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Sep 12, 2024

junwang-noaa approved these changes Sep 12, 2024

View reviewed changes

DavidNew-NOAA approved these changes Sep 12, 2024

View reviewed changes

junwang-noaa reviewed Sep 12, 2024

View reviewed changes

DavidNew-NOAA reviewed Sep 12, 2024

View reviewed changes

emcbot added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Sep 13, 2024

only link cubed-sphere output files from the model when DO_JEDIATMVAR…

6bd0178

…=YES

aerorahul dismissed stale reviews from DavidHuber-NOAA and WalterKolczynski-NOAA via 6bd0178 September 13, 2024 13:28

aerorahul added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Sep 13, 2024

CatherineThomas-NOAA approved these changes Sep 13, 2024

View reviewed changes

DavidHuber-NOAA approved these changes Sep 13, 2024

View reviewed changes

emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Sep 13, 2024

DavidHuber-NOAA merged commit 4ad9695 into NOAA-EMC:develop Sep 13, 2024
8 of 11 checks passed

aerorahul deleted the feature/hr4-updates branch September 13, 2024 15:19

emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Sep 13, 2024

emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Sep 13, 2024

emcbot added CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Sep 13, 2024

CatherineThomas-NOAA mentioned this pull request Dec 2, 2024

Atm history files incorrectly written with new compression options #3129

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates for HR4 tag #2914

Updates for HR4 tag #2914

aerorahul commented Sep 12, 2024

emcbot commented Sep 12, 2024

DavidHuber-NOAA left a comment

emcbot commented Sep 12, 2024

junwang-noaa left a comment

DavidNew-NOAA left a comment

junwang-noaa Sep 12, 2024

DavidNew-NOAA Sep 12, 2024 •

edited

Loading

aerorahul Sep 12, 2024

aerorahul Sep 12, 2024

DavidNew-NOAA Sep 12, 2024

DavidNew-NOAA Sep 13, 2024 •

edited

Loading

DavidNew-NOAA Sep 12, 2024 •

edited

Loading

emcbot commented Sep 13, 2024

aerorahul commented Sep 13, 2024

CatherineThomas-NOAA left a comment

DavidHuber-NOAA left a comment

aerorahul commented Sep 13, 2024

DavidHuber-NOAA commented Sep 13, 2024

aerorahul commented Sep 13, 2024

RuiyuSun commented Sep 13, 2024

emcbot commented Sep 13, 2024

emcbot commented Sep 13, 2024

Updates for HR4 tag #2914

Updates for HR4 tag #2914

Conversation

aerorahul commented Sep 12, 2024

Description

Type of change

Change characteristics

How has this been tested?

Checklist

emcbot commented Sep 12, 2024

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

emcbot commented Sep 12, 2024

junwang-noaa left a comment

Choose a reason for hiding this comment

DavidNew-NOAA left a comment

Choose a reason for hiding this comment

junwang-noaa Sep 12, 2024

Choose a reason for hiding this comment

DavidNew-NOAA Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

aerorahul Sep 12, 2024

Choose a reason for hiding this comment

aerorahul Sep 12, 2024

Choose a reason for hiding this comment

DavidNew-NOAA Sep 12, 2024

Choose a reason for hiding this comment

DavidNew-NOAA Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

DavidNew-NOAA Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

emcbot commented Sep 13, 2024

aerorahul commented Sep 13, 2024

CatherineThomas-NOAA left a comment

Choose a reason for hiding this comment

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

aerorahul commented Sep 13, 2024

DavidHuber-NOAA commented Sep 13, 2024

aerorahul commented Sep 13, 2024

RuiyuSun commented Sep 13, 2024

emcbot commented Sep 13, 2024

emcbot commented Sep 13, 2024

DavidNew-NOAA Sep 12, 2024 •

edited

Loading

DavidNew-NOAA Sep 13, 2024 •

edited

Loading

DavidNew-NOAA Sep 12, 2024 •

edited

Loading