Refactor ATM mode to work with python-based g-w structure

[RussTreadon-NOAA]

Addition of the GDASApp Aerosol mode to the global-workflow was done using a python based approach (see g-w PR #1106). This issue is opened to refactor the GDASApp ATM mode to use the same python-based g-w approach.

[RussTreadon-NOAA]

g-w specific work for this issue is documented in g-w issue #1313.

[RussTreadon-NOAA]

Work for this issue will be done in RussTreadon-NOAA:feature/refactor_atm. This branch was created as a copy of develop at a86b2e4.

[RussTreadon-NOAA]

@CoryMartin-NOAA & @ADCollard

I encountered a problem with parm/atm/obs/config/sondes.yaml when running fv3jedi_var.x in the 2021122100 gdasatmanlrun job. fv3jedi_var.x fails with

2: ABORT: FieldMetadata::getLongNameFromAnyName: Searching for a field called surface_geometric_height in the long, short and io names but not found anywhere.
2:        in file '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/sorc/gdas.cd/fv3-jedi/src/fv3jedi/FieldMetadata/FieldsMetadata.cc', line 142
2: Abort(1) on node 2 (rank 2 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2

The above is related to the following line found twice in parm/atm/obs/config/sondes.yaml

          geovar_sfc_geomz: surface_geometric_height

If surface_geometric_height is replaced with surface_geopotential_height, fv3jedi_var.x successfully runs to completion for 2021122100.

test_gdasapp_atm_jjob_var_run runs the 2021032318 case. As currently configured, sondes are not processed. I added sondes to the list of obs to process. test_gdasapp_atm_jjob_var_run failed with the same abort.

3: ABORT: FieldMetadata::getLongNameFromAnyName: Searching for a field called surface_geometric_height in the long, short and io names but not found anywhere.
3:        in file '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/fv3-jedi/src/fv3jedi/FieldMetadata/FieldsMetadata.cc', line 142

I replaced surface_geometric_height with surface_geopotential_height and reran test_gdasapp_atm_jjob_var_run. The rerun successfully completed.

surface_geopotential_height was replaced with surface_geometric_height in PR #328 (click here for diff).

May we safely revert surface_geometric_height back to surface_geopotential_height or is a different solution preferred?

[RussTreadon-NOAA]

@CoryMartin-NOAA , g-w PR #1354 has been opened with the refactored UFS DA ATM var and ens components. The UFS DA ATM var and ens jobs work from g-w.

I am now updating the GDASApp ctests. test_gdasapp_atm_jjob_var_prep is replaced by test_gdasapp_atm_jjob_var_init. The init test fails because it is looking for the amsua nc4 dump file in /scratch1/NCEPDEV/global/glopara/dump/gdas.20210323/18/atmos This file is not in the GDA. Previously test_gdasapp_atm_jjob_var_prep used R2D2 to get this file from your shared R2D2.

Eventually we can add a bufr-to-ioda step to convert the GDA amsua to a format fv3jedi_var.x can read. We're not at this point yet. How do we address this now? Does it make sense to create a GDA directory containing UFS DA pre-processed dump files? That is, copy obs dump files from your shared R2D2 to a new GDA directory. We would rename the files when populating the new GDA directory so they have the name the python based g-w expects.

[CoryMartin-NOAA]

sorry @RussTreadon-NOAA for missing the geopotential/geometric comment. I think it's safe for now and we can revisit later. Ideally FV3-JEDI should handle either and compute which it needs from the other.

The approach I was taking for the prototype aerosol cycling was to, as you suggested, create a GDA directory that has already converted IODA format files. Presumably this would most closely mimic a future implementation because either the 'obsproc' side would be handling the conversion, or we would read BUFR as-is into IODA.

[RussTreadon-NOAA]

Status Update

The g-w work for this issue was completed in RussTreadon-NOAA:feature/refactor_jediatm. g-w PR #1354 was opened and reviewed by EIB staff. Suggestions were incorporated and the updates accepted.

Instead of merging g-w PR #1354 into g-w develop, EIB requested that PR #1354 be broken into three PRs. The changes in g-w PR #1354 have been repackaged into the following g-w PRs:

• PR #1370 - UFS-DA aero component
• PR #1372 - UFS-DA ATM var component
• PR #1373 - UFS-DA ATM ens component

One challenge in doing this was that some of the files modified in PR #1354 contain changes for more than one UFS-DA component. An attempt was made to untangle these changes along with other component specific changes into the following forked branches:

• RussTreadon-NOAA:feature/update_jediaero
• RussTreadon-NOAA:feature/refactor_jedivar
• RussTreadon-NOAA:feature/refactor_jediens

The above three branches correspond to g-w PRs #1370 , #1372, and #1373, respectively.

GDASApp ctests run from g-w PR #1354 found that 6 tests fail

  Test #1377: test_gdasapp_atm_jjob_var_prep
  Test #1378: test_gdasapp_atm_jjob_var_run
  Test #1379: test_gdasapp_atm_jjob_var_post
  Test #1380: test_gdasapp_atm_jjob_ens_prep
  Test #1381: test_gdasapp_atm_jjob_ens_run
  Test #1382: test_gdasapp_atm_jjob_ens_post

This is expected given the changes in g-w #1354. These GDASApp tests will be in various states of pass or fail while the three g-w PRs above move through the review process. As the g-w PRs are merged into g-w develop updates will be made to this issue to fix broken ctests. The goal is to restore all six GDASApp ctests to pass shortly after all three g-w PRs have been merged into g-w develop.

[RussTreadon-NOAA]

While the following g-w PRs remain open

• PR #1370 - UFS-DA aero component
• PR #1372 - UFS-DA ATM var component
• PR #1373 - UFS-DA ATM ens component

a g-w parallel $HOMEgfs has been set up with changes from all three PRs. The parallel, prufsatm has cycled several days from 2021122100 on Hera.

The prufsatm uses GDASApp branch RussTreadon-NOAA:feature/refactor_atm. GDASApp ctests

  Test #1378: test_gdasapp_setup_atm_cycled_exp
  Test #1377: test_gdasapp_atm_jjob_var_prep
  Test #1378: test_gdasapp_atm_jjob_var_run
  Test #1379: test_gdasapp_atm_jjob_var_post
  Test #1380: test_gdasapp_atm_jjob_ens_prep
  Test #1381: test_gdasapp_atm_jjob_ens_run
  Test #1382: test_gdasapp_atm_jjob_ens_post

have been replaced by

  Test #1389: test_gdasapp_setup_atm_cycled_exp
  Test #1390: test_gdasapp_atm_jjob_var_init
  Test #1391: test_gdasapp_atm_jjob_var_run
  Test #1392: test_gdasapp_atm_jjob_var_final
  Test #1393: test_gdasapp_atm_jjob_ens_init
  Test #1394: test_gdasapp_atm_jjob_ens_run
  Test #1395: test_gdasapp_atm_jjob_ens_final

The ctests were updated to run with the changes in the g-w PRs. The atm var and ens ctests now run to completion. The changes required for this to happen were committed to feature/refactor_atm at 38cae2c.

[RussTreadon-NOAA]

Changes to GDASAPP_TESTDATA

To get the atm var and ens ctests to work with the pending g-w PRs, it is necessary to modify the lowres directory in GDASAPP_TESTDATA. These changes are listed below

• The updated global workflow changes the enkfgdas directory tree. The new structure is enkfgdas.$PDY/$cyc/memXXX/atmos.
• The updates in GDASApp feature/refactor_atm and the g-w PRs move away from using R2D2 to stage files. The files to stage (observations, bias correction, ...) have been moved into appropriate $GDASAPP_TESTDATA/lowres directories.

The restructured lowres directory is on Hera: /scratch1/NCEPDEV/stmp2/Russ.Treadon/CI/GDASApp/data/lowres.

@CoryMartin-NOAA , we will need to update lowres in the official GDASAPP_TESTDATA when the g-w PRs and GDASApp branch feature/refactor_atm are merged into their respective develop branches.

[CoryMartin-NOAA]

@RussTreadon-NOAA are you ready for me to do that now or should I wait until this PR is approved by others?

[RussTreadon-NOAA]

@CoryMartin-NOAA , I would like to move forward but I don't think we can. If we change GDASAPP_TESTDATA/lowres now, we break the existing atm_jjob ctests.

Let me open a GDASApp PR for this issue and note it's dependency on the three g-w PRs. We can update GDASAPP_TESTDATA/lowres once these four PRs are merged into their respective develop.