Preparation for turning on CrIS NPP radiances and include IASI Metop-C in operational GFS system

[emilyhcliu]

Timeline of Issus and Actions related to CrIS NPP

• 20210521 around 11:30 EDT: CrIS NPP began to show side-2 LW band problem
• 20210521 18Z: NESDIS turned off the CrIS NPP data stream (no CrIS NPP data at all)
• 20210528 12Z: NCO implemented satinfo change (set all CrIS NPP channels to -1)
• 20210602 12Z: NESDIS began to distribute side-2 MW+SW (no CrIS NPP LW data at all)
• 20210712: NESDIS began the process of switching CrIS NPP from side-2 to side-1
• 20210714: NESDIS completed the switch of CrIS NPP to side-1 and began the evaluation process
• 20210721: NESDIS completed the evaluation and held a review meeting (slides).
  The side-2 LW+SW data is in good shape. Data quality is comparable to those from side-1 before. After the side switch occurs, SNPP CrIS data will only be available from CLASS and GRAVITE. These data will be suspended from NDE/PDA until cleared by the calibration/validation science team.
• NOW: we need to prepare for using CrIS NPP again.

Official messages from NESDIS regarding the status CrIS NPP data

Preparation for CrIS NPP (switch back to side-1 for LW + SW)

Plan-A
The plan to cope with the scenario that we have to re-estimate the
correlated observation errors due to a perceivable change in data quality.

In this case, we should run cycled experiments in low resolution to get
the new estimation for CrIS NPP (We had previously verified that the
difference in error estimation from high-resolution and low resolution
is not significant. So we can use the low-resolution estimation in 
high-resolution run).

Plan-B
For the best-case scenario where the data quality from side-1 is good and
has similar characteristics as those from side-2, we can do the following:
(1) turn off water vapor channels (set iuse flag to -2; do not use the data at all).
(2) use the existing obs error matrix and get rid of the rows/columns that 
       correspond to H2O channels
(3) run a single-cycle experiment with conditions from (1) and (2)
      together and validate
(4) we will run a cycled experiment if time and resources allow.  

To do list:

• Prepare global_satinfo.txt file for the following changes: turn on CrIS NPP channels in LW band; set MW band channels to -2 (do not use; no bias coefficient update); and set SW band channels to -1 (monitoring; bias coefficient update for passive channels)
• Perform single-cycle test for a cycle before the CrIS NPP problem (Plan-B)
• Update correlated observation error matrix (see explanation why this is necessary from @KristenBathmann-NOAA)
• Perform a drill parallel experiment based on v16x_sept_ctl to test the updated satinfo and Rcov_crisnpp files (Sanity Check: gsistat plots}
• Run high-resolution single-cycle run with the real-time LW+SW data, updated satinfo, and the revised correlated obs error file (extended tasks)
• Create the RadMon plots for this parallel run and also replot the operational RadMon plots with longer time range to include the data from May 2021 (@HaixiaLiu-NOAA)
• Prepare abias and abis_pc (see Notes below) --- We will initialize CrIS NPP bias coefficients from zero
• Turn on the assimilation of IASI-C (using IASI-B correlated observation error for IASI-C)
• Run short high-resolution parallel experiment
  (1) update satinfo use flags for both CrIS NPP and IASI-C with the following updated configuration
  (2) use updated Rcov_crisnpp
  (3) use IASI-B Rcov for IASI-C
  (4) zero bias/bias_pc for CrIS NPP
  -[ ] Estimate Rcov for IASI-C using the high-resolution parallel run

Notes for abias and abias_pc (bias and pc files)

Background:

• We set the use flag for CrIS NPP in satinfo file to -1. This set all CrIS NPP channels to passive channels for monitoring and so the bias coefficients and pre-conditioning values are being estimated and updated unless the use flag is set to -2 (do not use) or no data flow at all.

• The NESDIS turned off the data stream entirely from 20210521 18Z, and then turned on the data (MW+SW) again from 20210602 12Z. The following describes the evolution of the bias and pc for CrIS NPP after NESDIS turned on MW+SW
  (1) LW channels: the bias coefficients stays the same as the 20210521 12Z (right before NESDIS turning off the data stream
  the pc values are set to default (10000); a large value since the data count is zero
  (2) MW channels: the bias coefficients were updated (since iuse_rad is set to -1) since the MW channels were available
  the pc values were estimated since the data count is not zero and greater than a pre-set threshold
  (3) SW channels: the bias coefficients were updated (since iuse_rad is set to -1) since the SW channels were available
  the pc values were estimated since the data count is not zero and greater than a pre-set threshold
  MW and SW channels are pass channels (not used in the assimilation at all). However, the bias estimation for these passive channels was not valid since LW channels were missing for QC (cloud detection). This won't impact the assimilation result.

What to do with the bias and pc values for CrIS NPP? --- we are going for Method#3

Method #1
Give NCO an updated bias and pc files with changes in CrIS NPP only
Use the bias and pc values for CrIS NPP channels from the cycle (20200521 12z) right before the CrIS NPP side-2 issue

Method #2
Give NCO an updated bias and pc files with changes in CrIS NPP only
(1) For LW channels: copy the bias coefficients and pre-conditioning values from the cycle (20200521 12Z) right before the CrIS NPP side-2 issue
(2) For MW and SW channels set both bias coefficients and pc to zero
(3) Remove CrIS NPP diagnostic file from the radstat (for the first cycle)

Method #3
Give NCO an updated bias and pc files with changes in CrIS NPP only
(1) Start bias and pc estimation from zero for CrIS NPP, if this is allowed in the operational system
(2) Remove CrIS NPP diagnostic file from the radstat (for the first cycle)

We better do a drill run for a few cycles before we hand these changes to NCO.

[emilyhcliu]

@RussTreadon-NOAA Please take a look at this issue when you have time. We can talk about it when you come back from your leave.

[emilyhcliu]

Update CrIS NPP iuse_rad flags in global_satinfo.txt:
LW channels (CO2 channels 1 - 713): turn on (same iuse_rad flag as CrIS N20)
MW channels (H2O channels 714 - 1578): set to -2 (do not use)
SW channels (Solar-affected channels 1579 - 2211): set to -1 (monitoring)

Updated global_satinfo.txt can be found in the following ProdGSI branch:
ProdGSI Branch: rev2_crisnpp

[emilyhcliu]

@KristenBathmann-NOAA performed single-cycle test to check if the correlated observation errors work properly with the LW+SW CrIS NPP data from side-1.
Here is the summary of the result from @KristenBathmann-NOAA:

I ran two single-cycle tests, the 2020121618 gdas analysis, and the 2020122700 gfs analysis. 
I set iuse to -2 for all CrIS npp MW channels (channels 714-1570). 
This reduces the number of active channels from 100 to 92. 
I've attached plots of the gdas cycle obs locations, and the gdas and gfs cycles OMA statistics. 
These results are compared to my control. 
Everything looks as it should.
However, updates to the CrIS npp covariance file are necessary to run with correlated error.

This is the result from GFS cycle. Here is the result from GDAS cycle

[emilyhcliu]

@MichaelLueken-NOAA Is it possible to add multiple assignees to this issue?
If so, please add the following people:
@KristenBathmann-NOAA
@HaixiaLiu-NOAA
@RussTreadon-NOAA

[emilyhcliu]

Messages from NESDIS

Administrative: Update #4 - CrIS on SNPP switch to Side 1 on July 12, 2021 - Issued July 21, 2021 1900 UTC

Update #4: SNPP CrIS SDR products have been declared Provisional following the SDR Science review team approval on July 21, 2021. SNPP CrIS SDR data flow through ESPC will resume on July 22, 2021 at 1400 UTC. The following three IDPS products will resume: CrIS-FS-SDR, CrIS-SDR-GEO, and CRIS-SCIENCE-RDR. Additionally, the following BUFR products will resume: CrIS_C0431_BUFR and CrIS_C2211_BUFR.

Update #3: The SNPP CrIS Side Switch Activities have been successfully completed and the Side-1 LWIR and SWIR bands are functional, while the MWIR band is non-operational. The SNPP CrIS SDR products will be declared Provisional on Wednesday July 21, 2021 following the SDR Science review team approval. EDRs will be available once validated by the science team.

Update #2: Day 2 activities have been completed as planned. Remaining activities are on schedule to be completed on July 14th.

Update #1: Day 1 commanding activities have been completed as planned. The remaining activities are on schedule to be conducted on July 13-14, 2021 as outlined in the original notification.

[KristenBathmann]

Emily added the stats plot from the gfs cycle that I ran. Here is the plot from the gdas cycle.

Removing the MW channels requires updates to the binary covariance file $FIXgsi/Rcov_crisnpp
See the end of $GSI/util/Correlated_Obs/cov_calc.f90 to understand the format of this file. It contains nch_active (number of actively assimilated channels, as specified in the satinfo) and nctot (total number of channels in the satinfo), which should be updated. It also contains indR, which are indices of the active channels, and Rcov, the covariance matrix. The MW channels need to be deleted from indR and Rcov. I have a fortran program that can make these changes.

[emilyhcliu]

@KristenBathmann-NOAA Thanks for performing the single-cycle test and the explanation of the required change for the R matrix.
I am setting up a drill parallel run to extend your single-cycle test.
I am using v16x_sept_ctl as control (reference) and v16x_sept_cris as an experiment to test the following:
(1) LW+SW (no MW) --- updated satinfo
(2) Updated correlated observation error file for CrIS --- updated Rcov_crisnpp

The parallel experiment setup can be found at the following location on HERA: /scratch1/NCEPDEV/da/Emily.Liu/para/v16x/v16x_sept_cris

The updated in satinfo and Rcov_crisnpp files are collected under the following directory:
/scratch1/NCEPDEV/da/Emily.Liu/para/v16x/v16x_sept_cris/fix_gsi

Please let me know when you have the updated Rcov_crisnpp. I will kick off the run after we have the updated Rcov_crisnpp.
We do not have to run long (maybe a week) and then run GSI statistics check against the control.

[emilyhcliu]

@MichaelLueken-NOAA Is it possible to add multiple assignees to this issue?
If so, please add the following people:
@KristenBathmann-NOAA
@HaixiaLiu-NOAA
@RussTreadon-NOAA

@MichaelLueken-NOAA Thank you!

[KristenBathmann]

The new file is:
/scratch1/NCEPDEV/da/Kristen.Bathmann/archive/v16rt2/Rcov_crisnpp_lwsw
You should rename it to Rcov_crisnpp when you copy it to the fix directory.
After your first analysis run, please pause the experiment so that I confirm it is set up correctly. Do this for the first gdas analysis, and the first gfs analysis. I will just need to look at the log files.

[emilyhcliu]

@KristenBathmann-NOAA
The drill run is running now with updated satinfo and your revised R file for CrIS NPP.
EXPDIR:/scratch1/NCEPDEV/da/Emily.Liu/para/v16x/v16x_sept_cris
ROTDIR:/scratch1/NCEPDEV/stmp4/Emily.Liu/ROTDIRS/v16x_sept_cris

The first GDAS analysis is 2020083006
The first GFS analysis is 2020083100

[RussTreadon-NOAA]

@RussTreadon-NOAA Please take a look at this issue when you have time. We can talk about it when you come back from your leave.

Thank you, Emily, Kristen, and Mike, for working together on this. My first day back at work is 7/29. What is the timeline for getting these changes into operations? Does this implementation only change fix files or are there other changes, too? Have we reached out NCO?

The Hera parallel is C384/C192, right? Is lo-res testing sufficient before we pass these changes to NCO? We may be able to squeeze an operational resolution short duration parallel on the production Dell. We don't want any surprises when these changes are implemented in operations.

[KristenBathmann]

Correlated error seems to be configured correctly for the 2020083006 gdas analysis

[emilyhcliu]

@KristenBathmann-NOAA
The drill run is running the first GDAS analysis.
Here is the gdas log file for you to check:
/scratch1/NCEPDEV/stmp4/Emily.Liu/ROTDIRS/v16x_sept_cris/logs/2020083006/gdasanal.log

[emilyhcliu]

@RussTreadon-NOAA Please take a look at this issue when you have time. We can talk about it when you come back from your leave.

Thank you, Emily, Kristen, and Mike, for working together on this. My first day back at work is 7/29. What is the timeline for getting these changes into operations? Does this implementation only change fix files or are there other changes, too? Have we reached out NCO?

The Hera parallel is C384/C192, right? Is lo-res testing sufficient before we pass these changes to NCO? We may be able to squeeze an operational resolution short duration parallel on the production Dell. We don't want any surprises when these changes are implemented in operations.

Hello, @RussTreadon-NOAA
First, about the timeline, NESDIS declared the CrIS NPP data (switching to side-1) provisional yesterday, and began to release data today. The data is provisional pending user feedback. So, we do have some time to work on this.

The single-cycle test done by Kristen is to find out if there is any issue in using the existing correlated observation error with the updated satinfo file (turning on LW channels, setting the missing MW channels to -2, and setting SW channels to -1) and the revised CrIS NPP data (LW+SW only). The test indicated that the MW channels should be removed from the Rcov file. The OMF and OMA are comparable with the single-cycle control run.

The drill parallel experiment (low-resolution) is to find out if the updated satinfo and Rcov for CrIS NPP work properly in the cycling run. And we have the control experiment (v16x_sept_ctl) as a reference for verification. We will keep this drill experiment running for a few days, and then do statistics check with the control experiment.

A few weeks ago, Kristen had made an assessment regarding the impact of model/analysis resolution on the Rcov estimation. The conclusion is that the impact is small and we could use the Rcov estimation from a high-resolution run in the low-resolution experiment and vice versa.

We need to think about abias and pc files for turning on the revised CrIS NPP data. Please see the Notes I wrote in the description section of this issue. It would be great if we can run a high resolution run for the changes we will hand over to NCO:
(1) updated satinfo for CrIS NPP
(2) updated Rcov for CrIS NPP
(3) abias and pc files

The single-cycle run and the drill parallel run are to make sure the items (1) and (2) work without problem.

[emilyhcliu]

Correlated error seems to be configured correctly for the 2020083006 gdas analysis

Yeh! Thanks for checking @KristenBathmann-NOAA
I will let you know when we have the first GFS analysis (2020083100)

I will add your revised Rcov_crisnpp to the ProdGSI branch rev2_crisnpp. So, we will have two pieces of required changes in one branch.

[RussTreadon-NOAA]

@RussTreadon-NOAA Please take a look at this issue when you have time. We can talk about it when you come back from your leave.

Thank you, Emily, Kristen, and Mike, for working together on this. My first day back at work is 7/29. What is the timeline for getting these changes into operations? Does this implementation only change fix files or are there other changes, too? Have we reached out NCO?

The Hera parallel is C384/C192, right? Is lo-res testing sufficient before we pass these changes to NCO? We may be able to squeeze an operational resolution short duration parallel on the production Dell. We don't want any surprises when these changes are implemented in operations.

@RussTreadon-NOAA Please take a look at this issue when you have time. We can talk about it when you come back from your leave.

Thank you, Emily, Kristen, and Mike, for working together on this. My first day back at work is 7/29. What is the timeline for getting these changes into operations? Does this implementation only change fix files or are there other changes, too? Have we reached out NCO?
The Hera parallel is C384/C192, right? Is lo-res testing sufficient before we pass these changes to NCO? We may be able to squeeze an operational resolution short duration parallel on the production Dell. We don't want any surprises when these changes are implemented in operations.

Hello, @RussTreadon-NOAA
First, about the timeline, NESDIS declared the CrIS NPP data (switching to side-1) provisional yesterday, and began to release data today. The data is provisional pending user feedback. So, we do have some time to work on this.

The single-cycle test done by Kristen is to find out if there is any issue in using the existing correlated observation error with the updated satinfo file (turning on LW channels, setting the missing MW channels to -2, and setting SW channels to -1) and the revised CrIS NPP data (LW+SW only). The test indicated that the MW channels should be removed from the Rcov file. The OMF and OMA are comparable with the single-cycle control run.

The drill parallel experiment (low-resolution) is to find out if the updated satinfo and Rcov for CrIS NPP work properly in the cycling run. And we have the control experiment (v16x_sept_ctl) as a reference for verification. We will keep this drill experiment running for a few days, and then do statistics check with the control experiment.

A few weeks ago, Kristen had made an assessment regarding the impact of model/analysis resolution on the Rcov estimation. The conclusion is that the impact is small and we could use the Rcov estimation from a high-resolution run in the low-resolution experiment and vice versa.

We need to think about abias and pc files for turning on the revised CrIS NPP data. Please see the Notes I wrote in the description section of this issue. It would be great if we can run a high resolution run for the changes we will hand over to NCO:
(1) updated satinfo for CrIS NPP
(2) updated Rcov for CrIS NPP
(3) abias and pc files

The single-cycle run and the drill parallel run are to make sure the items (1) and (2) work without problem.

Got it. Thank you, Emily.

[KristenBathmann]

The gfsanal also appears to be set up correctly.

[emilyhcliu]

The gfsanal also appears to be set up correctly.

@KristenBathmann-NOAA Thank you for checking.
I will let the drill run continue for a few cycles more and then I will do the gsistat plots for a sanity check.

[emilyhcliu]

These are the GSI statistics plots based on gsistat output:
There is a degradation in the RMSE for winds. The RMSE gap should become smaller as we cycling the run longer. We will continue the parallel experiment.

[emilyhcliu]

The LW+SW data flow is on now.
To do list:

• @emilyhcliu will prepare the input files for the single-cycle run.
• @KristenBathmann-NOAA will do a high-resolution single-cycle run with the real-time data (LW+SW), with the updated satinfo and the revised CRIS NPP correlated observation error.

[HaixiaLiu-NOAA]

I generated the RadMon plots for the period from 20210510 to 20210731 covering before and after CrIS_NPP side switches. This longer time series show different OmFnbc stats. Here is an example plot for a surface channel 194.

OmFnbc bias plot:

OmFnbc standard deviation:

[HaixiaLiu-NOAA]

Here is the link to the RadMon plots https://www.emc.ncep.noaa.gov/gmb/wx20hl/radmon.opr.CrIS_NPP/

[emilyhcliu]

@KristenBathmann-NOAA @HaixiaLiu-NOAA @RussTreadon-NOAA
I know that we will combine the IASI-C into the CrIS NPP work.
During our Friday meeting, the decisions made for IASI-C are the following:
(1) Use IASI-B correlation obs error estimation for IASI-C
(2) Do we want to keep IASI-C in monitoring mode or we will turn it on??

@kristen what is the status of your parallel experiment for replacing IASI-AB with IASI-BC?

[HaixiaLiu-NOAA]

@emilyhcliu @KristenBathmann-NOAA

I plan to turn IASI-C on in the parallel. My understanding is this high-reso parallel is going to be used to estimate the Rcov for IASI-C.

I have 2 questions about setup the parallel.

1, I have zeroed out bias correction coeff for CrIS_NPP, do we want to zero out the bias correction coeff for IASI-C as well? I plan to not zero out those coeff for IASI-C.
2, do I need to zero out abias_int?

[KristenBathmann]

My low resolution experiment turns on IASI-C and turns off IASI-A. IASI-B and both CrIS use updated covariance matrices computed from recent operations. IASI-C uses the new IASI-B matrices. The experiment is still in the beginning of September, so it has a while to go.

[RussTreadon-NOAA]

Impact of TLNMC modes for 2021092618 gdas

Run 2021092618 gdas case with various number of vertical modes, NVMODES_KEEP, retained by the TLNMC. Operations and the v16iasicris2 run with NVMODES_KEEP=8 Tabulated below are the final (min,max) sensible temperature analysis and increments (K) along with the global_gsi.x wall time (seconds). All jobs ran the global_gsi.x on Mars Phase 3 nodes (dev) with 420 tasks, ptile=7, and 4 threads per task.

NVMODES_KEEP	min TSEN_anl	max TSEN_anl	min TSEN_inc	max TSEN_inc	wall time
8	-7.197208336965E+01	3.162091326562E+02	-8.620760370897E+01	1.793433683642E+02	4784.132894
16	-2.453231766415E+05	1.807245904767E+05	-2.489749882328E+05	1.563747923358E+05	3528.656965
32	1.545324084268E+02	3.161397330673E+02	-1.095196881517E+01	9.224151897082E+00	5930.787827
64	1.655445993616E+02	3.161260995374E+02	-8.269896903773E+00	7.442405664575E+00	6509.331250

The NVMODES_KEEP=16 minimization aborted in the second outer loop at iteration 34 with the message

 PCGSOI: WARNING **** Stopping inner iteration ***
 Penalty increase or constant   2   34  0.826323728307584750E+15  0.826323728307584625E+15

This explains the low wall time for this configuration.

The gdas and enkf forecasts from the NVMODES_KEEP=8 run seg faulted shortly after execution began. Forecasts were not run from the NVMODES_KEEP=32 or NVMODES_KEEP=64 cases. Since these configurations retained positive analysis temperatures it is expected that the forecasts, if run, would have run to completion.

None of the above gsi runs placed a lower bound of qmin=1.0e-07 on the qsat computed in genqsat.f90. Adding the line
qsat(i,j,k) = max(qmin,qsat(i,j,k)) to genqsat.f90 and rerunning the 2021092618 gdas case with NVMODES_KEEP=8 yields the following sensible temperature results:

NVMODES_KEEP	min TSEN_anl	max TSEN_anl	min TSEN_inc	max TSEN_inc
8	1.190163096796E+02	3.162124911269E+02	-2.720773128018E+01	1.418466351114E+01

The numbers above come from the v16iasicris2 after updating the GSI source code. The 2021092618 gdas fcst ran past the point of the previous failures. Similar success is expected with the enkf forecasts.

[HaixiaLiu-NOAA]

Updated Release Notes for gfsv16.1.5

[HaixiaLiu-NOAA]

1.190163096796E

@RussTreadon-NOAA Should the above number be 1.190163096796E+02? Setting a lower bound of qmin=1.0e-07 on the qsat in genqsat.f90 results in relatively reasonable tsen_anl while not requiring a lot more wall time. EIB resumed the v16iasicris2 parallel and all the jobs have completed without issue at the 2021092612 cycle. The gdasanal.log at this cycle shows reasonable analysis and increment values.

[HaixiaLiu-NOAA]

@RussTreadon-NOAA Thank you for running those tests which provide helpful wall time guidance if the number of retained vertical modes increases.

[RussTreadon-NOAA]

1.190163096796E

@RussTreadon-NOAA Should the above number be 1.190163096796E+02? Setting a lower bound of qmin=1.0e-07 on the qsat in genqsat.f90 results in relatively reasonable tsen_anl while not requiring a lot more wall time. EIB resumed the v16iasicris2 parallel and all the jobs have completed without issue at the 2021092612 cycle. The gdasanal.log at this cycle shows reasonable analysis and increment values.

Yes, @HaixiaLiu-NOAA , you are correct. Thank you for reading the comment and pointing out my error. The table has been corrected.

[HaixiaLiu-NOAA]

The GFS.v16.1.5 will be implemented on October 26 at 1430Z (RFC 8823).

[HaixiaLiu-NOAA]

RFC 8823 - WCOSS:
Upgrade to GFS v16.1.5 for GSI changes ... @ Tue Oct 26, 2021
10:30 - 15:30 (EDT), Start to implement this RFC. 1930Z - This
RFC is completed.

[HaixiaLiu-NOAA]

@RussTreadon-NOAA @ADCollard My understanding is we should add an additional historical fix file to the gfsda.v16.1.5 release branch fix/gfsv16_historical/global_satinfo.2021102612 following the GFS v16.1.5 implementation. Is that right?

[RussTreadon-NOAA]

Yes, this is the pattern we have been following for the past few implementations. As you say, we cp fix/global_satinfo.txt fix/gfsv16_historical/global_satinfo.2021102612.

To which fix branch do we commit? The most complete approach is to commit to fix branch DA_GFSv16.1.5_global_only and then merge into rev2. If we commit to DA_GFSv16.1.5_global_only we should recreate tag gfsda.v16.1.5. One could also just commit to rev2 since this is the fix master.

Your question raises a larger question. What is the purpose of files in fix/gfsv16_historical? Is the intention that the dated files in this directory reflect when each file was implemented in operations? This is what the above accomplishes. fix/gfsv16_historical is a historical record of operational global fix files changes.

Another possibility is that we want fix/gfsv16_historical to reflect when to start using various observation types in retrospective parallels. Developers, not NCO, use the contents of fix/gfsv16_historical. Currently global-workflow configuration files config.anal and config.prep reference fix/gfsv16_historical via $RUN_ENVIR = "emc" blocks. These EMC only sections, in turn, contain logical tests which compare $CDATE with date ranges.
This allows us to toggle between various fix files as a retrospective parallel cycles forward.

There is a time lag between when we test updated fix files and when the changes are implemented. The $CDATE appended to files in fix/gfsv16_historical could reflect the date at which we determine it is safe, even desirable, to use the given fix file in retrospective parallels.

Whatever the meaning of the $CDATE we append to files in fix/gfsv16_historical, we need to update parm/config/config.anal and/or parm/config/config.prep accordingly. A check of the global-workflow develop parm/config/config.anal and parm/config/config.prep shows that this has not yet been done for GFS v16.1.5. Should we make the same changes to global-workflow tag EMC-v16.1.5?

My personal preference is to do away with fix/fv3_historical and fix/gfsv16_historical. Instead we have a source controlled usage database. This database tracks when we can/should use a given data type along with the operational evolution of usage for the given data type. The workflow queries the usage database for the given $CDATE and builds info and error files on the fly.

An additional keyword could be set in a configuration file to specify how we build the info and error files. dev or para mode mode means we use the data at the earliest date at which it is safe to use the data. opr or prod mode means we use the data when it was used in operations. If NCO prefers to use static fix files instead of building files on the fly, we create a snapshot of the fix files to be implemented. We currently do this with our DA_GFSvX.Y.Z_global_only branches in the fix repo.

Of course, the usage database approach has its pitfalls. How do we define the safe to use dates? These dates can (will) change as we develop new techniques to process data. Data usage varies from one DA system to another. Regional DA may (will) have different dates (both safe_to_use and implementation) than global DA. Similar comments apply for the CDAS, WDAS, etc. Do we have one usage database which covers all DA or multiple databases - one for each DA system?

We are moving away from GSI based DA. What will be the JEDI approach to addressing this topic?

[KateFriedman-NOAA]

Whatever the meaning of the $CDATE we append to files in fix/gfsv16_historical, we need to update parm/config/config.anal and/or parm/config/config.prep accordingly. A check of the global-workflow develop parm/config/config.anal and parm/config/config.prep shows that this has not yet been done for GFS v16.1.5. Should we make the same changes to global-workflow tag EMC-v16.1.5?

I'm looking to wrap up the v16.1.5 global-workflow work this week and merge release/gfs.v16.1.5 into the operations branch. I'm happy to ingest config changes for the fix file if-blocks before submitting the PR, let me know what you'd like to do. Thanks!

[CatherineThomas-NOAA]

Going back a bit further, the corresponding config changes were also not included for v16.1.3 (IASI-A) and v16.1.4 (DO-3). DO-2 is already in. Let me know if I can help.

[ADCollard]

@RussTreadon-NOAA, for what it's worth, the original intent of these files was to replicate usage in operations so that operations can be used as a control.

I think your suggestion of a database is the way forward. We need to start the conversation within the JEDI community on how to achieve this.

[ADCollard]

Unless someone else is volunteering to do it, I am happy to commit the changes.

So if I understand what Russ is saying we should first commit to DA_GFSv16.1.5_global_only and retag before merging with the master?

[RussTreadon-NOAA]

@ADCollard , thanks for the reminder. This agrees with the $CDATE suffix for most, if not all, the files in fix/gfsv16_historical. We need to ensure the workflow properly toggles between various historical fix files. At times developers may want to bypass this logic. This can be done in their personal checkout of the workflow.

The steps you outline sound right. @HaixiaLiu-NOAA noted that we need to add fix/gfsv16_historical/global_satinfo.2021102612. This is a copy of fix/global_satinfo. This gets committed to the fix branch you mention, DA_GFSv16.1.5_global_only.

From here we could update the fix submodule in release/gfsda.v16.1.5 and create a new gfsda.v16.1.5 tag, This isn't necessary but it's nice for completeness. We could skip this step and directly merge DA_GFSv16.1.5_global_only into rev2 and update the master fix submodule.

PR #237 has a DA Review Committee deadline of 11/1/2021. Pending a green light form the committee, Mike could begin working on PR #237 tomorrow. Fully closing PR #237 includes updating the fix and libsrc modules. We could work with Mike and combine the above v16.1.5 fix file addition with v16.x fix file changes.

[MichaelLueken]

When working with the fix submodule, I can begin by merging DA_GFSv16.1.5_global_only into DA_GFSv16.x_global_only, then merge DA_GFSv16.x_global_only into rev2 (wrapping up the fix submodule update).

Please comment on this issue once the file has been added to DA_GFSv16.1.5_global_only, then I will move forward with the fix submodule updates.

[ADCollard]

Looking in fix/gfsv16_historical of DA_GFSv16.1.5_global_only, I'm seeing global_satinfo.txt.202109220 which appears to be the v16.1.3 implementation and global_convinfo.txt.2021091612 which appears to be the v16.1.4 implementation.

The date associated with the latter appears to be when data became available rather than when it went operational (2021092712). Do we agree that this date should be changed?

[CatherineThomas-NOAA]

I'm okay with changing the date for DO-3. I went by the date first available, as I also did for DO-2 (global_convinfo.txt.2021031712).

Should all files be changed for their operational date? For instance, the earliest satinfo file here, global_satinfo.txt.2019021900, includes instruments not implemented until March 2021 with v16 (AHI Himawari-8, ABI G16, ATMS Ch. 15, etc). It seems like this would get messy very fast. My memory for this particular v16 directory was not for matching operations, but for running the pre-implementation retrospective parallels.

[RussTreadon-NOAA]

We have competing ideas for what historical fix files represent and how they are used. One approach is that they faithfully record the cycle at which the given fix file was implemented. This approach has an operational focus. The historical files are a record of operational data usage. Another approach is that info files represent the cycle at which it is safe to use data in the given file. This approach focuses on testing and development. We maximize the use of data in retrospective parallels. One directory of files can't support both options.

…

On Mon, Nov 1, 2021 at 1:57 PM CatherineThomas-NOAA < ***@***.***> wrote: I'm okay with changing the date for DO-3. I went by the date first available, as I also did for DO-2 (global_convinfo.txt.2021031712). Should all files be changed for their operational date? For instance, the earliest satinfo file here, global_satinfo.txt.2019021900, includes instruments not implemented until March 2021 with v16 (AHI Himawari-8, ABI G16, ATMS Ch. 15, etc). It seems like this would get messy very fast. My memory for this particular v16 directory was not for matching operations, but for running the pre-implementation retrospective parallels. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#186 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNN634HBMV5ABBIC4I2JNLUJ3IJFANCNFSM5AYQB5WQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[ADCollard]

So things are definitely a little messy. And I am beginning to think that I do not have the proper perspective on the v16 implementation.

Firstly, there are two directories: fv3_historical and gfsv16_historical. fv3_historical has not been updated since Jan 2020 (for an ozone change) with satinfo and convinfo changes last done in 2018.

I was assuming gfsv16_historical replaced the functionality of fv3_historical, but I guess now that the intent was to use the former for the v16 parallels.

So should we update fv3_historical to reflect the more recent operational upgrades?

[ADCollard]

Can I suggest we have a quick meeting to discuss this?

[ADCollard]

So at yesterday's meeting @RussTreadon-NOAA , @HaixiaLiu-NOAA, @CatherineThomas-NOAA and myself decided to relax the requirement for a definitive list of changes but to retain the gfsv16_historical/ directory to help with running retrospectives for the v16.x implementation only.

To that end, I have updated the DA_GFSv16.1_global_only fix branch to add new satinfo file to gfsv16_historical/ (reflecting the turning on of MetOp-C IASI on 2021102612) and have added a 0readme file to this directory that documents the changes between each of the historical conv/oz/satinfo files.

I have also removed the fv3_historical/ directory as that was becoming more and more out of date.

@MichaelLueken-NOAA, I believe this is ready to move forward with the fix submodule updates.

[RussTreadon-NOAA]

NCO creates implementation tags in their svn repository. Tags for GFS implementations are in

https://svnwcoss.ncep.noaa.gov/gfs/tags/

As of 11/2/2021 this directory has GFS tags from gfs.v12.0.2 to gfs.v16.1.5. GFS versions 12 to 14 do not include the GFS DA components. GFS DA, including DA fix files, came under the gfs directory structure starting with gfs.v15.1.1.

The gfs.v16.1.5 tag contains DA fix files in both the workflow location, fix/fix_gsi, and in the sorc/gsi.fd fix directory.

[RussTreadon-NOAA]

Prior to gfs.v15.1.1, NCO had the GFS in three main directories: gfs, gdas, and global_shared.

Tags for the DA components of GFS implementations v13.0.1 to v14.1.7 are in two directories:

• https://svnwcoss.ncep.noaa.gov/global_shared/tags/
• https://svnwcoss.ncep.noaa.gov/gdas/tags/

The global_shared directory contains GSI source code, scripts, jobs, and fix files. The gdas directory contains source code, scripts, jobs, and fix files for the ensemble part of the GFS DA package.

[KateFriedman-NOAA]

So at yesterday's meeting @RussTreadon-NOAA , @HaixiaLiu-NOAA, @CatherineThomas-NOAA and myself decided to relax the requirement for a definitive list of changes but to retain the gfsv16_historical/ directory to help with running retrospectives for the v16.x implementation only.

Do you guys still want to adjust the global_*txt file if-blocks in config.anal global-workflow or leave them as is and have users adjust their own config.anal files based on the GSI 0readme contents for now? Looking to wrap up the workflow side for this implementation. Thanks!

[ADCollard]

So at yesterday's meeting @RussTreadon-NOAA , @HaixiaLiu-NOAA, @CatherineThomas-NOAA and myself decided to relax the requirement for a definitive list of changes but to retain the gfsv16_historical/ directory to help with running retrospectives for the v16.x implementation only.

Do you guys still want to adjust the global_*txt file if-blocks in config.anal global-workflow or leave them as is and have users adjust their own config.anal files based on the GSI 0readme contents for now? Looking to wrap up the workflow side for this implementation. Thanks!

@KateFriedman-NOAA , I'll get you the workflow changes this morning.

[RussTreadon-NOAA]

Tag release/gfsda.v16.1.5 at c96150b as gfsda.v16.1.5.