Enhance IODA2NC to support IODA v2.0 format

[TaraJensen]

Describe the Enhancement

Rob from the Met Office received a sample IODA file from their DA group and the headers is very different from the currently supported format. Will M. took a look at it and confirmed the header is consistent with his understanding of the IODA v2.0 format. He is going to reach out to JEDI developers to determine if the format is set. If it is, then it would be good to add support for this format to IODA2NC. Here is the sample header that Rob sent. We will still need to get a sample file, but the file Rob had was very large and not conducive for testing.

Below is the header info which I got from running ncdump -h on the file I was given to look at: (Satwind data)

netcdf jopa_satwind_20210701T1200Z_out_0000 {
dimensions:
               nlocs = UNLIMITED ; // (2815062 currently)
variables:
               int nlocs(nlocs) ;
                              nlocs:suggested_chunk_dim = 10000LL ;
// global attributes:
                              string :_ioda_layout = "ObsGroup" ;
                              :_ioda_layout_version = 0 ;
group: DerivedObsError {
  variables:
               float eastward_wind(nlocs) ;
                              eastward_wind:_FillValue = -3.368795e+38f ;
               float northward_wind(nlocs) ;
                              northward_wind:_FillValue = -3.368795e+38f ;
  } // group DerivedObsError

group: DerivedObsValue {
  variables:
               float eastward_wind(nlocs) ;
                              eastward_wind:_FillValue = -3.368795e+38f ;
               float northward_wind(nlocs) ;
                              northward_wind:_FillValue = -3.368795e+38f ;
  } // group DerivedObsValue

group: Diag {
  variables:
               string satwind_id(nlocs) ;
                              string satwind_id:_FillValue = "*** MISSING ***" ;
  } // group Diag

group: EffectiveError {
  variables:
               float eastward_wind(nlocs) ;
                              eastward_wind:_FillValue = -3.368795e+38f ;
               float northward_wind(nlocs) ;
                              northward_wind:_FillValue = -3.368795e+38f ;
  } // group EffectiveError

group: EffectiveQC {
  variables:
               int eastward_wind(nlocs) ;
                              eastward_wind:_FillValue = -2147483643 ;
               int northward_wind(nlocs) ;
                              northward_wind:_FillValue = -2147483643 ;
  } // group EffectiveQC
 
group: GrossErrorProbability {
  variables:
               float eastward_wind(nlocs) ;
                              eastward_wind:_FillValue = -3.368795e+38f ;
               float northward_wind(nlocs) ;
                              northward_wind:_FillValue = -3.368795e+38f ;
  } // group GrossErrorProbability
 
group: MetaData {
  variables:
               float PGE_u_over_v(nlocs) ;
                              PGE_u_over_v:_FillValue = -3.368795e+38f ;
               float QI_common(nlocs) ;
                              QI_common:_FillValue = -3.368795e+38f ;
               float QI_estimated_error(nlocs) ;
                              QI_estimated_error:_FillValue = -3.368795e+38f ;
               float QI_recursive_filter_function(nlocs) ;
                              QI_recursive_filter_function:_FillValue = -3.368795e+38f ;
               float QI_weighted_mixture_exc_forecast_comparison(nlocs) ;
                              QI_weighted_mixture_exc_forecast_comparison:_FillValue = -3.368795e+38f ;
               float QI_with_forecast(nlocs) ;
                              QI_with_forecast:_FillValue = -3.368795e+38f ;
               float QI_without_forecast(nlocs) ;
                              QI_without_forecast:_FillValue = -3.368795e+38f ;
               float air_pressure(nlocs) ;
                              air_pressure:_FillValue = -3.368795e+38f ;
               float air_pressure_original(nlocs) ;
                              air_pressure_original:_FillValue = -3.368795e+38f ;
               int64 dateTime(nlocs) ;
                              dateTime:_FillValue = -9223372036854775801LL ;
                              string dateTime:units = "seconds since 1970-01-01T00:00:00Z" ;
               float latitude(nlocs) ;
                              latitude:_FillValue = -3.368795e+38f ;
               float longitude(nlocs) ;
                              longitude:_FillValue = -3.368795e+38f ;
               int originating_centre(nlocs) ;
                              originating_centre:_FillValue = -2147483643 ;
               int originating_subcentre(nlocs) ;
                              originating_subcentre:_FillValue = -2147483643 ;
               float percent_confidence_1(nlocs) ;
                              percent_confidence_1:_FillValue = -3.368795e+38f ;
               float percent_confidence_2(nlocs) ;
                              percent_confidence_2:_FillValue = -3.368795e+38f ;
               float percent_confidence_3(nlocs) ;
                              percent_confidence_3:_FillValue = -3.368795e+38f ;
               float percent_confidence_4(nlocs) ;
                              percent_confidence_4:_FillValue = -3.368795e+38f ;
               int priority(nlocs) ;
                              priority:_FillValue = -2147483643 ;
               float priority_numerator(nlocs) ;
                              priority_numerator:_FillValue = -3.368795e+38f ;
               int64 receiptdateTime(nlocs) ;
                              receiptdateTime:_FillValue = -9223372036854775801LL ;
                              string receiptdateTime:units = "seconds since 1970-01-01T00:00:00Z" ;
               int satellite_identifier(nlocs) ;
                              satellite_identifier:_FillValue = -2147483643 ;
               string satwind_id(nlocs) ;
                              string satwind_id:_FillValue = "*** MISSING ***" ;
               float sensor_central_frequency(nlocs) ;
                              sensor_central_frequency:_FillValue = -3.368795e+38f ;
               float sensor_zenith_angle(nlocs) ;
                              sensor_zenith_angle:_FillValue = -3.368795e+38f ;
               float total_observation_error(nlocs) ;
                              total_observation_error:_FillValue = -3.368795e+38f ;
               int wind_computation_method(nlocs) ;
                              wind_computation_method:_FillValue = -2147483643 ;
               int wind_generating_application_1(nlocs) ;
                              wind_generating_application_1:_FillValue = -2147483643 ;
               int wind_generating_application_2(nlocs) ;
                              wind_generating_application_2:_FillValue = -2147483643 ;
               int wind_generating_application_3(nlocs) ;
                              wind_generating_application_3:_FillValue = -2147483643 ;
               int wind_generating_application_4(nlocs) ;
                              wind_generating_application_4:_FillValue = -2147483643 ;
  } // group MetaData

group: MetaDataError {
  variables:
               float air_pressure(nlocs) ;
                              air_pressure:_FillValue = -3.368795e+38f ;
  } // group MetaDataError

group: ObsBias {
  variables:
               float eastward_wind(nlocs) ;
                              eastward_wind:_FillValue = -3.368795e+38f ;
               float northward_wind(nlocs) ;
                              northward_wind:_FillValue = -3.368795e+38f ;
  } // group ObsBias

group: ObsValue {
  variables:
               float wind_from_direction(nlocs) ;
                              wind_from_direction:_FillValue = -3.368795e+38f ;
               float wind_speed(nlocs) ;
                              wind_speed:_FillValue = -3.368795e+38f ;
  } // group ObsValue

group: QCFlags {
  variables:
               int eastward_wind(nlocs) ;
                              eastward_wind:_FillValue = -2147483643 ;
               int northward_wind(nlocs) ;
                              northward_wind:_FillValue = -2147483643 ;
  } // group QCFlags

group: hofx {
  variables:
               float eastward_wind(nlocs) ;
                              eastward_wind:_FillValue = -3.368795e+38f ;
               float northward_wind(nlocs) ;
                              northward_wind:_FillValue = -3.368795e+38f ;
  } // group hofx
}

This issue is related to METplus issue #1453

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2799991

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[robdarvell]

Hi,
I did think the file I was given was a little bit large, but it does appear to have transferred into the anonymous ftp server (under darvell_data) if you want to take a look at it.
Note: This was the first time I have been given this type of file, so it maybe possible that it will be refined in the future (ie what is stored within it), but it should give people a chance to look at it. It is Satwind data.

[JohnHalleyGotway]

@hsoh-u and @willmayfield, I wanted to make sure we grabbed this sample file before it ages off of the FTP server. I copied it over into seneca:/d1/projects/MET/MET_issues/feature_2068/jopa_satwind_20210701T1200Z_out_0000.nc4

[willmayfield]

@robdarvell @JohnHalleyGotway @hsoh-u noting there are also (small) examples in the ioda test files here: https://dashrepo.ucar.edu/dataset/147_miesch/file.html (sample_hofx_output_sondes.nc4, etc.)

Also, the current IODA format documentation: https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/inside/conventions/objects_and_layouts.html

[willmayfield]

@TaraJensen @JohnHalleyGotway @hsoh-u @robdarvell I heard from IODA developer who said these test files above should have a stable layout, meaning the group hierarchy within netcdf, but that many of the names of variables/groups will soon be changing, to be finalized sometime this spring.

[JohnHalleyGotway]

As of a meeting with the MetOffice on 8/17/2022, the IODAv2 format has been finalized and is stable. @robdarvell will provide a sample NetCDF file for to use in development. Please also provide a link to the correct documentation.

[hsoh-u]

Answer from Stephen Herbener [email protected]

Documentation:
https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/inside/conventions/index.html

Python example:
https://github.com/JCSDA-internal/ioda/blob/develop/src/engines/Examples/Basic/06-ReadIodaOutputFile.py

Note that the documentation is using the naming conventions from the JEDI Observation Data Conventions. Work is in progress to get the ioda files and jedi using those variable names meaning that the current ioda v2 files are using the older naming conventions. I thought the documentation however might be useful to give you an idea of how the file is structured.

For your questions:

1. The old naming convention (which is currently in use) is based on CF names. The CF names are more oriented toward the model side of things, but wherever there is overlap (eg, latitude) the CF names are used. But many of the names in the ioda files are mimicking CF since CF does not have a name defined for those quantities.

2. The winds, in some cases, come from data providers in the speed/direction format and are transformed to u/v components in UFO before applying QC checks. That is why you see QCFlags with the u,v components. Note that QCFlags holds derived variables which in this case are the results of the UFO QC checks. I'm not 100% sure, but there should be a DerivedObs or ObsDerived group with the results of transforming the speed,direction inputs to u,v components. If that is the case, then the derived obs variables should correspond with the QCFlags group.

Note too that QC marks that come from the data provider would be in a different group named PreQC and that group should contain the speed,direction format variables (and correspond to the variables in ObsValues).

3. I do not know the meanings of the QCFlags values. That's outside the scope of what I work on. You can check with someone in the OBS group, perhaps Ben Ruston or Greg Thompson.

4. The data types of the meta data should mostly have remained unchanged. However, one significant change is the datetime format which switched from an ISO 8601 string to a numeric offset in seconds from a given epoch (ie, reference datetime). This was done since processing the strings was becoming a performance bottleneck.

The name of the new numeric datetime meta data variable has also changed to "MetaData/dateTime" (note the upper case 'T'). The epoch is stored as a string attribute (named "epoch") on the MetaData/dateTime variable (in ISO 8601 format), and the data type for this variable is int64_t. The values of the variable are offsets from the epoch in seconds.

5. The names of the meta data variables themselves are mostly unchanged. I believe that the change from datetime to dateTime is the only name change. The group is still named MetaData.

One change between ioda v1 and ioda v2 is the expression of both the group and variable names together. In ioda v1, the name is "latitude@MetaData" whereas in ioda v2 the name is "MetaData/latitude". I think the ioda reader (and python interface) will for now accept both naming styles, but it would be good to make the switch now since the ioda v1 naming style will eventually be deprecated.

Hope this helps, and let me know if you have more questions.

Steve

[hsoh-u]

The units attribute of the dateTime variable at MetaData group is a string data type. The NetCDF API does not support this data type issue 2042.

[robdarvell]

Just back from leave, and working through my emails.
I see a comment on the QC flag values. Looking at those numbers quoted they appear to match up with values which the MetOffice output use for their QC setting.
These values are 2^ and bits relate to different options, ie background check, buddy check etc.
These values can also include more than one entry, ie 72 would be 64 + 8
If the number is odd (ie ends 1, 3, 5, 7, 9) then the overall picture is it has been rejected for one reason or another.
This is something I have been having fun and games with the work I have done on the obs side of things. There is no defined list as to what values will appear in any particular file, sometimes there could be up to 50 different ones. Generally, in the verif we do we simply look at the rejected (ie odd numbers) obs if we are going to take action on any of the obs.