Enhance Point2Grid to support modified quality control settings for smoke/dust AOD data in GOES-16/17 as of April 16, 2024

[JohnHalleyGotway]

Describe the Enhancement

This issue arose from an email from Ho-Chun Huang on April 7, 2023. Per Amy Huff, as of April 16 (previously April 9th), 2023, a change will be made to the quality flags for the smoke and dust AOD data provided in GOES 16/17 files.

Per Ho-Chun, what matters to Points2Grid Smoke and Dust AOD processing probably is the reverse values of the smoke/dust detection confidence (in ADP files) for the Smoke and Dust AOD (detection confidence =0 is now high quality Smoke/Dust AOD).

This issue is to update the Point2Grid algorithm for the next version of MET.

Please be aware that the GOES-East/West ABI Aerosol Detection Product (ADP) smoke/dust mask current (Baseline) algorithm is being updated to the Enterprise algorithm tomorrow, April 9:
https://www.ospo.noaa.gov/data/messages/2024/04/MSG_20240404_1823.html

The "Smoke" and "Dust" variables in the ADP output file will remain the same, but other key variables/flag values for processing smoke/dust detections will change, as outlined in the following table.  Please update your scripts!

This issue is to review the logic and documentation of the Point2Grid tool in MET:

• Determine whether these changes are indicated by an updated version number in the data files... or whether they are based strictly on the timestamp. Either way, remember to add a new unit test that demonstrates the updated processing logic. And remember to confirm that data prior to April 9, 2024 continue to be processed in the "correct" way.
• Confirm that the quality control confidence values are being parsed correctly from NEW files.
• Confirm that the quality control confidence values are being parsed correctly from OLD files.
• Remember to update the MET and METplus documentation, if needed.

Time Estimate

1 day?

Sub-Issues

Consider breaking the enhancement down into sub-issues.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2784543 UFS R2O

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Review default alert labels
• Select component(s)
• Select priority
• Select requestor(s)

Milestone and Projects

• Select Milestone as the next official version or Backlog of Development Ideas
• For the next official version, select the MET-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
  This may impact the output produced by existing METplus Use Cases.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the next official version
  Select: MET-X.Y.Z Development project for development toward the next official release
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[AliciaBentley-NOAA]

Without this update, EMC will not be able to operationally verify HRRR and RAP Smoke AOD in EVS v2.0. That is why this update is required. Thank you!

[Ho-ChunHuang-NOAA]

ABI Algorithm updates of GOES East/West has been delayed to April 17 2024.

[hsoh-u]

Is the data file with a new algorithm available?
I wonder the meaning of the flags are saved into the ADP file. Here are variable attributes of the old ADP file (generated year 2019)
Filename: goes_16/OR_ABI-L2-ADPC-M6_G16_s20192662141196_e20192662143569_c20192662144526.nc
Variable attribiutes:

        ushort DQF(y, x) ;
                DQF:_FillValue = 65535US ;
                DQF:long_name = "ABI L2+ Aerosol Detection data quality flags" ;
                DQF:standard_name = "status_flag" ;
                DQF:valid_range = 0US, 255US ;
                DQF:flag_masks = 1US, 1US, 2US, 2US, 12US, 12US, 12US, 48US, 48US, 48US, 64US, 64US, 128US, 128US ;
                DQF:flag_values = 0US, 1US, 0US, 2US, 0US, 4US, 12US, 0US, 16US, 48US, 0US, 64US, 0US, 128US ;
                DQF:flag_meanings = "good_smoke_detection_retrieval_qf invalid_smoke_detection_due_to_snow_ice_clouds_or_degraded_source_data_qf good_dust_detection_retrieval_qf invalid_dust_detection_due_to_snow_ice_clouds_or_bad_source_data_qf low_confidence_smoke_detection_qf medium_confidence_smoke_detection_qf high_confidence_smoke_detection_qf low_confidence_dust_detection_qf medium_confidence_dust_detection_qf high_confidence_dust_detection_qf out_of_sun_glint_qf within_sun_glint_qf within_valid_solar_and_satellite_zenith_angle_range_qf outside_valid_solar_or_satellite_zenith_angle_range_qf" ;
                DQF:number_of_smoke_detection_qf_values = 2UB ;

[Ho-ChunHuang-NOAA]

Our data center are currently down so I lost the pathway to move the data among WCOSS2, ftp server, and local PC system (where I can drop the file to the GitHub issues). Will upload the new datafile after our Data Center is functioned again.

[Ho-ChunHuang-NOAA]

Recap email sent

Hi, Howard:

I hope my memory helps.

(1) If simply handling total AOD, we do not need to specify the -adp option for point2grid. In this case, the qc flag is read from the same AOD input file and -qc=0,1,2 means AOD quality is "high", "medium", and "low", respectively. I believe this part should remain the same.

(2) If a need to map Smoke/Dust AOD, additional information of the values of smoke/dust detection confidence need to be read from ADP file to determine the status of quality of mapped Smoke/Dust AOD. Because the corresponding value of smoke/dust detection confidence changed in the new algorithm, from your email, the conversion from detection confidence to quality flag may need to be updated to reflect the new values.

I think the logic of conversion needs to be reviewed to reflect the new definition of ADP detection confidence. I recall we madea table for the conversion to high. medium. and low quality Smoke/Dust AOD. The current rule maybe like

AOD qc flag=0 + ADP Smoke detection confidence=12 ==> high quality Smoke AOD.
AOD qc flag=0 + ADP Smoke detection confidence=4 ==> medium quality Smoke AOD.
AOD qc flag=0 + ADP Smoke detection confidence=0 ==> low quality Smoke AOD

AOD qc flag=1 + ADP Smoke detection confidence=12 ==> medium quality Smoke AOD.
AOD qc flag=1 + ADP Smoke detection confidence=4 ==> medium quality Smoke AOD.
AOD qc flag=1 + ADP Smoke detection confidence=0 ==> low quality Smoke AOD

AOD qc flag=2 + ADP Smoke detection confidence=12 ==> low quality Smoke AOD.
AOD qc flag=2 + ADP Smoke detection confidence=4 ==> low quality Smoke AOD.
AOD qc flag=2 + ADP Smoke detection confidence=0 ==> low quality Smoke AOD.

AOD qc flag=0 + ADP Dudt detection confidence=48 ==> high quality Dust AOD.
AOD qc flag=0 + ADP Dust detection confidence=16 ==> medium quality Dust AOD.
AOD qc flag=0 + ADP Dust detection confidence=0 ==> low quality Dust AOD

AOD qc flag=1 + ADP Dust detection confidence=48 ==> medium quality Dust AOD.
AOD qc flag=1 + ADP Dust detection confidence=16 ==> medium quality Dust AOD.
AOD qc flag=1 + ADP Dust detection confidence=0 ==> low quality Dust AOD

AOD qc flag=2 + ADP Dust detection confidence=48 ==> low quality Dust AOD.
AOD qc flag=2 + ADP Dust detection confidence=16 ==> low quality Dust AOD.
AOD qc flag=2 + ADP Dust detection confidence=0 ==> low quality Dust AOD.

If you found the conversion relationship in current is like what I described above, then

======== the new rule ==========

AOD qc flag=0 + ADP Smoke detection confidence=0 ==> high quality Smoke AOD.
AOD qc flag=0 + ADP Smoke detection confidence=4 ==> medium quality Smoke AOD.
AOD qc flag=0 + ADP Smoke detection confidence=8 ==> low quality Smoke AOD

AOD qc flag=1 + ADP Smoke detection confidence=0 ==> medium quality Smoke AOD.
AOD qc flag=1 + ADP Smoke detection confidence=4 ==> medium quality Smoke AOD.
AOD qc flag=1 + ADP Smoke detection confidence=8 ==> low quality Smoke AOD

AOD qc flag=2 + ADP Smoke detection confidence=0 ==> low quality Smoke AOD.
AOD qc flag=2 + ADP Smoke detection confidence=4 ==> low quality Smoke AOD.
AOD qc flag=2 + ADP Smoke detection confidence=8 ==> low quality Smoke AOD.

AOD qc flag=0 + ADP Dudt detection confidence=0 ==> high quality Dust AOD.
AOD qc flag=0 + ADP Dust detection confidence=16 ==> medium quality Dust AOD.
AOD qc flag=0 + ADP Dust detection confidence=32 ==> low quality Dust AOD

AOD qc flag=1 + ADP Dust detection confidence=0==> medium quality Dust AOD.
AOD qc flag=1 + ADP Dust detection confidence=16 ==> medium quality Dust AOD.
AOD qc flag=1 + ADP Dust detection confidence=32 ==> low quality Dust AOD

AOD qc flag=2 + ADP Dust detection confidence=0==> low quality Dust AOD.
AOD qc flag=2 + ADP Dust detection confidence=16 ==> low quality Dust AOD.
AOD qc flag=2 + ADP Dust detection confidence=32 ==> low quality Dust AOD.

Ho-Chun

[Ho-ChunHuang-NOAA]

By Howard

The DQF variable at the AOD file:

    byte DQF(y, x) ;
            DQF:long_name = "ABI L2+ Aerosol Optical Depth at 550 nm data quality flags" ;
            DQF:standard_name = "status_flag" ;
            DQF:_Unsigned = "true" ;
            DQF:valid_range = 0b, 3b ;
            DQF:flag_values = 0b, 1b, 2b, 3b ;
            DQF:flag_meanings = "high_quality_retrieval_qf medium_quality_retrieval_qf low_quality_retrieval_qf no_retrieval_qf" ;

So "AOD qc flag=0", "AOD qc flag=1", and "AOD qc flag=2" mean the DQF value at AOD file. Is it correct?

Question 1. Filtering out if "AOD qc flag=3". Is it correct?
Question 2. Filtering out if Smoke/Dust detection confidence does not belong to high, medium and low quality. Filtering out with following cases. Is it correct?
Any AOD qc flag + ADP Smoke detection confidence=8
Any AOD qc flag + ADP Dust detection confidence=32

Cheers,
Howard

[Ho-ChunHuang-NOAA]

@hsoh-u Yes to your question1. It is not clear to me whether your question 2 referred to Smoke or Dust AOD. Simply from the current algorithm, replacing threshold values 0 -> 8, then 12->0 for Smoke AOD. 0->32 then 48-> 0 for Dust AOD. We do not need to re-create the rule. I think we should keep the flexibility of choice for users. Your question seems to indicate the only choice is high quality Smoke AOD or Dust AOD.

ho-Chun

[Ho-ChunHuang-NOAA]

@JohnHalleyGotway Can you get data from Hera for Howard? That is another pathway for GOES AOD and ADP data transfer.

[hsoh-u]

Please ignore this comment.

The question 2 is how to handle ADP Smoke confidence=8 and ADP Dust confidence=32 for Baseline algorithm and ADP Smoke confidence=12 and ADP Dust confidence=48 for Enterprise algorithm.
Case 1: when -qc option is not given (MET does not filter out).
Case 2: if ADP Smoke confidence=8 and ADP Dust confidence=32 can be included with -qc 0,1,2,3.

The possible values for Baseline algorithm:

• possible ADP Smoke detection confidence: 0, 4, 8, 12.
• possible ADP Dust detection confidence: 0, 16, 32 48.
• Existing rule for ADP Smoke (0bDDD: binary number)
  • 0b0000 = 0 = low
  • 0b0100 = 4 = medium
  • 0b1000 = 8 (filtering out by -qc option)
  • 0b1100 = 8 + 4 = 12 = high.
• Existing rule for ADP Dust
  • 0b000000 = 0 = low
  • 0b010000 =16 = medium
  • 0b100000 = 32 (filtering out by -qc option)
  • 0b110000 = 32 + 16 = 48 = high.

[hsoh-u]

Please ignore above comment.

[Ho-ChunHuang-NOAA]

@hsoh-u Saw your latest post to ignore previous posting. But allow me to provide my comment for your consideration.
"YES" on
So "AOD qc flag=0", "AOD qc flag=1", and "AOD qc flag=2" mean the DQF value at AOD file. Is it correct?

The discussion below specifically points to Point2Grid with -adp option

If no -qc option is given, then qc=0,1,2 high+medium+low is the default, i.e., filtering AOD qc flag = 3 OR ADP Smoke detection confidence =12 (bad_missing_smoke_detection) for SMOKE AOD and ADP Dust detection confidence =48 (bad_missing_dust_detection) for Dust AOD.

One should not allow 3 as an option in point2grid, 3 mean all obs should be filtered out. I do not think any user will choose that option for verification purpose.

• AOD qc=3=no retrieval
• ADP Smoke detection confidence=12=bad_missing_smoke_detection
• ADP Dust detection confidence=48=bad_missing_dust_detection

Available options for -qc

• 0 high only
• 1 medium only
• 2 low only
• 0,1 high+medium
• 0,1,2 high+medium+low (default if no -qc is given)

Borrowing your description, I think for the "Enterprise Algorithm"

The possible values for Enterprise Algorithm:

• possible ADP Smoke detection confidence: 0, 4, 8, 12.
• possible ADP Dust detection confidence: 0, 16, 32 48.
• Revised rule for ADP Smoke (0bDDD: binary number)
  • 0b0000 = 0 = high
  • 0b0100 = 4 = medium
  • 0b1000 = 8 = low
  • 0b1100 = 8 + 4 = 12 = (filtering out )
• Revised rule for ADP Dust
  • 0b000000 = 0 = high
  • 0b010000 =16 = medium
  • 0b100000 = 32 = low
  • 0b110000 = 32 + 16 = 48 = (filtering out).

[hsoh-u]

@Ho-ChunHuang-NOAA Thank you for the comments.

• "no retrieval" will be filtered out without -qc option
• I agree MET user won't add 3 into -qc option

[Ho-ChunHuang-NOAA]

The file size is over the limit for add file function.

Please obtain the sample aod and adp file on 04/19/2024

adp_20240419.tar aod_20240419.tar

from

https://www.emc.ncep.noaa.gov/mmb/hchuang/ftp/

[hsoh-u]

Thank you for the files.
The algorithm can be detected by looking at the DQF:flag_values and DQF:flag_meanings.

DQF:flag_values and DQF:flag_meanings (20190801/OR_ABI-L2-ADPC-M6_G16_s20192131001462_e20192131004235_c20192131005097.nc)

• 0US: good_smoke_detection_retrieval_qf
• 1US: invalid_smoke_detection_due_to_snow_ice_clouds_or_degraded_source_data_qf
• 0US: good_dust_detection_retrieval_qf
• 2US: invalid_dust_detection_due_to_snow_ice_clouds_or_bad_source_data_qf
• 0US low_confidence_smoke_detection_qf
• 4US: medium_confidence_smoke_detection_qf
• 12US: high_confidence_smoke_detection_qf
• 0US: low_confidence_dust_detection_qf
• 16US: medium_confidence_dust_detection_qf
• 48US: high_confidence_dust_detection_qf
• 0US: out_of_sun_glint_qf
• 64US: within_sun_glint_qf
• 0US: within_valid_solar_and_satellite_zenith_angle_range_qf
• 128US: outside_valid_solar_or_satellite_zenith_angle_range_qf

DQF:flag_valaues and DQF:flag_meanings (20240419/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc)

• 2US: low_confidence_ash_detection_qf
• 1US: medium_confidence_ash_detection_qf
• 0US: high_confidence_ash_detection_qf
• 3US: bad_missing_ash_detection_qf
• 8US: low_confidence_smoke_detection_qf
• 4US: medium_confidence_smoke_detection_qf
• 0US: high_confidence_smoke_detection_qf
• 12US: bad_missing_smoke_detection_qf
• 32US: low_confidence_dust_detection_qf
• 16US: medium_confidence_dust_detection_qf
• 0US: high_confidence_dust_detection_qf
• 48US: bad_missing_dust_detection_qf
• 128US: low_confidence_nuc_detection_qf
• 64US: medium_confidence_nuc_detection_qf
• 0US: high_confidence_nuc_detection_qf
• 192US:: bad_missing_nuc_detection_qf

[Ho-ChunHuang-NOAA]

@hsoh-u Do you mean current Point2Grid (without modification) can automatically detect new threshold value for high/medium/low quality?

[hsoh-u]

No, the auto detection is not available at the previous release. It's for the next release.

[hsoh-u]

The data type is changed from ubyte to byte:
from OR_ABI-L2-ADPC-M6_G16_s20192662141196_e20192662143569_c20192662144526.nc:

   ubyte Smoke(y, x) ;
          Smoke:valid_range = 0UB, 1UB ;
   ubyte Dust(y, x) ;
          Dust:valid_range = 0UB, 1UB ;

from OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc

   byte Smoke(y, x) ;
          Smoke:valid_range = 0b, 1b ;
   byte Dust(y, x) ;
          Dust:valid_range = 0b, 1b ;

[Ho-ChunHuang-NOAA]

Thanks to find out the new changes. NESDIS did not tell us all the changes.

Do you need to read and use the values of
byte Smoke(y, x) ; and byte Dust(y, x)?

I thought all you need from the ADP files is the values read from ushort DQF(y, x)? am I correct?

[hsoh-u]

MET reads Smoke and Dust variables at ADP file and filters out if the data value is 0 (smoke_absent or dust_absent) before applying QC flags.

  Smoke:flag_values = 0UB, 1UB ;
  Smoke:flag_meanings = "smoke_absent smoke_present" ;
  Dust:flag_values = 0UB, 1UB ;
  Dust:flag_meanings = "dust_absent dust_present" ;

  Smoke:flag_values = 0b, 1b ;
  Smoke:flag_meanings = "smoke_absent smoke_present" ;
  Dust:flag_values = 0b, 1b ;
  Dust:flag_meanings = "dust_absent dust_present" ;

[Ho-ChunHuang-NOAA]

I see. Thanks for letting me know.

[jprestop]

@hsoh-u I noticed this issue was closed (likely automatically with the merge of the PR into main). I'm thinking it needs to be reopened for the work in develop, unless that's already been done. Can you please confirm and reopen if necessary?

[Ho-ChunHuang-NOAA]

Hi, Hank: I have question on your statement if #2874 if only high quality is given with-qc 0, all data will be filtered out. There should be 4 records left, with AOD QC=high + ADP QC (For smoke) = high, isn't it? Ho-Chun

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

On Fri, May 3, 2024 at 4:25 PM Julie Prestopnik ***@***.***> wrote: @hsoh-u <https://github.com/hsoh-u> I noticed this issue was closed (likely automatically with the merge of the PR into main). I'm thinking it needs to be reopened for the work in develop, unless that's already been done. Can you please confirm and reopen if necessary? — Reply to this email directly, view it on GitHub <#2853 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3HOTTEYUVLH54GYQ3DZAPXBZAVCNFSM6AAAAABF63TCU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJTG4YTGNZQGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hsoh-u]

No, it's before applying AOD QC. I will add the count of adjusted confidences for the MET v12 release.

The comment #2874 is for unit test case:

• input AOD and ADP files from you:
  • 20240419/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc
  • 20240419/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc
• -field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0,1

The debug messages

DEBUG 4:    AOD QC: high=8092 medium=6910, low=47149, no_retrieval=0
DEBUG 4:    ADP QC: high=4 medium=87, low=62050, no_retrieval=10, adjusted=47142

• AOD QC values:
  • The high confidence count: 8092
  • The medium confidence count: 6910
  • The low confidence count: 47149
• ADP Smoke QC values as raw data:
  • The high confidence count: 4
  • The medium confidence count: 87
  • The low confidence count: 62050
  • adjusted: 47142 confidence were reduced (from high to medium/low or from medium to low).

With two inputs above, all high confidences were reduced to medium or low confidence.

[hsoh-u]

Oops, the adjusted number is not correct (the original low count was added accidentally) . I will correct it.

[Ho-ChunHuang-NOAA]

Hi, Hank: I thought the logic should be from the user's guide. with -adp option and for Amoke_AOD and Dust_AOD; 0 (high only) : [high AOD QC + High ADP QC] 1 (medium only): [high AOD QC + medium ADP QC]+[medium AOD QC + high ADP QC]+ [medium AODQC+medium ADP QC] 0,1 (high+medium): [high AOD QC + High ADP QC]+[high AOD QC + medium ADP QC]+[medium AOD QC + high ADP QC]+ [medium AODQC+medium ADP QC] You meant the sample files give you all medium quality SMoke_AOD? number of "0" options (high only) is zero but not four? Ho-Chun

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

On Fri, May 3, 2024 at 5:54 PM Howard Soh ***@***.***> wrote: Oops, the adjusted number is not correct (the original low count was added accidentally) . I will correct it. — Reply to this email directly, view it on GitHub <#2853 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3HGGLN7JEMJBGMA3KLZAQBRHAVCNFSM6AAAAABF63TCU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJTHAYTCMZYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hsoh-u]

MET follows the logic you described. It's a just log message. The log message will be changed to

DEBUG 4:    AOD QC: high=8092 medium=6910, low=47149, no_retrieval=0
DEBUG 4:    ADP QC: high=0 (4), medium=24 (87), low=62117 (62050), no_retrieval=10
DEBUG 4:    adjusted: high to medium=0, high to low=4, medium to low=63, total=67

• high=0 (4): adjusted high count = 0 (raw high count = 4)
• medium=24 (87): adjusted medium count = 24 (raw medium count = 87)
• low=62117 (62050): adjusted low count = 62117 (raw low count = 62050)

[Ho-ChunHuang-NOAA]

The numbers mean there are 4 AOD pixels with low AOD DQF read and high smoke confidence ADP DQF read, therefore, the algorithm adjusts AOD QC from HIGH to LOW (as final Smoke_AOD QC), am I correct?

[hsoh-u]

The numbers mean there are 4 AOD pixels with low AOD DQF read and high smoke confidence ADP DQF read, therefore, the algorithm adjusts AOD QC from HIGH to LOW (as final Smoke_AOD QC), am I correct?

Yes, that's correct

[hsoh-u]

@Ho-ChunHuang-NOAA The fix was merged into the main_v11.1 bugfix branch and the develop branch three weeks ago.

[Ho-ChunHuang-NOAA]

Howard, thank you for the update.