Feature 348 dmap reformat (#356)

* Update version * Per dtcenter/METplus#2336, call custom GHA to trigger METplus use case tests when changes are pushed to main_vX.Y and develop branches * Update keep_all_mpr_col to keep_all_cols now that DMAP linetype is also supported * Added test for reformatting the DMAP (grid_stat) linetype for supporting scatter plots * Test config file for reformatted DMAP data used for scatter plot * Replace keep_all_mpr_cols with keep_all_cols now that DMAP linetype can be used for scatter plots, line plots, and contour plots * update keep_all_mpr_cols to keep_all_cols * update the comment * Removed debug print statements * Added test for reformatted DMAP output for line and contour plots * DMAP sample data * Add values for DMAP reformatting * Config file for testing reformatting DMAP for line and contour plot support * Example data and reformatted data for stat_name/stat_value/confidence level reformatting * Refactored version of original. Content needed for matched pair data, TCMPR data, and aggregation statistics data * Added content for ranked data values, by all common stat+linetype-specific columns. Made some edits. * New example data * Clean up formatting and wording, added content for the TCDIAG reformatting * Examples for TCDIAG reformatting * Delete METreformat/test/data/ensemble_stat/ensemble_stat_20100101_120000V.stat no longer relevant due to additional columns introduced in one of the beta releases * Updated the data to the latest version of ECNT (includes additional columns) * Accidentally removed this input file * Added DMAP support * Updated for new ensemble stat ECNT * DMAP reformatting support * change aggregation status in config file * Update the ECNT/ensemble stat tests with latest data and make testing less reliant on hard-coding. * Add better explanation for the input_stats_aggregated setting * update counts for the ensemble stat data directory data: orank from 1426 to 1921 in new data, phist 18 from 9. * updated values for line_data_phist_bin from 180 to 340 and line_data_rhist_rank from 84 to 102 * Delete METreformat/test/data/ensemble_stat/ensemble_stat_20100101_120000V.stat no longer valid/out of date * fix numbe of line_data_phist_bin to 360 * update line_data_rhist_rank * Update ECNT aggregation reformatting test to not use hard-coded values * Remove use of named tuple now that test has been simplified * update the yaml file to use keep_all_cols and updated DMAP line type support in comment * Added reformatting information for ECNT * Example reformatted output for ECNT * Cleaned up format in 6.3.1 for easier reading. * Update docs/Users_Guide/reformat_stat_data.rst use suggested text Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst use suggested reformatting Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst use suggested formatting Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst use suggested reformatting Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst use suggested format Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst use suggested format/reformatting Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested formatting Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested formatting Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested fix Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested fix Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested fix Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested fix Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested fix Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested fix Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst apply suggested Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst applying suggested change Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/reformat_stat_data.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update reformat_stat_data.rst updated MET tool names from lower case to camel case with '-' in between the two words * Update reformat_stat_data.rst Replace text version of table references to links to the MET User's Guide * Address duplicate explicit target name in links for table 11.1, 12.1, and 13.1 * Capitalizing MET tool names * Per #348, updated spacing to attempt to fix formatting and update capitalization of MET tools * Per #348, updated spacing to attempt to fix formatting and update capitalization of MET tools * Per #348, removing additional space before literal include directive so it will render correctly in RTD * Per #348, removing additional space before literal include directive so it will render correctly in RTD * Per #348, adjusting alignment for appropriate alignment in rendering * Per #348, modified spacing and capitalization * Fixed some ambiguous text that was meant to be removed or reworded from the previous version of documentation --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]>
dtcenter · Dec 13, 2024 · 508711d · 508711d
1 parent c624251
commit 508711d
Show file tree

Hide file tree

Showing 36 changed files with 17,517 additions and 4,229 deletions.
diff --git a/METdbLoad/test/test_met_db_load.py b/METdbLoad/test/test_met_db_load.py
@@ -47,10 +47,10 @@ def assert_count_rows(cur, table, expected_count):
             ENSEMBLE_STAT_DATA_DIR,
             "ensemble_stat",
             {
-                "line_data_orank": 1426,
-                "line_data_phist_bin": 180,
-                "line_data_rhist_rank": 84,
-                "line_data_phist": 9,
+                "line_data_orank": 1921,
+                "line_data_phist_bin": 360,
+                "line_data_rhist_rank": 108,
+                "line_data_phist": 18,
             },
         ),
         (

diff --git a/METdbLoad/ush/constants.py b/METdbLoad/ush/constants.py
@@ -1319,7 +1319,7 @@
      "fcst_init_beg)"]
 
 #
-# Used by MET reformatter
+# Used by METdataio reformatter
 # 1 more than the MET User's Guide to
 # account for the FCST_INIT_BEG
 #
@@ -1336,6 +1336,7 @@
 NUM_STAT_MCTC_COLS = 28
 NUM_STAT_MCTS_COLS = 45
 NUM_STAT_MPR_COLS = 40
+NUM_STAT_DMAP_COLS =49
 # Number of columns BEFORE the "variable" fields (i.e. VERSION, MODEL, ..., LINETYPE), including the
 # FCST_INIT_BEG column .
 # For PCT the "variable" fields are N_THRESH, THRESH_i, OY_i, ON_i,..., THRESH_N, where the
@@ -1693,5 +1694,14 @@
                    'fcst_climo_mean', 'fcst_climo_stdev']
 
 MPR_SPECIFIC = [cur_stat_header.upper() for cur_stat_header in LC_MPR_SPECIFIC]
-
 MPR_HEADERS = LC_COMMON_STAT_HEADER + ['total'] + LC_MPR_SPECIFIC
+
+
+# DMAP
+LC_DMAP_SPECIFIC =  ['fy', 'oy', 'fbias', 'baddeley', 'hausdorff',
+                          'med_fo', 'med_of', 'med_min', 'med_max', 'med_mean',
+                          'fom_fo', 'fom_of', 'fom_min', 'fom_max', 'fom_mean',
+                          'zhu_fo', 'zhu_of', 'zhu_min', 'zhu_max', 'zhu_mean',
+                          'g', 'gbeta', 'beta_value']
+DMAP_SPECIFIC = [cur_stat_header.upper() for cur_stat_header in LC_DMAP_SPECIFIC]
+DMAP_HEADERS = LC_COMMON_STAT_HEADER + ['total'] + LC_DMAP_SPECIFIC
diff --git a/METreformat/reformat_stat.yaml b/METreformat/reformat_stat.yaml
@@ -16,12 +16,12 @@ log_filename: stdout
 # most verbose is info, less verbose is error
 log_level: info
 
-# Currently support FHO, CTC, CTS, CNT, SL1L2, VL1L2, PCT, MCTC, VCNT, ECNT, RHIST, TCDiag, and MPR line types
+# Currently support FHO, CTC, CTS, CNT, SL1L2, VL1L2, PCT, MCTC, VCNT, ECNT, RHIST, TCDiag, MPR, and DMAP line types
 line_type: PCT
 
 #
-# FOR MPR linetype only
+# FOR MPR or DMAP linetypes only
 #
 
 # True if reformatting for scatter plot, False otherwise
-keep_all_mpr_cols: False
+keep_all_cols: False
diff --git a/METreformat/test/ECNT_for_agg.yaml b/METreformat/test/ECNT_for_agg.yaml
@@ -1,4 +1,4 @@
-input_stats_aggregated: True
+input_stats_aggregated: False
 output_dir: ../output
 output_filename: ensemble_stat_ecnt_for_agg.data
 line_type: ECNT

diff --git a/METreformat/test/data/ensemble_stat/ensemble_stat_20100101_120000V.stat b/METreformat/test/data/ensemble_stat/ensemble_stat_20100101_120000V.stat
diff --git a/METreformat/test/data/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V.stat b/METreformat/test/data/ensemble_stat/ensemble_stat_FILE_LIST_20120410_120000V.stat
diff --git a/METreformat/test/data/grid_stat/dmap/grid_stat_120000L_20050807_120000V.stat b/METreformat/test/data/grid_stat/dmap/grid_stat_120000L_20050807_120000V.stat
diff --git a/METreformat/test/data/grid_stat/dmap/grid_stat_APCP_12_120000L_20050807_120000V.stat b/METreformat/test/data/grid_stat/dmap/grid_stat_APCP_12_120000L_20050807_120000V.stat
diff --git a/METreformat/test/data/grid_stat/dmap/grid_stat_APCP_24_240000L_20050808_000000V.stat b/METreformat/test/data/grid_stat/dmap/grid_stat_APCP_24_240000L_20050808_000000V.stat
diff --git a/METreformat/test/data/grid_stat/dmap/grid_stat_FV3_core_lsm1_010000L_20190521_010000V.stat b/METreformat/test/data/grid_stat/dmap/grid_stat_FV3_core_lsm1_010000L_20190521_010000V.stat
@@ -0,0 +1,3 @@
+VERSION MODEL    DESC FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD OBS_VALID_BEG   OBS_VALID_END   FCST_VAR     FCST_UNITS FCST_LEV OBS_VAR     OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD  INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE
+V11.1.0 FV3_core NA   010000    20190521_010000 20190521_010000 000000   20190521_010000 20190521_010000 fcst_obj_raw NA         *,*      obs_obj_raw NA        *,*     GOES   FULL    NEAREST      1           <999        <999       NA         NA    DMAP      1814400 64663 123297     0.52445      59.29468 181.17671 10.80896 0.83382 0.83382 10.80896 5.82139 0.56002 0.49584 0.49584 0.56002 0.52793 5.5015 0.51393 0.51393 5.5015 3.00771 4558.55938 0.94245 1646023680000
+V11.1.0 FV3_core NA   010000    20190521_010000 20190521_010000 000000   20190521_010000 20190521_010000 fcst_obj_raw NA         *,*      obs_obj_raw NA        *,*     GOES   FULL    NBRHD_SQUARE 1           <999        <999       >=0.5      NA    NBRCTC    1814400 59822   4841 63475       1686262
diff --git a/METreformat/test/data/grid_stat/dmap/grid_stat_FV3_core_lsm1_020000L_20190521_020000V.stat b/METreformat/test/data/grid_stat/dmap/grid_stat_FV3_core_lsm1_020000L_20190521_020000V.stat
@@ -0,0 +1,3 @@
+VERSION MODEL    DESC FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD OBS_VALID_BEG   OBS_VALID_END   FCST_VAR     FCST_UNITS FCST_LEV OBS_VAR     OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD  INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE
+V11.1.0 FV3_core NA   020000    20190521_020000 20190521_020000 000000   20190521_020000 20190521_020000 fcst_obj_raw NA         *,*      obs_obj_raw NA        *,*     GOES   FULL    NEAREST      1           <999        <999       NA         NA    DMAP      1814400 122478 124217     0.986      37.25676 95.90099 4.91174 4.83528 4.83528 4.91174 4.87351 0.75398 0.75347 0.75347 0.75398 0.75372 2.55647 2.51824 2.51824 2.55647 2.53735 4453.14484 0.94635 1646023680000
+V11.1.0 FV3_core NA   020000    20190521_020000 20190521_020000 000000   20190521_020000 20190521_020000 fcst_obj_raw NA         *,*      obs_obj_raw NA        *,*     GOES   FULL    NBRHD_SQUARE 1           <999        <999       >=0.5      NA    NBRCTC    1814400  86624  35854 37593     1654329
diff --git a/METreformat/test/data/grid_stat/dmap/grid_stat_FV3_core_mp1_010000L_20190521_010000V.stat b/METreformat/test/data/grid_stat/dmap/grid_stat_FV3_core_mp1_010000L_20190521_010000V.stat
@@ -0,0 +1,3 @@
+VERSION MODEL    DESC FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD OBS_VALID_BEG   OBS_VALID_END   FCST_VAR     FCST_UNITS FCST_LEV OBS_VAR     OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD  INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE
+V11.1.0 FV3_core NA   010000    20190521_010000 20190521_010000 000000   20190521_010000 20190521_010000 fcst_obj_raw NA         *,*      obs_obj_raw NA        *,*     GOES   FULL    NEAREST      1           <999        <999       NA         NA    DMAP      1814400 42968 123297     0.34849      88.36694 246.28845 23.97453 0.61461 0.61461 23.97453 12.29457 0.3947 0.33924 0.33924 0.3947 0.36697 12.09423 0.41427 0.41427 12.09423 6.25425 6279.69715 0.84955 1646023680000
+V11.1.0 FV3_core NA   010000    20190521_010000 20190521_010000 000000   20190521_010000 20190521_010000 fcst_obj_raw NA         *,*      obs_obj_raw NA        *,*     GOES   FULL    NBRHD_SQUARE 1           <999        <999       >=0.5      NA    NBRCTC    1814400 41616   1352 81681       1689751
diff --git a/METreformat/test/data/grid_stat/dmap/grid_stat_FV3_core_mp1_020000L_20190521_020000V.stat b/METreformat/test/data/grid_stat/dmap/grid_stat_FV3_core_mp1_020000L_20190521_020000V.stat
@@ -0,0 +1,3 @@
+VERSION MODEL    DESC FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD OBS_VALID_BEG   OBS_VALID_END   FCST_VAR     FCST_UNITS FCST_LEV OBS_VAR     OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD  INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE
+V11.1.0 FV3_core NA   020000    20190521_020000 20190521_020000 000000   20190521_020000 20190521_020000 fcst_obj_raw NA         *,*      obs_obj_raw NA        *,*     GOES   FULL    NEAREST      1           <999        <999       NA         NA    DMAP      1814400 99988 124217     0.80495      35.7628 108.22661 7.25606 2.82767 2.82767 7.25606 5.04186 0.68722 0.66673 0.66673 0.68722 0.67697 3.72527 1.51107 1.51107 3.72527 2.61817 4331.2607 0.95064 1646023680000
+V11.1.0 FV3_core NA   020000    20190521_020000 20190521_020000 000000   20190521_020000 20190521_020000 fcst_obj_raw NA         *,*      obs_obj_raw NA        *,*     GOES   FULL    NBRHD_SQUARE 1           <999        <999       >=0.5      NA    NBRCTC    1814400 77791  22197 46426       1667986