add in-flight icing products through libIFI

[SamuelTrahanNOAA]

Description

This adds the latest version of the In-Flight Icing products via a new libIFI library. Specifically, the latest UCAR FIP2 code was rewritten as a library, libIFI, that can be linked to the post. The PR is still a draft because we haven't installed the library on WCOSS2 yet, the only platform where this functionality will be supported. That restriction is for copyright and licensing reasons. (NOAA wrote libIFI, but it is a derived work from closed-source software, so the original creators retain the right to it.)

Issue and Related PRs

Fixes #537

ufs-weather-model PR = ufs-community/ufs-weather-model#1361

fv3atm PR = NOAA-EMC/fv3atm#568

The New Fields

All of the code to calculate the new fields is in the libIFI library, but there is some "glue code" to connect it to UPP, mostly in IFI.F. Inputs are native model data, CAPE, CIN, surface height, and hourly accumulated precipitation.

Output fields are:

1. ICE_PROB = icing probability on flight levels, every 500 feet from 500 to 30000
2. ICE_SEV = icing severity category (thresholds of ICE_PROB), every 500 feet from 500 to 30000
3. SLD = supercooled liquid droplet concentration, every 500 feet from 500 to 30000

These fields already have standard GRIB2 headers understood by g2 and g2tmpl, so there are no changes in those libraries.

Users request the fields like so:

   <param>
       <post_avblfldidx>1100</post_avblfldidx>
       <shortname>ICE_PROB_IFI_FLIGHT_LEVEL</shortname>
       <pname>ICPRB</pname>
       <table_info>NCEP</table_info>
       <fixed_sfc1_type>spec_alt_above_mean_sea_lvl</fixed_sfc1_type>
       <scale>4.0</scale>
       <!-- Level units are in 0.1 feet, so final output will be in units of 0.1 meters -->
       <scale_fact_fixed_sfc1>1</scale_fact_fixed_sfc1>
       <level>5000. 10000. 15000. 20000. 25000. 30000. 35000. 40000. 45000. 50000. 55000. 60000. 65000. 70000. 75000. 80000. 85000. 90000. 95000. 100000. 105000. 110000. 115000. 120000. 125000. 130000. 135000. 140000. 145000. 150000. 155000. 160000. 165000. 170000. 175000. 180000. 185000. 190000. 195000. 200000. 205000. 210000. 215000. 220000. 225000. 230000. 235000. 240000. 245000. 250000. 255000. 260000. 265000. 270000. 275000. 280000. 285000. 290000. 295000. 300000.</level>
    </param>

    <param>
       <post_avblfldidx>1101</post_avblfldidx>
       <shortname>SLD_IFI_FLIGHT_LEVEL</shortname>
       <pname>SIPD</pname>
       <table_info>NCEP</table_info>
       <fixed_sfc1_type>spec_alt_above_mean_sea_lvl</fixed_sfc1_type>
       <scale>4.0</scale>
       <!-- Level units are in 0.1 feet, so final output will be in units of 0.1 meters -->
       <scale_fact_fixed_sfc1>1</scale_fact_fixed_sfc1>
       <level>5000. 10000. 15000. 20000. 25000. 30000. 35000. 40000. 45000. 50000. 55000. 60000. 65000. 70000. 75000. 80000. 85000. 90000. 95000. 100000. 105000. 110000. 115000. 120000. 125000. 130000. 135000. 140000. 145000. 150000. 155000. 160000. 165000. 170000. 175000. 180000. 185000. 190000. 195000. 200000. 205000. 210000. 215000. 220000. 225000. 230000. 235000. 240000. 245000. 250000. 255000. 260000. 265000. 270000. 275000. 280000. 285000. 290000. 295000. 300000.</level>
    </param>

    <param>
       <post_avblfldidx>1102</post_avblfldidx>
       <shortname>ICE_SEV_CAT_IFI_FLIGHT_LEVEL</shortname>
       <pname>ICSEV</pname>
       <table_info>NCEP</table_info>
       <fixed_sfc1_type>spec_alt_above_mean_sea_lvl</fixed_sfc1_type>
       <scale>4.0</scale>
       <!-- Level units are in 0.1 feet, so final output will be in units of 0.1 meters -->
       <scale_fact_fixed_sfc1>1</scale_fact_fixed_sfc1>
       <level>5000. 10000. 15000. 20000. 25000. 30000. 35000. 40000. 45000. 50000. 55000. 60000. 65000. 70000. 75000. 80000. 85000. 90000. 95000. 100000. 105000. 110000. 115000. 120000. 125000. 130000. 135000. 140000. 145000. 150000. 155000. 160000. 165000. 170000. 175000. 180000. 185000. 190000. 195000. 200000. 205000. 210000. 215000. 220000. 225000. 230000. 235000. 240000. 245000. 250000. 255000. 260000. 265000. 270000. 275000. 280000. 285000. 290000. 295000. 300000.</level>
    </param>

The grib2_module automatically converts from tenths of a foot to meters.

MPI and OpenMP

IFI's algorithm requires an unusual smoother that cannot be parallelized in the i-j direction, so it is parallelized in the k direction instead. That requires a gather and a scatter. The library does not implement this itself because it has no knowledge of MPI. Instead, the IFI.F in UPP provides the gather and scatter. Halo communication comes from EXCH_c_float.F.

The libIFI internally has its own OpenMP directives. It is fully OpenMP parallelized, even when running its own test programs.

New MPI Routines in UPP

1. EXCH_c_float (EXCH_c_float.f) = same as EXCH but with real kind c_float.
2. global_gather (IFI.F) = gathers (ista:iend,jsta:jend) regions from each MPI rank to an (im,jm) array on any one compute rank specified in the global_rank argument to the function
3. global_scatter (IFI.F) = opposite of global_scatter: scatters an (im,jm) array from one compute rank to (ista:iend,jsta:jend) arrays on each compute rank

The gather is done first along rows and then columns, in the row and column of the global_rank. This way, the computational cost of rearranging data is split across each rank in the destination column. It also avoids having to rearrange data at the destination rank. Here's an example:

  1 2 3 4 5 6 7
1 - - | - - - -
2 - - G - - - -
3 - - | - - - -
4 - - | - - - -

The global_rank G is at x=3, y=2, which wants to gather from the rest of the grid. In each of rows 1 through 4, there is a gather to column 3 in that row, labeled with |. Then all ranks in column 3 do a gather to x=3, y=2. The data is already in the correct order in memory when it reaches the global_rank G, as it was in gathers in the old post without x decomposition. Hence, we avoid the expensive computational cost of rearranging the full grid data on the global_rank. The global_rank still needs to do a one-row data rearrangement though, for its row.

A global_scatter uses the same process in reverse: scattering across column 3, and then across each row from the rank in column 3. Like with the gather, the global_rank G does no full grid computation since the data is already in the order needed for the scatter. The only computation is in the ranks in column 3, which must rearrange data for the scatter in the row direction. (That includes the global_rank.)

To handle halo regions, the code has a temporary (ista:iend,jsta:jend) array on each rank for communication. A global_gather copies the data into the array before communication, and the scatter does so afterwards. The caller provides the I and J bounds of the input array, so the routine can take inputs with different halo sizes (or no halos). There is no temporary (im,jm) array since the caller provides one already.

With only one column, the process trivially reduces to some array copies and a single gather or scatter.

Testing

There are several regression test cases for libIFI standalone (C++ API, Fortran API, and configuration), three for UPP (HRRR WRF, RRFS FV3, global FV3), and the inline post (RRFS FV3, global FV3). It is easy to setup regression test cases since you only need to add the requested fields.

The libIFI library can be compiled with a -DIFI_DEBUG option which will enable bounds checking, and many other safeguards, inside the code. This can be enabled even with optimization turned on. There's also a -DDEBUG which turns off optimization and enables additional compiler-originated checks. Neither of these should be enabled on operational code, but they may assist in debugging if there's strangeness in the output products.

Licensing

The UPP changes are freely available as anything else in UPP, but the libIFI library has restrictive licensing. It was created by NOAA (specifically, @SamuelTrahanNOAA) based on the proprietary UCAR FIP2 library. Thus it is considered a derived work, and is distributed under the UCAR Government Use License. This restricts it to government machines, and it will be available on an as-needed basis.

Presently, the plan is for all users on WCOSS2 to be able to access the compiled library, and a small subset of the users on Jet and Hera under a new "ifi" project. The source code will be available only to a very small number of users, in a private DTC repository called "UPP_IFI" (where it resides now). On disk, it will only be available in compiled form, and only shared (.so) versions of the library, so people don't accidentally distribute executables that contain libIFI. This way, you won't have to protect any UPP or UFS you compile with libIFI, since nobody can run it without access to the library.

[WenMeng-NOAA]

@SamuelTrahanNOAA Could you add change log with your name in the fortran code files you modified/added? You might refer to change log format in sorc/ncep_post.fd/ALLOCATE_ALL.f, such as:

!! -  22-11-08  Kai Wang - Replace acfcmaq_on with aqf_on

[SamuelTrahanNOAA]

I have updated the change logs.

[WenMeng-NOAA]

@hu5970 and @EricJames-NOAA This PR is in a good shape for merging process. Please let me know if you have more comments from the GSL side.

[WenMeng-NOAA]

@SamuelTrahanNOAA It seems to me you have a new commit at the IFI repository after 01/18/2023. Do you think rebuilding the ifi lib on Hera and Jet is needed?

[SamuelTrahanNOAA]

It seems to me you have a new commit at the IFI repository after 01/18/2023. Do you think rebuilding the ifi lib on Hera and Jet is needed?

No, the library did not change. I only changed one of the test programs.

[EricJames-NOAA]

@WenMeng-NOAA it looks good to me! Once this PR is merged, I will open my PR to resolve remaining differences between EMC develop and GSL RRFS_dev.

[WenMeng-NOAA]

@SamuelTrahanNOAA and @EricJames-NOAA Thanks for clarifying.

[hu5970]

@WenMeng-NOAA I do not have further comments. Thanks, Ming

[WenMeng-NOAA]

The UPP RT tests were completed on WCOSS2, Hera and Orion. No baseline change is needed.