Merge remote-tracking branch 'origin/worleyph/cam/twin_alg_init_opt' …

…into next PR #1347

When load balancing physics computation on a latlon mesh,
there is an advantage to assigning columns to chunks in pairs,
pairing each column with its 'twin' in the other hemisphere,
latitude reflected around the equator, and offset 180 degrees
in longitude. While a similar approach should be advantageous,
or at least no worse than no doing it, with a non-latlon mesh,
the cost of initialization of this 'twin' algorithm was too high
for the HOMME spectral element mesh to be acceptable even though
only part of the initialization cost. Because of this, the twin
algorithm is disabled by default when using a non-latlon mesh.
(For a latlon mesh, the twin algorithm is enabled by default.)
Note that a runtime parameter can be used to override the default.

A simple change to the initialization algortihm drops the cost
to an acceptable level, less than 2 minutes even on one core of
Mira and for an ne120 HOMME mesh. In performance experiments, using
the twin algorithm has not improved the load balance compared to
not using it for current ACME production-like cases, and the default
disabling the twin algorithm is not being changed. However, this
optimization to the initialization algorithm will make it feasible
to re-examine this option in the future.

This change will not be tested by the ACME test suites, as the
twin algorithm is disabled, and it is also irrelevant unless
atmospheric physics load balancing is enabled.

Experiments with F compsets and the CAM-SE dycore for both
ne30 and ne120 meshes have been conducted, indicating that the
same load balancing redistribution of columns is computed by
the twin algorithm with and without this optimization. Tests with
FV and Spectral Eulerian dycores have not been conducted.

[BFB]

components/cam/src/physics/cam/phys_grid.F90

@@ -144,6 +144,9 @@ module phys_grid
 
 
    integer, dimension(:), allocatable, private :: clon_p_cnt ! number of repeats for each longitude
+   integer, dimension(:), allocatable, private :: clon_p_idx ! index in lonlat ordering for first 
+                                                             ! occurrence of longitude corresponding to 
+                                                             ! given latitude index
    real(r8), dimension(:), allocatable :: clon_p  ! unique longitudes (radians, increasing)
 
    integer, dimension(:), allocatable, private :: lat_p      ! index into list of unique column latitudes
@@ -703,6 +706,17 @@ subroutine phys_grid_init( )
        ! Allocate and initialize chunks data structure, then
        ! assign chunks to processes.
        !
+       if  (twin_alg .eq. 1) then
+          ! precompute clon_p_idx: index in lonlat ordering for first 
+          ! occurrence of longitude corresponding to given latitude index,
+          ! used in twin option in create_chunks; used in create_chunks
+          allocate( clon_p_idx(1:clon_p_tot) )
+          clon_p_idx(1) = 1
+          do i=2,clon_p_tot
+             clon_p_idx(i) = clon_p_idx(i-1) + clon_p_cnt(i-1)
+          enddo
+       endif
+
        call create_chunks(lbal_opt, chunks_per_thread)
 
        ! Early clean-up, to minimize memory high water mark
@@ -711,6 +725,7 @@ subroutine phys_grid_init( )
        deallocate( latlon_to_dyn_gcol_map )
        if  (twin_alg .eq. 1) deallocate( lonlat_to_dyn_gcol_map )
        if  (twin_alg .eq. 1) deallocate( clon_p_cnt )
+       if  (twin_alg .eq. 1) deallocate( clon_p_idx )
        if ((twin_alg .eq. 1) .or. (lbal_opt .eq. 3)) deallocate( clat_p_cnt )
 
        !
@@ -4617,22 +4632,13 @@ subroutine find_twin(gcol, smp, proc_smp_mapx, twingcol_f)
    integer :: ibtwin(npes)               ! global column indices
    integer :: twinproc, twinsmp          ! process and smp ids
 
-   integer :: clon_p_idx(clon_p_tot)     ! index in lonlat ordering for first 
-                                         !  occurrence of longitude corresponding to 
-                                         !  given latitude index
-
    real(r8):: twopi                      ! 2*pi
    real(r8):: clat, twinclat             ! latitude and twin
    real(r8):: clon, twinclon             ! longitude and twin
 
 !-----------------------------------------------------------------------
    twingcol_f = -1
 
-   ! precompute clon_p_idx
-   clon_p_idx(1) = 1
-   do i=2,clon_p_tot
-      clon_p_idx(i) = clon_p_idx(i-1) + clon_p_cnt(i-1)
-   enddo
 !
 ! Try day/night and north/south hemisphere complement first
 !