Semi-implicit barotropic mode solver updates

[hyungyukang]

This PR updates the semi-implicit barotropic mode solver :

• Restructure and make codes simpler in the main time-stepping code
• Make subroutines for the semi-implicit solver (preconditioining, matrix-vector multiplications, global reductions)
• Provide the s-step conjugate gradient method when config_btr_si_partition_match_mode is turned on to provide a faster symmetric iterative solver
• Small changes in stage 1 and 3 to match the split-explicit code

Below is PASS/FAIL test results:

• With default options
  FAIL PEM_Ln9_D.T62_oQU240.GMPAS-IAF.cori-knl_intel (FAIL is expected for PEM tests.)
  PASS PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel
  PASS SMS.T62_oEC60to30v3wLI.GMPAS-DIB-IAF-ISMF.cori-knl_intel
  PASS SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.cori-knl_intel

• With config_btr_si_partition_match_mode=.true.
  PASS PEM_Ln9_D.T62_oQU240.GMPAS-IAF.cori-knl_intel
  PASS PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel
  PASS SMS.T62_oEC60to30v3wLI.GMPAS-DIB-IAF-ISMF.cori-knl_intel
  PASS SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.cori-knl_intel

[jonbob]

@hyungyukang - why is FAIL is expected for PEM tests?

[philipwjones]

@jonbob If the config_btr_si_partition_match_mode flag is false, the implicit solver uses a more optimal preconditioner that depends on the partitioning so won't be bfb (and uses faster non-reproducible sums as well). When the flag is true, both reproducible sums and a reproducible preconditioner are used that will guarantee a passing PEM test but with reduced performance. This was done when the SI solver was first introduced so this behavior hasn't changed for this PR, other than using the newer repro sums.

[jonbob]

Thanks @philipwjones - I appreciate the clarification, since I misread it and assumed the default settings meant generic E3SM settings which need to be BFB

[hyungyukang]

@philipwjones , Thanks a lot for your detailed explanation! Nothing to add more.
@jonbob, Sorry for confusing. I should've added reasons for the fail-expected tests. Thanks.

[philipwjones]

Visual review of changes looks good.

Passes compass PR test when option is off so is bfb as a stealth feature. Unfortunately, I forgot to build compass with LAPACK on so wasn't able to test with the option on.

It runs in a QU240 case on Summit with CPU only. It fails with OpenACC on Summit, but that's because some of our recent changes in array transfers weren't propagated to the si driver. So that's on us and I will open an issue on that and fix later.

Since I'm off on vacation for a bit, I will go ahead an approve based on my code review and the testing to date so I don't hold this up.

[hyungyukang]

@philipwjones , Thanks for reviewing and approving this PR. Next PR will be GPU porting of the semi-implicit solver, so maybe I can look into that error.

[mark-petersen]

I compiled and ran, with this feature off, with intel and gnu, debug and optimized and the nightly suite compares bfb with previous. Reviewed by visual inspection.

[hyungyukang]

I compiled and ran, with this feature off, with intel and gnu, debug and optimized and the nightly suite compares bfb with previous. Reviewed by visual inspection.

@mark-petersen , Thanks Mark for testing and reviewing!

[jonbob]

passes:

• ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel
• SMS_D_Ld3.T62_oQU120.CMPASO-IAF.chrysalis_intel
• SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel
• PEM_Ln9.ne30pg2_EC30to60E2r2.WCYCL1850.chrysalis_intel

merged to next

[jonbob]

merged to master