Speedup MPICH builds

[vicentebolea]

Currently the test stage of the MPICH builds takes over 40mins to complete this is unacceptable.

• Significantly reduce test exec time.

[scottwittenburg]

I scraped the CDash results from some recent runs on master (scraped because "download results as csv" is broken), and wrote some python to generate a csv comparing the test timings. The two sets of timings I compared were from master_ubuntu20.04-gcc10-ompi and master_ubuntu20.04-gcc10-mpich. I've attached the timings comparison csv, and pasted below the top offenders.

Top Offenders

These are the tests where tests run in the mpich build took longest compared to tests run in the open mpi build:

test_name,ompi_time,mpich_time,delta
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc23DSel/*.BP5.MPI,1.78,43.86,42.08
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc21DSel/*.BP5.MPI,1.55,42.81,41.260000000000005
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc21D/*.BP5.MPI,1.56,42.82,41.26
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc23D/*.BP5.MPI,1.62,42.66,41.04
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc22D/*.BP5.MPI,1.77,42.74,40.97
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc22DSel/*.BP5.MPI,1.77,42.47,40.699999999999996
Engine.BP.*/BPStepsFileGlobalArrayParameters.EveryOtherStep/*.BP5.MPI,0.56,21.93,21.37
Engine.BP.*/BPAppendAfterStepsP.Test/*.BP5.MPI,0.45,19.07,18.62
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc23D/*.BP3.MPI,1.8199999999999998,18.5,16.68
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc22DSel/*.BP4.MPI,1.15,17.78,16.630000000000003
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc22D/*.BP4.MPI,1.44,17.7,16.259999999999998
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc21DSel/*.BP4.MPI,1.26,17.41,16.15
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc21D/*.BP4.MPI,1.21,17.26,16.05
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc23D/*.BP4.MPI,1.43,17.47,16.04
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc21DSel/*.BP3.MPI,1.79,17.82,16.03
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc23DSel/*.BP3.MPI,1.35,17.2,15.85
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc23DSel/*.BP4.MPI,1.21,17.02,15.809999999999999
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc22D/*.BP3.MPI,1.9300000000000002,17.48,15.55
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc22DSel/*.BP3.MPI,1.58,17.08,15.499999999999998
Engine.BP.*/BPWriteReadBlosc2.ADIOS2BPWriteReadBlosc21D/*.BP3.MPI,1.51,16.84,15.33
Engine.BP.*/BPParameterSelectStepsP.Stream/*.BP5.MPI,0.34,14.36,14.02
Engine.BP.*/BPChangingShapeWithinStep.MultiBlock/*.BP5.MPI,5.33,18.88,13.549999999999999
Engine.BP.*/BPParameterSelectStepsP.Read/*.BP5.MPI,0.29,13.28,12.99
Engine.MHS.MhsEngineTest.TestMhsMultiReader.MPI,0.7,13.29,12.59
Staging.3x5EarlyExit.BP5,0.48,12.68,12.2
Engine.MHS.MhsEngineTest.TestMhsMultiRank.MPI,0.73,12.76,12.03
Staging.3x5LockGeometry.BP5,1.13,12.48,11.350000000000001
Engine.SSC.SscEngineTest.TestSscString.MPI,0.4,11.64,11.24
Engine.SSC.SscEngineTest.TestSsc3DMemSelect.MPI,0.35,11.38,11.030000000000001
Engine.BP.*/BPWriteReadMGARD.BPWRMGARD2D/*.BP5.MPI,6.95,17.66,10.71
Performance.*/TestManyVars.DontRedefineVars/*.MPI,0.55,10.58,10.03
Engine.SSC.SscEngineTest.TestSscVaryingSteps.MPI,0.74,10.45,9.709999999999999
Engine.BP.*/BPWriteReadMGARD.BPWRMGARDSel2D/*.BP4.MPI,7.48,16.98,9.5
Engine.BP.*/BPStepsFileLocalArrayParameters.EveryOtherStep/*.BP5.MPI,0.43,9.18,8.75
Staging.1x1LockGeometry.CommMin.BP5.SST,0.69,9.44,8.75
Staging.1x1LockGeometry.CommMin.BP.SST,0.85,9.31,8.46
Engine.BP.*/BPStepsFileGlobalArrayReaders.EveryStep/*.BP5.MPI,0.35,8.61,8.26
Engine.BP.*/BPStepsFileGlobalArrayReaders.NewVarPerStep/*.BP5.MPI,0.36,8.56,8.200000000000001
Bindings.C.*/BPWriteAggregateReadLocalTest.Aggregate1D/*.MPI,0.28,8.07,7.79
Engine.BP.*/BPWriteReadMGARD.BPWRMGARD1D/*.BP4.MPI,13.64,21.3,7.66

Full results downloadable below.

output.csv

[vicentebolea]

Fixed by #3883