Running tests and interpreting results towards establishing a baseline for regression testing

[jr3cermak]

I've been successful at running a couple of test problems on two types of Amazon EC2 instances. I get consistent test results between the two instances. I can also obtain consistent results using the wiki instructions and the Makefile. For one problem, I get different results if I choose a different number of processors. The test case that I am currently working with is ocean_only/double_gyre. The results were initially different between the Makefile and the wiki instructions. For the wiki I chose 4 for number of processors and in the Makefile the default is 8 processors -- that generated different results. Updating the Makefile to 4 and re-running obtains identical results. From this point on, I will just refer to using the instructions on the wiki.

My goal is to establish some sort of baseline for regression testing. I'd also be interested in automatically testing the 2x2 matrix of static vs. dynamic and symmetric vs. non-symmetric. No matter what mode, we should theoretically arrive at the same result? Will there be a different baseline/checksum obtained for different compiler/cpu?

First, lets look at instance details & compilers. I've added a local Ubuntu system (home laptop).

A.
c4.large 2 vCPU 3.75Gb RAM
High frequency Intel Xeon E5-2666 v3 (Haswell) processors optimized specifically for EC2
Fedora Core 21
Compiler: gcc version 4.9.2 20150212 (Red Hat 4.9.2-6) (GCC)
OpenMPI: 1.8.3
Netcdf-fortran-openmpi: 4.2

B.
m4.xlarge 4 vCPU 16Gb RAM
2.4 GHz Intel Xeon® E5-2676 v3 (Haswell) processors
Fedora Core 23
Compiler: gcc version 5.3.1 20160406 (Red Hat 5.3.1-6) (GCC)
OpenMPI: 1.8.8
Netcdf-fortran-openmpi: 4.3.3.1

C.
Laptop 4 CPU 8 Gb RAM
Intel(R) Core(TM) i3-2310M CPU @ 2.10GHz
Ubuntu 14.04
Compiler: gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1)
OpenMPI: 1.6.5
Netcdf: 4.1.3

I have carefully run tests with 4 and 8 processors. The results across A, B and C are consistent, but the results differ between 4 and 8 processors/threads. Is this reasonable to see a change between number of processors? Would this be considered our baseline for running this test?

4 processors:
(rm Depth_list.nc; . ./../../build/gnu/env ; mpirun -np 4 ../../build/gnu/ocean_only/repro/MOM6)

MOM Day 0.000 0: En 1.424331E-13, MaxCFL 0.00000, Mass 5.288178268008E+18 Total Energy: 4126FC775E10DF73 7.5321168372247962E+05 Total Mass: 5.2881782680077681E+18, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00)
MOM Day 1.000 72: En 4.580432E-06, MaxCFL 0.00023, Mass 5.288178268008E+18 Total Energy: 42B607A80A851DA8 2.4222139843869656E+13 Total Mass: 5.2881782680077681E+18, Change: -1.7202096100332227E+01 Error: -1.72021E+01 (-3.3E-18) ....

8 processors:
(rm Depth_list.nc; . ./../../build/gnu/env ; mpirun -np 8 ../../build/gnu/ocean_only/repro/MOM6)

MOM Day 0.000 0: En 1.424361E-13, MaxCFL 0.00000, Mass 5.288178268008E+18 Total Energy: 4126FC96B8ADCFE7 7.5322736070108123E+05 Total Mass: 5.2881782680077681E+18, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00)
MOM Day 1.000 72: En 4.580432E-06, MaxCFL 0.00023, Mass 5.288178268008E+18 Total Energy: 42B607A80A852EA4 2.4222139843886641E+13 Total Mass: 5.2881782680077681E+18, Change: -1.7202096100332227E+01 Error: -1.72021E+01 (-3.3E-18) ....

[jr3cermak]

I spent most of the day trying other examples and trying to break them using different amounts of processors. They reproduce the same results no matter how many processes I set for them and the results are the same across the platforms. The two other examples I have been working with are: ocean_only/Phillips_2layer and ocean_only/DOME.

For the ocean_only/Phillips_2layer, checking the difference in the ocean.stats vs ocean.stats.gnu, the only difference is in the Frac Mass Err.

ocean.stats.gnu (from github):

  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean sea level,   Total Mass,    Frac Mass Err
            [days]                 [m2 s-2]           [Nondim]        [m]             [kg]           [Nondim]
     0,       0.000,     0, En 3.336196516993E-01, CFL  0.01092, SL -0.0000E+00, Mass 3.95904E+18, Me  0.00E+00
   144,       1.000,     0, En 3.311361662195E-01, CFL  0.01064, SL -0.0000E+00, Mass 3.95904E+18, Me  1.30E-17
   288,       2.000,     0, En 3.289611871487E-01, CFL  0.01038, SL  2.2737E-13, Mass 3.95904E+18, Me  7.13E-17
   432,       3.000,     0, En 3.270448577416E-01, CFL  0.01010, SL -0.0000E+00, Mass 3.95904E+18, Me -2.34E-17

ocean.stats (recent run):

  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean sea level,   Total Mass,    Frac Mass Err
            [days]                 [m2 s-2]           [Nondim]        [m]             [kg]           [Nondim]
     0,       0.000,     0, En 3.336196516993E-01, CFL  0.01092, SL -0.0000E+00, Mass 3.95904E+18, Me  0.00E+00
   144,       1.000,     0, En 3.311361662195E-01, CFL  0.01064, SL -0.0000E+00, Mass 3.95904E+18, Me  2.23E-17
   288,       2.000,     0, En 3.289611871487E-01, CFL  0.01038, SL  2.2737E-13, Mass 3.95904E+18, Me  7.56E-17
   432,       3.000,     0, En 3.270448577416E-01, CFL  0.01010, SL -0.0000E+00, Mass 3.95904E+18, Me -3.04E-17

[adcroft]

The numbers in ocean.stats.* are meant to be independent of processor count and layout. If they are not, there is a bug.

Unfortunately, the values are very much a function of compiler, compiler version and chipset. Testing and validating across different platforms that give different "bits" is something we've had multiple discussions about but have no good solution for (yet).

The results across A, B and C are consistent, but the results differ between 4 and 8 processors/threads.

This is confusing (to me as well as you) - I would expect the exact opposite based on what I said above. Was the 4-PE and 8-PE run on the same platform with the same executable? I just checked that 4- and 8-PEs gives the same answers as each other on my laptop (Ubuntu, gfortran 4.9.2, running under Oracle VirtualBox) for both double_gyre and Phillips. We check this routinely and and almost continuously. So I am quite confused but based on your second post I might be misreading your report.

To establish a baseline is a very good idea. The challenge is how to maintain them. For example, we just had a pair of bugs reported to do with vertical viscosity that will change a lot of the answers and we will update ocean.stats.* when we fix the bugs. But we can only do that on our own platform and you'll have to "notice" that the answers changed and regenerate some new ocean.stats of your own. Coordinating this is challenging but we do need to do something.

Re: second post

I spent most of the day trying other examples and trying to break them using different amounts of processors. They reproduce the same results no matter how many processes I set for them and the results are the same across the platforms.

Good. That is what we like to see - I wish things were as unbreakable here!

This second post suggests you do have layout/count independent results afterall?

[jr3cermak]

In all cases, I use the same executable for each platform. I noticed for the double_gyre, you need to make sure the Depth_list.nc is deleted between runs or you will amazingly get the same checksums. Rerun the 4-PE and 8-PE tests and I think you will get different results.

How much cleanup is required between runs? This is what I did just recently.

(rm -rf RESTART;mkdir RESTART;rm *.nc;. ./../../build/gnu/env ; mpirun -np 4 ../../build/gnu/ocean_only/repro/MOM6)

(rm -rf RESTART;mkdir RESTART;rm *.nc;. ./../../build/gnu/env ; mpirun -np 8 ../../build/gnu/ocean_only/repro/MOM6)

The pattern is weird. For the 4-PE, I can produce the same checksums across the platforms. For the 8-PE test for double-gyre, I get different checksums, but these different ones match across platforms.

Actually, let me attach a console log of two runs with 4-PE and 8-PE with cleared out files. The third run is the 4-PE without clearing files out. You will see that I get the 8-PE checksums for the third 4-PE run when they were clearly different in the first run.

4vs8PE.txt

[adcroft]

Did you hear the forehead slaps coming from New Jersey?

The Depth_list.nc file is a sorted list which involves a cumulative sum. It is used in calculating the APE (available potential energy) of the model which is included in the "Energy" metric we've been looking at. The cumulative sum is order-of-operations dependent at the round-off level and so we get different least-significant bits depending on how many cores are used. When we've been regression testing we've been running the the model first with a particular core count and then using the same Depth_list.nc file thereafter.

Ideally we need a way to calculate the sorted list and cumulative sums in a way that is independent of core count. One way could be to generate this file with a 1-core run and then use that file for all tests but 1-core runs are impractical. Alternatively, since we already have reproducing sums, @Hallberg-NOAA thinks we should be able to implement a reproducing cumulative sum.

In the meantime, I'm also wondering if it would be better to not use APE in the regression metrics. APE is useful for looking at solutions but this core count dependence gotcha reminds me we need to be very careful about selecting metrics.

[adcroft]

It appears the suggestion of removing APE from the Energy metric prompted development of a solution. When you see the fix (shortly), be sure to remove exisiting Depth_list.nc files. Thereafter, their presence of absence will not matter...

[Hallberg-NOAA]

This has been corrected by revising the code that created the ordered list of depths and areas. It is now assured to generate
the same list as if it had been created on 1 PE. Previously the code concatenated the lists of the PEs in order before sorting them by depth, which could lead to changes in order when multiple cells with different areas had exactly the same depth. For the PE counts that we had been testing regularly, the lists just happen to be the same, and on 1 PE the old and new code reproduce each other. For the future, it might be worthwhile to revisit this again using the bitwise-reproducing sums of Hallberg and Adcroft (Parallel Computing, 2014), or by resorting identical depth cells by increasing cell area, to give exact reproduction of these diagnostics for 90 or 180-degree rotations of the problem.

[Hallberg-NOAA]

This issue was closed by commit NOAA-GFDL/MOM6@1621b6b