print more timing statistics

[stevengj]

Closes #924.

[stevengj]

Would be nice to verify that the standard deviation of the timing is related to load balancing.

For example, you could try running the benchmark in #681, where we know that split_by_cost improved the load balancing compared to split_by_effort, and verify that the timing variance goes down as well for split_by_cost.

[oskooi]

This seems to be working but the results for split_by_cost and split_by_effort are swapped which might indicate that there is a bug in the split_chunks_evenly parameter of the Simulation object that is mixing up the boolean values (i.e., split_chunks_evenly=True actually corresponds to split_by_cost rather than split_by_effort).

Running the unmodified example in #681 for the two cases and plotting the results (just the time-stepping portion of the simulation) is shown below.

split_by_effort

$ for n in `seq 4`; do mpirun -n ${n} python tall_skinny_cell_3d.py -e |grep "time stepping"; done
            time stepping: 1.68271 s
            time stepping: 0.966419 s +/- 0.00839127 s
            time stepping: 0.675613 s +/- 0.0180802 s
            time stepping: 0.553308 s +/- 0.0140741 s

split_by_cost

$ for n in `seq 4`; do mpirun -n ${n} python tall_skinny_cell_3d.py |grep "time stepping"; done
            time stepping: 1.68216 s
            time stepping: 1.57454 s +/- 0.0353577 s
            time stepping: 1.6826 s +/- 0.0899637 s
            time stepping: 1.73653 s +/- 0.0841767 s

Note that the magnitude of the error bars for the "split_by_effort" case is smaller than those for "split_by_cost" which is expected if the results are swapped.

[stevengj]

Anyway, it seems like the variance is doing what it is supposed to: it is larger in the case with poorer load-balancing.

The surprise here is that split_by_cost is worse load-balancing, but that seems like a separate issue.

[stevengj]

note that this is computing stddev = ∑(xᵢ-x̄)² / (n-1) = (∑xᵢ² - nx̄²) / (n-1)