Test examples in parallel

[JDBetteridge]

Given that most users will probably try and run an example from the examples directory in parallel, enable parallel testing for all examples.

I do not anticipate this adding significant time to CI runs as the parallel tests will benefit from having a hot cache.

Any test in unit-tests or integration-tests can be made to run in parallel by using the Python decorator:

@pytest.mark.parallel(nprocs=4)

The above example would run the test with 4 MPI ranks.

Fixes #377.

Requires #416 to be addressed.

[JDBetteridge]

Tests take ~3min longer, seems like a win

[tommbendall]

This is great, thanks! It raises the question of which of the other unit tests or integration tests should be in parallel. The nice thing about the examples is that they cover most of the capabilities so this should give us reasonable coverage.

Just one small suggestion...

[JDBetteridge]

I agree, it's probably worth having a discussion at some point which of the unit/integration tests should be run in parallel. Using the pytest marker makes it very easy to update existing tests.

A short term suggestion would be: If you find any parallel bugs add a parallel test, or update an existing test to run in parallel.

[JDBetteridge]

Twice in a row! Nice!

I will pull out my debugging code and then this is ready for review

[tommbendall]

I'd really like us to start parallel testing by default, so I thought it would be helpful to update the branch.

But as we saw a few months ago, we can still get failures. On this latest run it was the skamarock_klemp_hydrostatic test, and it looks like other failures we've seen:

On the first call to the mixed solver, one rank seems to fall behind the others and not begin the ImplicitSolver_solve. Then all the ranks hang. Output from most log files:

2024-03-13 08:36:20,335 INFO     Semi-implicit Quasi-Newton: Mixed solve (0, 1)
2024-03-13 08:36:20,335 INFO     Compressible linear solver: rho average solve
2024-03-13 08:36:20,340 INFO     Compressible linear solver: Exner average solve
2024-03-13 08:36:20,345 INFO     Compressible linear solver: hybridized solve
2024-03-13 08:36:20,359 DEBUG            Residual norms for ImplicitSolver_condensed_field_ solve
2024-03-13 08:36:20,359 DEBUG            0 KSP unpreconditioned resid norm 4.413507124086e+07 true resid norm 4.413507124086e+07 ||r(i)||/||b|| 1.000000000000e+00
2024-03-13 08:36:20,362 DEBUG            1 KSP unpreconditioned resid norm 5.900637824684e+01 true resid norm 5.900637824491e+01 ||r(i)||/||b|| 1.336949881035e-06
2024-03-13 08:36:20,364 DEBUG            2 KSP unpreconditioned resid norm 5.041852127746e-05 true resid norm 5.042033608880e-05 ||r(i)||/||b|| 1.142409758753e-12
2024-03-13 08:36:20,376 DEBUG        Residual norms for ImplicitSolver_ solve
2024-03-13 08:36:20,376 DEBUG        0 KSP no resid norm 4.765315244582e+07 true resid norm 5.046385477193e-05 ||r(i)||/||b|| 1.058982505498e-12

Output from one log file:

2024-03-13 08:36:20,335 INFO     Semi-implicit Quasi-Newton: Mixed solve (0, 1)
2024-03-13 08:36:20,335 INFO     Compressible linear solver: rho average solve
2024-03-13 08:36:20,340 INFO     Compressible linear solver: Exner average solve
2024-03-13 08:36:20,345 INFO     Compressible linear solver: hybridized solve
2024-03-13 08:36:20,359 DEBUG            Residual norms for ImplicitSolver_condensed_field_ solve
2024-03-13 08:36:20,359 DEBUG            0 KSP unpreconditioned resid norm 4.413507124086e+07 true resid norm 4.413507124086e+07 ||r(i)||/||b|| 1.000000000000e+00
2024-03-13 08:36:20,362 DEBUG            1 KSP unpreconditioned resid norm 5.900637824684e+01 true resid norm 5.900637824491e+01 ||r(i)||/||b|| 1.336949881035e-06
2024-03-13 08:36:20,364 DEBUG            2 KSP unpreconditioned resid norm 5.041852127746e-05 true resid norm 5.042033608880e-05 ||r(i)||/||b|| 1.142409758753e-12

[tommbendall]

In the failing example that I posted above, the hanging comes between the condensed_field solve (the preconditioning step for the hybridized solver) -- which suggests to me that it could be worth reviewing whether the solver options are valid in parallel for our domains:

    solver_parameters = {'mat_type': 'matfree',
                         'ksp_type': 'preonly',
                         'pc_type': 'python',
                         'pc_python_type': 'firedrake.SCPC',
                         'pc_sc_eliminate_fields': '0, 1',
                         # The reduced operator is not symmetric
                         'condensed_field': {'ksp_type': 'fgmres',
                                             'ksp_rtol': 1.0e-8,
                                             'ksp_atol': 1.0e-8,
                                             'ksp_max_it': 100,
                                             'pc_type': 'gamg',
                                             'pc_gamg_sym_graph': None,
                                             'mg_levels': {'ksp_type': 'gmres',
                                                           'ksp_max_it': 5,
                                                           'pc_type': 'bjacobi',
                                                           'sub_pc_type': 'ilu'}}}

[JDBetteridge]

I wonder if we're caching something we shouldn't on a mesh hierarchy. We recently fixed failing tests in Firedrakeland by ensuring a fresh mesh hierarchy each time. That approach wouldn't work here but might be a good place to start. It would be good to whittle this down to a minimal failing example to work on.

[tommbendall]

I don't think we actually use a mesh hierarchy in any tests -- we only use algebraic multigrid and not geometric, although this is definitely something that we need to do to improve performance.

I agree about the minimum failing example, the hard thing is that sometimes it will pass!!

[JDBetteridge]

Some recent changes in Firedrake my have fixed some parallel bugs. I'm running CI again and crossing my fingers 🤞 .

If this still doesn't work we should try tackling this at the hackathon.

resolved

[tommbendall]

This has been talked about extensively offline. Thanks Jack and I approve!