drude gpu sample fails CI

[jngrad]

Two samples have been randomly failing tests recently:

• Grand Canonical has large deviations from target concentration (for the last 2 weeks)
• Drude in BMIM PF6 has broken bonds (for the last 2 days)

[jngrad]

Drude is still failing:
https://gitlab.icp.uni-stuttgart.de/espressomd/espresso/-/jobs/156474
https://gitlab.icp.uni-stuttgart.de/espressomd/espresso/-/jobs/154727

[jngrad]

Finally found the source of the error: bad P3M parameters from the tuning function. There seems to be a pattern in P3M parameters from simulations that crash: the r_cut value is almost twice as small as in simulations that run fine. To reproduce it locally or on a coyote with ubuntu-python3:cuda-10.1:

make local_samples
cd testsuite/scripts/samples
sed -i  "/system.actors.add(p3m)/i p3m._params = {'cao': 7, 'inter': 32768, 'r_cut': 2.5491526892051883, 'alpha': 1.286486160729783, 'accuracy': 0.0009884728050820963, 'mesh': [120, 120, 120], 'epsilon': 0.0, 'mesh_off': [0.5, 0.5, 0.5], 'tune': True, 'check_neutrality': True, 'prefactor': 1389.3612645, 'alpha_L': 47.67025070635568, 'r_cut_iL': 0.06879447050636864, 'cao_cut': [0.0, 0.0, 0.0], 'a': [0.0, 0.0, 0.0], 'ai': [0.0, 0.0, 0.0], 'inter2': 0, 'cao3': 0, 'additional_mesh': [0.0, 0.0, 0.0]}" local_samples/drude_bmimpf6.py
rm -f local_samples/drude_bmimpf6_gpu_processed.py; ../../../pypresso test_drude_bmimpf6_with_gpu.py

The LJ sigmas are in the range 3.4-5.0, in simulations that don't crash the tuned P3M r_cut is in the range of 3.3-5.1, while in crashed simulations r_cut is in the range 2.5-2.8. If this is the real cause, we could add a lower bound to r_cut in the parameters of the tuning function.

Note: checkpointing the particle positions/forces/velocities obtained from a bad tuning (i.e., the simulation eventually crashed) into a simulation with good P3M parameters doesn't lead to a crash.

[fweik]

Do we have any theory why a low P3M r_cut leads to crashes? That does not make sense to me.

[KaiSzuttor]

closing in favor of #3842