P3M benchmark randomly fails CI

[espresso-ci]

https://gitlab.icp.uni-stuttgart.de/espressomd/espresso/pipelines/7734

[jngrad]

This is a recurring error:
https://gitlab.icp.uni-stuttgart.de/espressomd/espresso/-/jobs/125861
https://gitlab.icp.uni-stuttgart.de/espressomd/espresso/-/jobs/126068
https://gitlab.icp.uni-stuttgart.de/espressomd/espresso/-/jobs/126071
Re-starting the job usually fixes it. It is caused by this line:

espresso/maintainer/benchmarks/p3m.py

Line 167 in b7812cb

energies = system.analysis.energy()

With error message: Exception: calc_long_range_energies failed: ERROR: number of cells 1 is smaller than minimum 8 (interaction range too large or min_num_cells too large) in function void dd_create_cell_grid(). It happens at random. The P3M benchmark script is MPI-capable, but its test currently runs without MPI. I never had any issue running this benchmark without MPI, even with a larger number of particles.

[RudolfWeeber]

Either the p3m tuning doesnot respect box_l/2 as maximum real space cutoff
or tune_skin(), which is called after p3m tuning does not.
Maybe some code needs to be inserted to calculate max_skin from the current interaction cutoffs and the local box length.

[fweik]

I'm looking into this, I think there is a bug in the tuning.

[jngrad]

I can't reproduce the bug locally nor locally in the docker container, but I can reproduce it on coyote8 in the docker container (after 175 tries). I'll try again while printing the random seed and check if it's deterministic.

[fweik]

Please don't spend any more time on this, I will fix it, I know what the error is.

[RudolfWeeber]

#2961
I assume that the resulting p3m parameters are outside usual values in the CI environment.
Short term, it might be enough to disable the tune_skin in the test, but generally #2961 has to be fixed.

[RudolfWeeber]

Actually, the maximum safe skin might be available from s.cell_system.get_state()["max_skin"]

[RudolfWeeber]

Did this re-occur, since the adjust_max_skin was added in the tune_skin() call?
Otherwise, it can be closed.

[jngrad]

Hasn't yet in CI. But I was able to trigger a similar error today on coyote8 after 493 retries:

[...same as in the original logfile...]
  File "/home/espresso/espresso/build/testsuite/scripts/benchmarks/local_benchmarks/p3m_processed.py", line 164, in <module>
    adjust_max_skin=True)))
  File "cellsystem.pyx", line 308, in espressomd.cellsystem.CellSystem.tune_skin
  File "utils.pyx", line 261, in espressomd.utils.handle_errors
Exception: Error during tune_skin: ERROR: number of cells 1 is smaller than minimum 8 (interaction range too large or min_num_cells too large)

The last 3 lines changed. The error now throws in espressomd.cellsystem.CellSystem.tune_skin instead of espressomd.analyze.Analysis.energy.

[RudolfWeeber]

So that means that tune_skin() doesn't chose the maximum permissible skin for adjust_max_skin=True correctly?

[RudolfWeeber]

There is a global 'max_skin' domain_decomposition.cpp: max_skin = min_cell_size - max_cut; which is set to sth different than in tune_skin() double const max_permissible_skin = 0.5 * min_local_box_l - max_cut;

[fweik]

The first one is correct, the second one isn't. You should also maybe have a look at #3053, which clarifies some of these things.

[jngrad]

I've just merged #3053 locally in python and was able to get the same error message, plus a new one:

resulting parameters: mesh: (22 22 22), cao: 7, r_cut_iL: 3.6018e-01,
                      alpha_L: 9.0136e+00, accuracy: 9.9536e-05, time: 12.38

0: rs_mesh overflow! (pos 12.448619, nmp=24)
0: allowed coordinates: -1.600000 - 13.477258
0: rs_mesh overflow! (pos 12.482639, nmp=24)
0: allowed coordinates: -1.600000 - 13.477258
0: rs_mesh overflow! (pos 12.516053, nmp=24)
0: allowed coordinates: -1.600000 - 13.477258
[1617038c16f4:13557] *** Process received signal ***
[1617038c16f4:13557] Signal: Segmentation fault (11)
[1617038c16f4:13557] Signal code:  (128)
[1617038c16f4:13557] Failing at address: (nil)
[1617038c16f4:13557] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f2b8a88b390]
[1617038c16f4:13557] [ 1] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x22)[0x7f2b8a534512]
[1617038c16f4:13557] [ 2] /home/espresso/espresso/build3/src/core/EspressoCore.so.4(Particle::~Particle()+0x3c)[0x7f2b88d3ffbc]
[1617038c16f4:13557] [ 3] /home/espresso/espresso/build3/src/core/EspressoCore.so.4(invalidate_ghosts()+0x6f)[0x7f2b88dab89f]
[1617038c16f4:13557] [ 4] /home/espresso/espresso/build3/src/core/EspressoCore.so.4(cells_resort_particles(int)+0x2b)[0x7f2b88d3babb]
[1617038c16f4:13557] [ 5] /home/espresso/espresso/build3/src/core/EspressoCore.so.4(integrate_vv(int, int)+0x25a)[0x7f2b88db6e4a]
[1617038c16f4:13557] [ 6] /home/espresso/espresso/build3/src/core/EspressoCore.so.4(mpi_integrate(int, int)+0x77)[0x7f2b88d51ed7]
[1617038c16f4:13557] [ 7] /home/espresso/espresso/build3/src/core/EspressoCore.so.4(tune_skin(double, double, double, int, bool)+0x27b)[0x7f2b88e24a9b]
[1617038c16f4:13557] [ 8] /home/espresso/espresso/build3/src/python/espressomd/cellsystem.so(+0x1282f)[0x7f2b546cc82f]
[1617038c16f4:13557] [ 9] /home/espresso/espresso/build3/src/python/espressomd/script_interface.so(+0x1999c)[0x7f2b7e6af99c]
[1617038c16f4:13557] [10] /usr/bin/python3(PyObject_Call+0x47)[0x5c20e7]
...
[1617038c16f4:13557] *** End of error message ***
Segmentation fault (core dumped)

[jngrad]

master is still failing:
https://gitlab.icp.uni-stuttgart.de/espressomd/espresso/-/jobs/153761

[jngrad]

even after merging #3132, it's still possible to get the p3m benchmark to fail on coyote7 after 200 trials: Exception: Error during tune_skin: ERROR: number of cells 1 is smaller than minimum 8 (interaction range too large or min_num_cells too large)

[RudolfWeeber]

Out of ideas. De-milestoning.

[jngrad]

We might as well disable this test in CI. We already know this benchmark cannot be used due to the non-deterministic nature of the P3M tuning function. It also fails due to the other, less frequent bug reported above. We can re-enable the test once the tuning function gets re-implemented.

[fweik]

I agree do disable it now. We could also use fixed parameters to avoid the problem.

…

On Thu, Nov 28, 2019, 23:39 Jean-Noël Grad ***@***.***> wrote: We might as well disable this test in CI. We already know this benchmark cannot be used due to the non-deterministic nature of the P3M tuning function. It also fails due to the other, less frequent bug reported above <#2924 (comment)>. We can re-enable the test once the tuning function gets re-implemented. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2924?email_source=notifications&email_token=AAG2FX6JJ4QDYCZ3HF2JX4TQWBCD5A5CNFSM4HY4JVAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFNQQEY#issuecomment-559613971>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG2FXYUQOCWZ6TXPKICAJTQWBCD5ANCNFSM4HY4JVAA> .