Disable P3M benchmark test #3347
Conversation
The benchmark test is fragile and breaks regularly in CI. The P3M benchmark itself is not useful at the moment, as the P3M tuning method is non-deterministic. Multiple attempts at fixing the tuning function did not resolve the issue.
Codecov Report
@@ Coverage Diff @@
## python #3347 +/- ##
=======================================
+ Coverage 86% 86% +<1%
=======================================
Files 538 538
Lines 25348 25348
=======================================
+ Hits 21830 21831 +1
+ Misses 3518 3517 -1
Continue to review full report at Codecov.
|
|
Hm, if you just comment it out, than it will decay and stop working. I think it would be better to either make it deterministic by fixing the parameters or remove it altogether. |
All the information I got from coworkers, from code introspection and from debugging sessions where parameters and random seeds are fixed, are consistent with a non-deterministic P3M tuning algorithm, for a continuous period of 6 months. More specifically, the bisection uses a heuristic that depends on the exact timing of all previous iterations, and these timings are affected by all other threads running on the same CI machine. I don't see what you have in mind here. Didn't you once suggest we should re-implement the P3M tuning in Python with a different algorithm to make it more deterministic?
True. Let's wait until #3326 is merged, then we can disable the benchmark tests only on the subset of CI jobs where it fails. |
|
I meant fixing the P3M parameters instead of tuning them. The tuning as it stands is non-deterministic because it depends on the (uncontrolled) run time. |
|
That could work on a single machine. Getting the P3M parameters once and re-using them for all other myconfig files would make the benchmark actually useful. But can we come up with P3M parameters that would work on any machine? |
Closes #2924