Alpha cannot be specified for critical difference diagram

[j-adamczyk]

Describe the bug

I haven't seen any way to provide alpha (0.05 or 0.1) for plotting critical difference diagram of Nemenyi-Friedman test. I'm trying to follow Demsar's paper link and this tutorial (which sadly uses Orange package), and:

• "The performance of two classifiers is significantly different if the corresponding average ranks differ by at least the critical difference"
• "critical values q_α are based on the Studentized range statistic divided by √2", and examples of such values are provided in the paper
• relevant scikit-posthocs code

So I get the p-values from the test. But looking at CD plotting function, we can see that it uses sign_array() function with constant arguments, which is alpha=0.05. So there is no way to plot CD diagram with any other alpha level. This should be configurable.

[maximtrp]

Hello! Thank you for pointing at this. This test was based on statsmodels' implementation of Studentized range distribution. It is bound to 0.1, there is no way to specify alpha value. Now we have SciPy's implementation which should be more appropriate here.

[maximtrp]

Hopefully fixed in v0.10.0