Shaprio Wilk tests for normality
https://docs.google.com/document/d/18q4hYuT4AXF1nrGsxX1V8Qh47lYplLGLNzZMN14o-bI/edit
The SW test is to check whether the sampled data comes from a normal distribution. By applying this test to our dataset of collected user predictions we can test whether the users are aligning themselves around some specific mean. If they are not, then it means there is no consensus among the crowd.
The null hypothesis is that the distribution of predictions follow a normal distribution i.e. people tend to organize themselves around a mean with some people voting below the mean and some above. If this hypothesis is rejected then it means people don’t tend to organize around a specific mean i.e. the underlying distribution is not normal. This in turn implies the mean of the predictions may not be relied upon as a good approximation of what the crowd believes. If this hypothesis is not rejected then we can conclusively say that the mean of the predictions is a good approximate of what the crowd cumulatively believes to be the predicted value.
The decision to reject or not reject a hypothesis is determined by 2 statistics calculated by the SW test. The first is a W statistic which tells whether a sample (user predictions) come from a normal distribution and the second is a p-value which is used for rejecting the null hypothesis given some significance level. Smaller values of W imply departure from normality. The p-value is the probability of obtaining a result equal to or "more extreme" than what was actually observed, assuming the null hypothesis is true. In simple terms if p-value=0.9456 then it means that if we assume the null hypothesis is true, then there is a 94.56% chance that you would see a result as extreme or more extreme from the same distribution where this sample was collected. Which means there is a very high chance that the sample came from a normal distribution. So if the significance level threshold is say α=0.05 then any p-value less than α rejects the null hypothesis and p-value greater than α does not reject the hypothesis. (Note: this does not mean we “accept” the null hypothesis of normality. By not rejecting the hypothesis we mean that there is a high probability that the observations were sampled from a normal distribution. In truth it is quite literally impossible to find out the exact distribution that results in the sampled data in real life scenarios.).
IF the null hypothesis is rejected, then it means we cannot rely on the mean, standard deviation because the observed data may not be sampled from the hypothesized normal. Therefore, more voting may be required. If everyone is done voting and still null hypothesis is rejected, then it means crowd is not rallying around a single value. The big assumption here is that every idea has a constant but unknown time, revenue or cost value that we are trying to find out.
Disclaimer about SW normality test SW tests may be biased for smaller sample sizes. For small samples the test almost always doesn’t reject the null hypothesis. It is believed in the light of massive data that is collected nowadays, the null hypothesis in a normality test will always be rejected given there will be slight variations from the perfect normal. But these arguments are true for all methods of normality testing, which begs the question if normality testing is even necessary.