Some question about the distribution of acc1-acc2

[JianSun411]

Hey there, CCIT contributors,
From line 389 in CCIT.py file, I think you believe that acc1-acc2 obeys the normal distribution N(0, 2\sigma(acc2)^2) where \sigma(acc2) is the standard variance of acc2. I think this is right, too. But based on this thought, there are two inconsistent points in the other part of the codes:

1. In line 373, only "s2 = np.std(cleaned, axis = 0, doff = 1)[4]" is the sample variance, the unbiased estimator of \sigma(acc2) (the standard variance of acc2). "np.std(cleaned, axis = 0)[4]" is the population standard variance which is not the unbiased estimator of \sigma(acc2).
2. In line 391, when bootstrap == False, why the standard variance is np.sqrt(2) * 1/np.sqrt(ntot) (np.sqrt(2) is multiplied in function "pvalue", line 325)? I think it should be np.sqrt(2) * np.sqrt(acc2 * (1-acc2)/ntot) since acc2 obeys the distribution N(acc2, acc2*(1-acc2)/ntot) (acc2 follows the normal distribution since it is generated from a Binomial Distribution where y_pret == y_test)

BTY, I appreciate your paper Model-powered Conditional Independence Test. It is great!

[rajatsen91]

Thanks for the comment. You are right on both counts, I will change it in a future revision.

I suspect that the performance gap due to (1) will be pretty small.

[JianSun411]

Thanks for your reply. Besides, I got puzzled by the explanation of CCIT function where it says "If pval is low CI is rejected if it's high we fail to reject CI.". However, the paper says "... when H_0 is true, the bias will be close to 0" (in the paragraph named "Algorithm with Bias Correction"). If so, CI is rejected if the pval (0.5 * erfc(x/np.sqrt(2))) is far away from 0.5. These two statements consistent with each other if the bias is always positive. Although the paper does state that the bias > 0, I find that, in practice, the pval can be higher than 0.5, i.e, the bias is negative.

Do you have any idea about why the bias can be negative? And when should we reject the CI?