Calculation of OOD within the paper

[dalbertweiss]

Hello together,

first of all I would like to thanks for making the paper and code to "WILDS: A Benchmark of in-the-Wild Distribution Shifts" publicly available. What caught my interest when reading the paper was the estimation of the in (IID) and out-of-distribution (OOD) which has been evaluated using empirical risk minimization (Table 1 page 20). My question is how the IIDs and OODs were calculated. Did you use the softmax with temperature scaling according to the paper "Enhancing the reliability of out of distribution image detection in neural networks"? If not, can you give reference to the way you tackled this problem?

Thank you in advance for your kindful reply

[kohpangwei]

No, we didn't use temperature scaling, and we're also not doing OOD detection. We trained the models listed in Section 5.3 of the paper (also see Appendix D for details), and then evaluated them on the ID and OOD test sets. Does that make sense?

[dalbertweiss]

Thank you very much. I did a little more research at this became much clearer to me. However the question I'm still having is how you did the evaluation study the ID and OOD performance. Could you allaborate more on how you obtained the results in Appendix D Table 3 (page 66)?

[dalbertweiss]

Ok. I rechecked the paper. Everything is resolved. Thank you again and best of luck for future research :-)