Standard deviations on Camelyon17

[Feuermagier]

Hi,

first of all, thanks for this great collection of datasets!

I've noticed the large standard deviations of the test accuracy on Camelyon17. Do you have any explanation for this? Similarly, are there any "baseline" results that just use standard SGD? Also, is it correct so simply use the "standard" dataloader when training on Camelyon?

Thank you!
Florian

[kohpangwei]

Hi Florian,

It's a good question and one that we're not quite sure of ourselves. One hypothesis is that because the data comes from a small number of underlying slides (which are then converted into patches), there's less variability in the data than one might otherwise expect from the number of data points, leading to large swings in accuracy. That said, there are definitely models that can do consistently well.

I believe our baseline ERM results just use standard SGD? https://github.com/p-lambda/wilds/blob/main/examples/configs/datasets.py#L60

And yes, the standard dataloader works if you're not using any particular group structure.

Cheers,
Pang Wei