Civil Comments Dataset

[vihari]

Thanks for the wonderful initiative!
I have been working with Civil-Comments (CC) dataset of WILDS. In this dataset, the reported numbers in the main section are when trained on two demographic groups: black and non-black. However, it is evaluated on all the 8 demographic and 2 classes: 16 combinations. I do not understand the rationale behind evaluating on all the 16 groups when training data contains only 4 groups. Group DRO and Reweighted seem clearly not equipped to avoid overfitting on unseen group assignments.

Please clarify, thanks.

[kohpangwei]

Hi Vihari,

Thanks for the question! As described in the CivilComments-WILDS appendix (Appendix E.4), the training data contains all of the requisite demographic information to form the 16 groups. However, the baseline methods require disjoint groups, whereas the 16 groups are overlapping (details in Appendix E.4.1). We agree that there's room for improvement here! From Appendix E.4.2, "adapting the baseline methods to handle multiple overlapping groups, which were not studied in their original settings, could be a potential approach to improving accuracy on this task".

We'll clarify this in the next version of the manuscript. Thanks for pointing it out.

[vihari]

I understand the shortcoming of the baseline methods. My question rather is this: wouldn't it be more appropriate to evaluate only the groups for which group assignment is supervised during the train time.
That is, evaluate DRO and Reweighting only on the demographic attribute used during training: black, non-black.

Hope the question is clear.

[kohpangwei]

No, the actual task we're interested in is to do well over all 16 groups. The shortcomings of these particular baseline methods shouldn't affect the task; instead, we should develop methods that can do better on the task!

[vihari]

Alright, that makes sense. The intent of WILDs is to provide datasets and tasks.