What is random baseline in paper (Amazon)? #61
-
|
Hi, In the paper(version 2) , figure 22 mensions a baseline which the ERM cxompared to. What is this random baseline? Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hi Yachuan, That refers to a random binomial baseline with the same average accuracy (as described in E.7.2.). Specifically, the DistilBERT-base-uncased ERM model achieves an average accuracy of 71.9%. The random binomial baseline matches this average accuracy by independently labeling each point correctly with a probability of 71.9% or wrongly with a probability of 28.1%. Figure 22 plots the resulting distribution of per-reviewer accuracies under this random binomial baseline. We can see that the tails of the distribution of per-reviewer accuracies under the ERM model are longer, which indicates that its tail performance on some reviewers is worse than what we'd expect if all reviews from all reviewers were treated equally. Hope that helps! |
Beta Was this translation helpful? Give feedback.
Hi Yachuan,
That refers to a random binomial baseline with the same average accuracy (as described in E.7.2.). Specifically, the DistilBERT-base-uncased ERM model achieves an average accuracy of 71.9%. The random binomial baseline matches this average accuracy by independently labeling each point correctly with a probability of 71.9% or wrongly with a probability of 28.1%. Figure 22 plots the resulting distribution of per-reviewer accuracies under this random binomial baseline. We can see that the tails of the distribution of per-reviewer accuracies under the ERM model are longer, which indicates that its tail performance on some reviewers is worse than what we'd expect if all reviews fro…