Skip to content

Commit

Permalink
Update Benchmark.md
Browse files Browse the repository at this point in the history
  • Loading branch information
Expertium authored Dec 13, 2024
1 parent 21144bc commit b7d6cec
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion Benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Next is AUC (Area Under the Curve). Unlike the previous two metrics, AUC is not
1. AUC measures how well an algorithm can tell classes apart; in our case, classes are "recalled" and "forgotten." You can think of AUC as a measure of how well the algorithm can draw a boundary between two classes, such that all members of class 1 are on one side of the boundary and all members of class 2 are on the other side.
2. AUC ranges from 0 to 1, but in practice it's almost always greater than 0.5. AUC less than 0.5 indicates that the algorithm is performing worse than random. Higher is better.

AUC can be rather unintuitive in some cases. Exaggerated example: suppose you have an algorithm that always outputs a 99% probability of having cancer for people who do have cancer and a 98% probability of having cancer for people who do not have cancer. It never outputs 98% for those who do have cancer, and it never outputs 99% for those who do. What do you think is the AUC of this algorithm? Answer: 1.0, because it can perfectly distinguish between these two classes, even if the calibration is absolutely terrible. AUC doesn't tell us anything about calibration, only about discrimination.
AUC can be rather unintuitive in some cases. Exaggerated example: suppose you have an algorithm that always outputs a 99% probability of having cancer for people who do have cancer and a 98% probability of having cancer for people who do not have cancer. It never outputs 98% for those who do have cancer, and it never outputs 99% for those who don't. What do you think is the AUC of this algorithm? Answer: 1.0, because it can perfectly distinguish between these two classes, even if the calibration is absolutely terrible. AUC doesn't tell us anything about calibration, only about discrimination.

Below is a diagram that explains AUC.

Expand Down

0 comments on commit b7d6cec

Please sign in to comment.