ROC and AUROC

The curve

Section 02 ended with the observation that a single threshold gives a single pair of rates, so sweeping the threshold gives a curve. That curve is the Receiver Operating Characteristic (ROC).

Read the definition for what is missing. Both coordinates are column ratios, so no prevalence appears anywhere on either axis, and has been swept away rather than chosen. It also puts the last four sections in their place. Accuracy, balanced accuracy and F1 each nominated a threshold, and a threshold is a point on this curve.

Where those three thresholds came from

Each was derived in an earlier section, and the model in the figure is a single fixed one, so the three rules can be applied to it and compared directly.

Those three numbers fall in the order , , , and that ordering is the reason the dots climb the curve. A lower threshold flags more cases, which raises and together and slides the dots to the right. The disagreement between the three metrics is not about the model. It is about how far along this one curve to walk. The curve says something none of those four numbers could. It describes the model across every operating point at once, so two models can be compared without first agreeing on where to stand.

The area, and what it means

To systematically compare models across their operating range, we calculate the area under their ROC curves (AUROC). The interesting part is that this geometric area is exactly equal to a specific ranking probability:

where is a positive case drawn at random and a negative one. The area under the ROC curve is the probability that the model scores a random positive above a random negative. It is a statement about ranking, and it never mentions a threshold because it considers all of them.

McDermott et al. give a second form that makes the structure explicit (McDermott et al., 2024).

Think of this as a step-by-step procedure. Take a positive case, use its score as a threshold, and record the false positive rate at that exact point. Average that result across all positive cases. Here's one of the most interesting parts to this. Every false positive is penalized equally, no matter where it sits on the score distribution, because the expectation is unweighted.

The invariance

Neither of the forms above contains the prevalence and this is structural since both are built from and , and section 02 established that those divide by column totals. What happens practically is that we can sample as many extra negatives as we like, drawn from the same population, and the ROC curve does not move.

Richardson et al. demonstrate exactly this by simulation, holding the class-conditional score distributions fixed and varying the ratio across , and . The ROC-AUC distribution is unchanged (Richardson et al., 2024). And their conclusion on this is that the ROC-AUC of a classifier is not inflated by class imbalance.

What the invariance costs

Now the other reading. Flores et al. prove that for a calibrated score, AUROC is not threshold-free at all. It is an average of threshold decisions (Flores et al., 2025).

The inner term, the prior-adjusted maximum net benefit (PAMNB), quantifies the model's payoff on a population with prevalence under equal misclassification costs. Consequently, AUROC represents this payoff averaged across a spectrum of prevalences. Contrary to popular belief, AUROC does not ignore deployment prevalence, it implicitly averages over all possible prevalences.

But that raises a question: how is that average weighted? The equation's subscript gives it away. The averaging distribution, , is just the model's own distribution of scores. The metric gives the most weight to the prevalences that sit closest to where your model happens to output most of its scores.

Ok... that's probably not clear. Let's try to make it easier. If we have a screening problem where the true prevalence is somewhere between and , AUROC will still anchor its evaluation wherever the scores pile up. If they cluster around , AUROC is heavily weighting its evaluation on how the model would perform on a dataset that is positive. The metric answers a question about a hypothetical population we might never see, and we cannot tell it otherwise, because the formula has no way to input the actual base rate.

Holding both readings

The two papers agree completely on the mathematics and disagree entirely on what it is worth. It is worth being precise about why, because it is not a dispute that resolves.

Both are right. A metric that does not move when we resample the negatives is measuring the model. A metric that cannot be told what prevalence it will deploy at is not answering a decision question. Invariance and neutrality are not the same thing. AUROC has an opinion about deployment prevalence. Its opinion is all of them, weighted by its own score histogram.

Two conditions worth naming

The class-conditional score distributions must be fixed. Adding negatives identical to the existing distribution preserves the AUROC. However, real-world prevalence shifts often introduce out-of-distribution negatives. As Richardson et al. demonstrate (Richardson et al., 2024), enriching a dataset with novel, low-scoring negatives alters the underlying score distribution and forces the AUROC to shift.

Scores must be calibrated. Flores's interpretation of AUROC as an averaged net benefit strictly requires calibrated scores. If scores are uncalibrated, the metric remains purely a measure of ordinal ranking. AUROC only translates decision-theoretic insights when the score values carry reliably probabilistic meaning.