Accuracy

Accuracy is a payoff

Accuracy is usually introduced as counting. How many did the model get right, out of how many were shown to it. Counting is what the arithmetic does, but it is not what the number means.

Let's visualize it in another way. Attach a value to each of the four outcomes before any counting starts. A correct call is worth and a mistake is worth .

Now run the model over the population, look up each case in the table, and take the average of what comes back.

where is the table above, read as a function: it takes the prediction and the truth, and returns what that pairing is worth. The expectation is taken over the population, so it averages across every case the model will meet.

That average is accuracy. Not a count of correct answers, but the expected payoff of running this model on this population, under one particular schedule of payoffs. The schedule above has a name, it is the zero-one loss, so called because every mistake costs exactly one and every success costs nothing.

This is also the shape every metric in the later sections turns out to have. Net benefit is this table with different numbers in it. So is cost-sensitive accuracy, and so is the bounded score at the end of the tutorial. Once a metric is a payoff, the interesting question stops being how it is computed and becomes which payoffs it assumes.

The threshold the table implies

Section 01 left a threshold sitting at with no justification beyond habit. The payoff table supplies one, provided the score is a calibrated probability. The aside below explains what we mean by that:

Take a single case with score , and work out what each of the two available answers is worth. Predicting positive pays if the case really is positive, which happens with probability , and otherwise. Predicting negative pays with probability .

Take whichever is larger. Predicting positive wins exactly when , which is to say when . So the accuracy of the model is maximised by the rule

which is the Bayes rule for the zero-one loss. The one half was never arbitrary after all. It fell out of the table, and it fell out because the table is symmetric: the two mistakes cost the same, so the tie sits in the middle. Put different numbers in the table and the tie moves.

A still threshold

There is a trap here and I think it's worth spending some time on it.

The rule above is optimal at any prevalence. Nothing in the derivation mentioned , so one half is the right cut whether positives are half the population or one case in a thousand. That sounds as though prevalence has stopped mattering.

It has not. The threshold is stable, but the quantity being thresholded is not. is a statement about a population as well as about a case. The same patient, with the same symptoms and the same measurements, carries a different probability of disease in a specialist referral clinic than in a general screening programme, because the two populations differ in how much disease is in them to begin with.

Move the population and every posterior moves with it, so cases that used to sit above one half can drop below it without a single feature changing. Prevalence did not leave the decision. It moved out of the threshold and into the score. A later section makes that migration exact.

Without calibration, none of this holds

Every line above assumed is a genuine probability. Most scores are not. A margin, a logit, an ensemble vote share and an uncalibrated network output are all perfectly usable for ranking, and for none of them does carry any meaning at all. It is a point on an arbitrary scale.

For such a score the accuracy-maximising threshold still exists, but it has to be found rather than derived: sweep across its range and read off the peak, exactly the curve from the opening section. The peak is wherever it is, and it moves with the prevalence, because for an uncalibrated score the threshold is doing the job the posterior was supposed to do.

Biased is the wrong word

Here is the standard objection, stated as generously as it deserves:

On a dataset where ninety-nine percent of cases are negative, a model that answers negative every time scores . Accuracy is therefore biased under class imbalance, and should be replaced by a metric that is robust to it.

The first sentence is arithmetic and it is correct. The advice at the end is broadly good as reporting a bare accuracy figure on a problem really is bad practice. Everything in between is somewhat wrong.

Nothing here is biased

An estimator is biased when its average value across repeated samples differs from the quantity it is meant to estimate. Accuracy computed on a test set is an average of independent indicators, and its expected value is exactly the population accuracy it names. It is unbiased, at any prevalence, for any model, always.

The in the objection is not a distortion of the truth. It is the truth. On that population, at that threshold, with those payoffs, the model really does earn per case. Nothing has been overstated. The number is correct and the complaint has to be about something else.

Three complaints

The phrase covers three separate objections, and they have different causes and different cures.

1. The number describes a population, not a model. Move to a population with a different mix and the number moves, which is what the opening section showed. That is real, but it is not bias. It is the metric correctly reporting a different quantity, because a different quantity is what was asked for. The cure is to state the prevalence, not to change metric.
2. The estimate is noisy when one class is scarce. With forty positives in the test set, any number computed from those forty carries a wide interval. Also real, also not bias, and not about accuracy at all since it would harm recall, precision, or anything else estimated from the same forty cases.
3. The payoff table is wrong for the problem. A missed case and a false alarm were entered as costing the same, and depending on the application they are not.

The third complaint is not about imbalance

Imbalance without the defect. Suppose the task is predicting whether a delivery arrives on a Tuesday, positives run at one percent, and the two mistakes genuinely cost the same because nothing much depends on either. Accuracy is fine. The baseline is real and uninteresting, and a model beating it is really doing something. The imbalance is extreme and no defect appears.

The defect without imbalance. Now take a perfectly balanced problem: half of the welds inspected have a flaw, half do not. A missed flaw eventually kills somebody. A false alarm costs an hour of re-inspection. Accuracy weights those two identically and is a catastrophic choice of metric, at , where nobody would think to call it an imbalance problem.

Imbalance is neither necessary nor sufficient for accuracy to mislead. What makes it mislead is a payoff table that does not match the application. Imbalance is merely the condition under which a bad table produces an embarrassing number rather than a merely wrong one, which is why the two get confused.

Why the confusion is expensive

A misdiagnosis leads to the wrong prescription. If the problem is believed to be imbalance, the cure looks like finding a metric that is robust to imbalance, and the search ends at whichever metric moves least when the mix changes. If the problem is understood as a wrong payoff table, the cure is to write down the right one.

What cannot be repaired

Only one of the three complaints was about accuracy itself, and it was the payoff table. So correct it. Work out what a missed case is worth, put that number in, and carry on.

That works, and it is the end of accuracy. The values , , , are not a setting accuracy ships with. They are what the word means. Replace them and the arithmetic still runs, but what comes out is a different quantity with a different name. There is no version of accuracy that carries a cost.

So why is it always low prevalence

Imbalance is neither necessary nor sufficient, and that stands. Low prevalence is still where accuracy fails nearly every time, for three unrelated reasons.

1. Rarity and asymmetry travel together. Things are usually rare on account of being bad: disease, fraud, fracture, default. Low prevalence does not imply a broken table. It predicts one.
2. The scale collapses. At everything worth arguing about is packed into the last hundredth, so large real differences surface in the third decimal place.
3. The measurement gets noisy. The half is estimated from however many positives the test set happened to hold. A later section is devoted to this.

The first needs a cost, the second a baseline, the third more positives. None of them is answered by a metric that moves less when the class mix changes.

If accuracy is reported anyway, print beside it, which is what a model earns by ignoring its input and always naming the larger class. The gap is the only part of the number that describes the model. At , a reported sits against a baseline of . At , a reported sits against a baseline of , and the gap is nothing at all.

Balanced accuracy, next, leaves the payoff table exactly where it is and changes the prevalence instead.