F1

The missing cell

F1 is the harmonic mean of precision and recall, which weights the smaller of the two more heavily than an ordinary average would.

Substituting the two definitions and simplifying gives the form worth staring at.

Three of the four cells of the confusion matrix appear. is absent, and it is absent completely. A model earns nothing for correctly leaving a negative case alone. The entire lower-right cell of section 02's table has been deleted.

This design is deliberate and traces back to the F1 score's origins in Information Retrieval. When a query retrieves a few relevant documents from a corpus of billions, the vast number of correctly ignored documents are practically irrelevant. A search engine deserves no credit for the billions of pages it successfully excludes.

So the first thing to ask of F1 is whether that description fits. If correctly identifying a negative case has real value, F1 is the wrong metric, and the reason has nothing to do with imbalance. A screening programme that reassures a healthy patient has done something useful, and F1 will not record it.

What it inherits

F1 is built from precision, and precision carries a prevalence term, so F1 carries one too. Writing the three cells in terms of the rates, with , and , and dividing through by :

The prevalence sits in the denominator, so F1 falls as positives get rarer, for the same reason precision did. Richardson et al. confirm it empirically, through an experiment in which, at a fixed threshold, F1 moves as the class mix changes (Richardson et al., 2024).

Take the same operating point as the previous section, and , and read F1 at the two prevalences.

Precision moved from to across the same gap. F1 moves from to , which is a good deal less, because recall is holding one end of the average still and recall does not depend on at all.

That is worth stating carefully, because it is the source of a common and understandable mistake. Moving less is not the same as not moving. F1 is more stable than precision under a change in the class mix, and it is not invariant to it. A metric that is merely less sensitive still cannot be compared across populations with different base rates.

The cost it adds

Now the part that belongs to F1 alone. Sections 03 and 04 each ended by asking where the metric puts its threshold, because that is where a metric states what a mistake costs. Accuracy answered 1/2 (one half). Balanced accuracy answered . F1 has an answer too, and it is a strange one, at least in my opinion.

Suppose the score is a calibrated probability, and consider flagging a small group of cases that all carry posterior , of total mass . Calibration is what makes the composition of that group known: is the fraction of it that is positive, so the group holds positives and negatives.

Until now all of them sat below the threshold and were being predicted negative. Flagging moves every one of them across one cell of the confusion matrix.

• The positives were misses and are now catches, so gains and loses the same . Formally, and .
• The negatives were correct rejections and are now false alarms, so gains . That is and .

Write with and . The numerator moves with alone, so it gains twice . Putting all three movements into the denominator, every term in cancels:

The denominator grows by whatever is, which is the whole trick. Differentiating the ratio, the change in is positive exactly when , that is when

At the optimum the threshold and the score agree, so the F1-maximising threshold satisfies

The threshold F1 asks for is half the F1 score we happen to achieve. Not half of something fixed in advance. Half of the metric's own value, which is a property of the model.

Why it lands there

The identity looks arbitrary until you notice that is a ratio, and ratios have a rule everybody already knows.

Look again at what one extra flagged case does. It adds exactly to the denominator, whatever its score, which was the line and is the whole trick. What it adds to the numerator does depend on its score: . So every case you flag brings of credit at a fixed price of .

Adding something to a ratio raises that ratio only when the thing you add is itself a bigger ratio. So flagging one more case helps when beats the already on the board, which is the condition we derived. It is the rule a batting average follows. A player hitting improves from almost any innings. A player hitting needs a very good one.

Reading the curve

That rule draws the shape in the figure below. Start at the far left, with near zero, where everything is flagged. Recall is perfect and precision is only the base rate, so is poor.

Now raise . The cases we give up first are the lowest scoring, and those are nearly all negatives, so you shed false alarms faster than you lose catches and climbs. Keep raising it and the balance eventually arrives, at this moment we start throwing away real positives faster than saving on false alarms, and falls again. The peak is the exact point where the next case is break-even, its equal to the we already have.

The dashed line is , the set of all break-even points, so every curve must peak where it crosses it. The better model's curve sits higher everywhere, so its running score is higher, so it is fussier about what counts as an improvement, so it crosses the line further to the right.

Read the consequence slowly, because I at least did not understand a thing when I first read this. Two teams work on the same problem, in the same hospital, facing the same consequences for a missed case. One team's model is better. F1 tells them to act at different thresholds. The cost of a mistake has not changed. The metric's opinion about it has, and it changed because of something that has no bearing on what harm a false alarm does.

Line the three up. Each is a claim about what to do, and they get progressively harder to notice:

• Accuracy acts above . A commitment, stated once, at least inspectable.
• Balanced accuracy acts above . A commitment handed over to whichever evaluation set was used.
• F1 acts above . A commitment handed over to the model being evaluated.

Nobody wrote any of the three down. The third is the one that could not have been written down, because it is not knowable until after the model is trained.

The problem, stated plainly

In practice, misclassifications carry tangible real-world costs. A missed case is worth a specific number of false alarms. The F1 score ignores this reality. It provides no parameter to input domain-specific costs, instead it enforces an implicit cost ratio of .

What makes this implicit cost even worse is that it's unstable. It drifts whenever we train a better model or evaluate on a dataset with a different base rate (prevalence ). As a consequence, the metric's internal cost assumptions fluctuate based on variables entirely unrelated to actual penalties.

This creates a flaw for model selection in that choosing the candidate with the highest F1 score inherently compares models at different thresholds. Evaluating each under its own arbitrary assumption about what a mistake costs. The winning model may simply be the one whose implied cost happened to artificially flatter its performance.

Beta, and what to do instead

The usual response to this is , which lets a preference for recall over precision.

While is an arbitrary default and explicitly prioritizes recall, neither solves the metric's core flaw. Modifying shifts the relative weight of precision and recall, but it does not transform the metric into a fixed payoff table. Because remains a ratio and its implicit cost threshold still fluctuates based on the model's overall score.

When F1 is the right choice

Two conditions, and they are the ones information retrieval satisfies.

1. True negatives genuinely carry no value, so deleting that cell costs nothing.
2. There is no predefined penalty for errors, and the system simply delivers a ranked list to a decision maker who determines the cutoff point.

Under these constraints, F1 serves as a reasonable summary of top-ranking performance. Its failure to encode a fixed cost is not a vulnerability here, but an honest reflection of a costless environment. However, it should always be reported alongside the class prevalence used during computation.

Step outside of that specific search-engine scenario, and the core problem remains. Every metric we have examined so far is just a threshold rule with a hidden cost buried inside it. The next section stops trying to pick a single threshold altogether, and instead explores how we can evaluate a model across every possible threshold at once.