Calibration: Knowing What You Know

You already learned to think in probabilities — to trade the yes/no switch for a dial, anchor on base rates, weigh bets by expected value, and update your beliefs by degrees with Bayes. This is the capstone of that path. It asks the question that sits on top of all the others: when you put a number on a belief, is the number any good?

Here is the whole idea in one line. A thinker is calibrated when their confidence matches their accuracy — when, of all the times they say “70% sure,” about 70% turn out to be right. Calibration is not about being smart, or even about being right more often. It is about being right about how right you are. A forecaster who says 90% and is correct 90% of the time is calibrated. So is one who says 60% and is correct 60% of the time. The failure isn’t being uncertain; the failure is claiming a certainty you haven’t earned.

And almost everyone fails it in the same direction. We are systematically overconfident: our 90% answers come true maybe 70% of the time, and the “90% confidence intervals” we draw around our estimates catch the real answer far less than 90% of the time. That gap — between the certainty we feel and the accuracy we actually have — is the single most common, most measurable, most fixable flaw in human judgment. This course measures it, names where it comes from, and hands you the tools to close it.

The route is hands-on. You’ll meet calibration precisely (the reliability curve that plots what you said against what came true) and drive an interactive calibration lab that scores your own confidence on a batch of true/false claims. You’ll separate the two faces of overconfidence — overestimation (your guesses run too high) and overprecision (your error bars run too narrow) — and feel the hard-easy effect that makes hard questions the most dangerous. You’ll learn to score a belief with proper scoring rules — the Brier score and the log score — that reward honesty and punish bluffing, so a probability becomes something you can be graded on like an exam. You’ll turn point estimates into honest ranges — 90% intervals wide enough to actually contain the truth. And finally you’ll learn how to get calibrated: track your predictions, give ranges, run calibration training, post-mortem your scores, and hold the two virtues that pull against each other — calibration (humility) and resolution (decisiveness) — at the same time, the way the best forecasters do.

By the end you’ll own the rarest meta-skill in all of judgment: not just estimating the odds, but knowing exactly how much to trust your own estimate — the honest error bar that every other model in this latticework quietly depends on.