This is the final exam for Calibration: Knowing What You Know. It pulls the whole course together: what calibration actually means (the match between your stated confidence and your real hit rate), the reliability curve and the two faces of overconfidence — overestimation and overprecision — the proper scoring rules that grade your honesty (the Brier and log scores), thinking in ranges instead of points, and the practical habits that get you calibrated. Several questions look easy until you notice the trap: a forecaster who is perfectly calibrated yet completely useless, a confident answer that should have been humble, or a Brier score read upside down. Reason each one through — remember that lower Brier is better, and that calibration is not the same thing as accuracy.
How this exam works
Read carefully — this exam is final. Each question appears one at a time. Once you submit an answer it is locked for good: there’s no going back, no retry, and no restart. Your score is hidden until the end, where you’ll see a pass/fail verdict. The pass mark is 70%. A few questions ask you to select all correct answers.
What does it mean to be "well calibrated"?
Select an answer to continue.
Course Recap
Big picture
Calibration, in one picture
- Calibration: Knowing What You Know
- What calibration means
- Stated confidence should match real hit rate: of all your 70% calls, ~70% come true. The reliability curve plots confidence (x) vs fraction-correct (y); the 45-degree line is perfect; below = overconfident, above = underconfident. Calibration is not accuracy — weather forecasters are the gold standard.
- Overconfidence & overprecision
- Two faces: overestimation (90% answers right ~75%) and overprecision (90% intervals catch truth only ~40-50% of the time). The hard-easy effect makes overconfidence worst on hard questions. Driven by not seeking disconfirming evidence, remembering hits, and social reward for sounding sure.
- Scoring your beliefs
- Proper scoring rules make honesty optimal. Brier = (p − o) squared, averaged; 0 best, 1 worst, LOWER is better. The log score is brutal on confident misses — never say 0% or 100%. Brier decomposes into calibration + resolution.
- Thinking in ranges
- Point estimate vs confidence interval. First intervals are too narrow — deliberately widen. Use the 90% interval discipline (expect ~1 miss in 10); set bounds you are 95% sure are too low and too high. Honest error bars size your margin of safety.
- Getting calibrated
- Journal predictions BEFORE outcomes (fights hindsight); give ranges + explicit probabilities; score, see the reliability curve, adjust. Separate decision quality from outcome luck; anchor on base rates (ties to Bayes). Superforecasters use base rates, small updates, fine-grained odds. A clock that always says 50% is calibrated but useless — aim for calibrated AND decisive.
- What calibration means
Key takeaways
Calibration is the quiet discipline of making your confidence tell the truth: of all the times you say 70%, the thing should come true about 70% of the time — that is the 45-degree line on the reliability curve, with overconfidence sitting below it and underconfidence above. It is not the same as accuracy: the 50%-always and 99%-always coin predictors share a hit rate but live worlds apart. Overconfidence wears two faces — guesses that run too high and error bars that run too narrow — and the cure is to score yourself honestly. The Brier score (lower is better, 0 to
- and the brutal log score reward truth-telling and punish confident misses, so you never say 0% or 100%. Think in ranges, not points: widen your first interval, expect about 1 miss in 10 from a real 90% interval, and let honest error bars size your margin of safety. Then build the habit loop — journal before outcomes, anchor on base rates, update in small steps like a superforecaster, and separate the quality of your decision from the luck of the result. Above all, remember the trap of the clock that always says 50%: perfectly calibrated, utterly useless. The goal is not just humility — it is calibration and decisiveness, together.