Finding the Denominator: How to Beat the Bias

Four lessons in, you can now spot survivorship bias in the wild — the erased funds, the dropout billionaires, the “built to last” houses that are really just the ones that lasted. Spotting it is nice. But a diagnosis you can’t act on is just anxiety with footnotes. This is the lesson where we hand you the tools: a small kit of concrete moves that turn “hmm, this smells like survivorship bias” into “here is exactly what I go looking for, and here is what I do when I can’t find it.”

Every technique below is a variation on one idea, so let’s name it up front. Survivorship bias deletes the losers; the cure is to put them back. Sometimes you find the real data on the dead. Sometimes you can only reason about the hole they left. Either way, you stop letting the survivors testify unopposed.

Technique 1 — Find the denominator

Here is the master move, the one that contains all the others. A survivor statistic is a numerator wearing a disguise. “Nine of ten successful founders did X.” “Most centenarians drank a little wine.” “The best-performing funds all used strategy Y.” Every one of these is a count of winners — a numerator — dressed up to look like a rate. And a numerator alone tells you nothing, because the question that matters is always: out of how many?

The real success rate of a recipe is:

winners who did X ÷ everyone who did X (winners and losers)

not the seductive fake rate the survivors hand you, which is really just winners ÷ winners — a number that’s rigged to be near 100 percent no matter what X is. This connects straight to the base-rate reasoning from thinking in probabilities (a prerequisite of this course): you cannot judge whether evidence supports a claim without the base rate of the whole population it came from.

Worked example: does “cold outreach” make founders?

A founder tells you: “Nine of ten of us who got funded did aggressive cold outreach to investors. Cold outreach works.” Sounds like a recipe. But you can only judge X (“cold outreach”) if you fill in the whole 2×2 table — trait present or absent, crossed with succeeded or failed:

Now compute the actual success rates, the thing the survivor story hid:

• Success rate with cold outreach: 90 ÷ (90 + 891) = 9.2 percent
• Success rate without cold outreach: 10 ÷ (10 + 9) = 52.6 percent

The founder’s “nine of ten winners did it” was true — look at the top row, 90 versus 10 — and completely backwards as advice. Once the enormous bottom-left cell (891 people who did cold outreach and failed) walks back into the room, cold outreach goes from “what winners do” to something the successful founders did despite, not because. The trait that looked like a cause is, if anything, a mild curse. You cannot see any of this from the top row alone — and the top row is all a survivor ever shows you.

When to use it

Reach for the denominator hunt the instant someone quotes a statistic about a group of winners, survivors, or still-here things and asks you to copy what they share. If you can’t name the denominator — the full count of everyone the trait was measured across — you don’t have a rate, you have a rumour.

Technique 2 — Hunt the missing cohort (“seek the dead”)

Finding the denominator is a mindset. Seeking the dead is the fieldwork. Deliberately, stubbornly, go looking for the failures the filter erased: the delisted funds, the founders who quietly folded and took a salaried job, the manuscripts that got rejected, the patients who never came back for a follow-up, the products pulled from shelves. They exist. They’re just not the ones who show up to be interviewed.

The mental switch here is cohort thinking versus cross-section thinking:

• Cross-section (the trap): look at who’s here now — the funds currently in the database, the founders currently on stage, the buildings still standing. This is a snapshot of survivors, pre-filtered.
• Cohort (the fix): pick a starting group at time zero — every fund that existed in 2005, every founder who incorporated in 2015 — and follow that same group forward, counting the deaths as they happen. Ask “what happened to everyone who started?” rather than “how are the ones still here doing?”

The two questions sound almost identical and give wildly different answers, because the cross-section has silently dropped everyone who died along the way. A cohort keeps the tombstones in the count.

When to use it

Any time your data source is “the ones currently in X” — the current roster, the live database, the standing buildings, the open businesses on the street. Stop and ask what the starting cohort was and where its casualties went. If you can’t find the dead, at minimum assume they were worse than the survivors, because that’s usually why they’re gone.

Technique 3 — Reconstruct the filter

Sometimes you genuinely cannot dig up the failures — they’re lost to history, or nobody kept records. You’re not helpless. You can still reconstruct the filter: name the selection process explicitly and reason about what it must have removed. The one question that does the work:

What had to be true for this data point to reach me?

Every piece of evidence you receive passed through some gauntlet to get to you. A war story required its teller to survive the war. A “this stock is a great long-term hold” chart required the company to not go bankrupt. A glowing product review required the customer to bother writing one. Once you name the gauntlet, you can ask what it silently filtered out — and how deadly the filter was.

The strength of the filter sets how hard you distrust the survivors:

The deadlier the filter, the more the surviving sample is a monument to the filter itself rather than to any trait the survivors share. Wald’s bombers are the extreme case: the filter (getting shot down) was so lethal that the survivor damage-map was almost pure artifact.

When to use it

Whenever the failures are unrecoverable — history, anonymous dropouts, deleted records. You can’t count the dead, but you can name the filter that killed them and scale your skepticism to its lethality. A deadly filter means the survivors are almost all selection and almost no signal.

Technique 4 — Use a control or comparison group

To move from “spot the bias” to “actually test whether a trait causes success,” you need more than the winners who had the trait. You need the full picture, and specifically the two cells the survivor story always hides: people with the trait who FAILED, and people without the trait who SUCCEEDED. That’s the same difference-in-differences logic used to defuse other biases (regression to the mean leans on it too): compare a group that got the trait against a matched group that didn’t, and see if the difference holds up.

It’s the 2×2 table from Technique 1, promoted from an afterthought to a study design. A trait earns the label “cause” only when the with-trait group genuinely outperforms the without-trait group across the whole population — not merely when the winners happen to share it. Winners sharing a trait is the cheapest, most abundant, most misleading evidence there is; a control group is what upgrades it to something real.

When to use it

Any time you’re tempted to say “X causes success” from a pile of successful people who did X. Before you believe it, demand the control: the X-doers who failed and the non-X-doers who won. No control group, no causal claim — just a pattern among survivors.

Technique 5 — Prefer full-population data over testimonial data

Some data sources come pre-filtered for survivors; others don’t. When you have a choice — and you often do — prefer records that capture the whole population over testimony that self-selects the winners. The distinction:

• Testimonial data is generated by whoever chose to show up: interviews, memoirs, reviews, conference talks, “here’s how I did it” threads. Showing up requires surviving, so testimony is survivor-selected by construction. It’s a greatest-hits playlist — every track is a hit because misses don’t make the playlist.
• Full-population data is generated by a process that records everyone, winner or loser: a pre-registered cohort, a birth or death registry, a clinical-trial registry, a “every fund that ever existed” database, total box-office-of-everything-released rather than the list of blockbusters. The failures are baked in because the record was made before anyone knew who’d win.

The magic of a registry — especially a pre-registered one, logged before outcomes are known — is that nobody can quietly drop the embarrassing entries afterward. The losers are nailed down in advance. That’s precisely why pre-registration became the antidote to publication bias in science: it stops the file-drawer filter from deleting the failed studies.

When to use it

Whenever both kinds of source exist for your question, reach past the vivid testimony for the boring complete record. A dull registry that includes the failures beats a thrilling memoir that can’t, every single time. When only testimony exists, treat every conclusion as a claim about survivors until proven otherwise.

A checklist you can carry

You won’t run a 2×2 table in your head at a dinner party. But you can run four quick questions — a portable version of the whole toolkit that fits behind any claim you meet.

The whole course on one map

You’ve climbed the full ladder: the missing data, Wald’s bombers, the model in the wild, why your brain falls for it, and now the defences. Here is all five lessons compressed onto a single map — a recap to lock the arc in place.

One last check before the exam

Two mixed questions to make sure the toolkit is loaded, not just read.

Next up is the Final Exam — graded, one question at a time, one-way. Once you submit an answer it locks: no back button, no retries, no restart, and your score stays hidden until the end. Commit before you click. It pulls together all five lessons, so bring the whole kit.