Skip to content
Mental Models

Evolutionarily Stable Strategies

The Equilibrium Nobody Chose

Populations settle on stubborn mixes of behaviour — so much fighting, so much bluffing, a one-to-one sex ratio — and hold there, though nobody decides the proportions. Meet the evolutionarily stable strategy: the equilibrium selection finds by breeding, not reasoning, and a tour of how the whole course builds it.

10 min Updated Jul 10, 2026

A red deer stag on a Scottish hillside meets a rival at the edge of his harem. He could charge — antlers down, full weight, a fight that might win him the females or might leave him gored and dead. Instead, nine times out of ten, the two of them do something almost comical: they roar at each other, walk stiffly side by side sizing each other up, and then one backs down without a blow landed. The escalation to real violence is the exception, not the rule. Watch a whole population of stags across a whole season and the proportions are eerily steady — a certain fraction of contests end in a genuine fight, the rest in a bluff and a retreat. Nobody sets that ratio. No stag committee meets to decide how much fighting the herd can afford. And yet the blend holds, year after year.

That steadiness is everywhere once you look. Roughly half of most animal species are born male and half female — not because it’s fair, but because the ratio is pinned there. Some male fish guard territories while others sneak in to fertilise eggs, in stable proportions. Some plants grow tall to steal light while others stay short and cheap. In every case a population has settled into a mix of behaviours that no individual designed and no individual can profitably break. This course is about the single idea that explains all of it.

Tip:

The one-sentence version

An evolutionarily stable strategy (ESS) is a strategy that, once (almost) everyone in a population is playing it, cannot be invaded by any rare mutant strategy — anything different does worse, so selection wipes it out and the population holds. It is the equilibrium biology reaches by breeding, not by anyone reasoning their way to it.

The move that makes it work

Classical game theory — which you’ve met — imagines clever players choosing their best response to each other and landing on a Nash equilibrium, a state where no one can do better by switching. Beautiful, but it assumes reasoning: players who compute, anticipate, and choose. A deer does not compute. A gene does not anticipate. So how does a mindless population end up at an equilibrium at all?

Maynard Smith and Price’s answer, in 1973, was to swap the rational chooser for a population. Don’t ask “what would a smart player pick?” Ask instead: “here is a population where each behaviour reproduces in proportion to the payoff it earns — which mix can’t be pushed off balance by a rare newcomer?” The reasoning agent disappears; differential reproduction does the optimising for free. A strategy that pays better leaves more offspring, so it spreads; a strategy that pays worse dwindles. The equilibrium is reached by selection, not selected by reason — and that is the whole conceptual leap of this course. Hold onto it: an ESS is what natural selection settles into when the payoffs depend on who else is around.

Before you read — take a guess

A population of animals has settled into a steady mix of 'fighters' and 'bluffers' that returns to the same proportions whenever it's disturbed. In the ESS way of thinking, WHY is the mix stable — what holds it there?

Feel it settle with your own hands

Here is the model’s flagship, the Hawk–Dove game, and you can run it right now. Two animals contest a resource worth V. A Hawk always escalates and fights; a Dove displays and backs down if the other fights. When two Hawks meet they brawl, and the loser pays an injury cost C. When a Hawk meets a Dove, the Dove flees and the Hawk takes the whole resource. When two Doves meet, they share.

Each line below is a type’s fitness against the current mix of the population. Both slope downward — the more Hawks around, the worse it is to be anyone, because Hawks are dangerous to meet. Where the two lines cross, Hawk and Dove earn exactly the same, so neither can spread: that crossing is the ESS. Drop the population at any starting mix and press let selection run to watch it climb or fall to that stable point on its own.

Hawk–Dove invasion lab

Watch a population find its own equilibrium

Two animals fight over a resource worth V; the loser of a real fight pays an injury cost C. Set V and C, then drag the share of the population playing Hawk. Each line is a type’s fitness against the current mix — where they cross, Hawk and Dove earn the same, so neither can spread. That crossing is the evolutionarily stable strategy.

ESS · 33%all Doves50/50all HawksShare of the population playing HawkFitness (payoff per contest)
Hawk fitness Dove fitness

Reading the population

Stable Hawk share

33%

Hawk payoff now

-3.2

Dove payoff now

0.2

Can a rare mutant invade?

Doves invade ↓

Now injury outweighs the prize (C > V), and the ESS is a MIX: no pure strategy is stable. In a world of Hawks, fights are ruinous, so a rare Dove that ducks every brawl does better and spreads; in a world of Doves, a rare Hawk grabs every resource unopposed and spreads. The two pressures balance at exactly V/C Hawks — a stable polymorphism nobody designed.

Do three experiments. FIRST, as loaded (V = 4, C = 12, fighting is dangerous): the population starts at 90% Hawks, but that's too many — press 'let selection run' and watch it FALL to a stable mix, the ESS at V/C ≈ 33% Hawks, all by itself. SECOND, drag the Hawk share down near 5% and run again: now Hawks are rare and grab resources unopposed, so the population CLIMBS back up to the same 33%. The equilibrium pulls from both sides — that's what 'stable' means. THIRD, drag the injury cost C down below the resource value V (say V = 8, C = 4): now fighting is cheap, the crossing vanishes, and selection drives the population to ALL Hawk — pure aggression is the ESS when losing a fight barely hurts.

Notice what just happened, because it is the entire course in miniature. Nobody told the population where to go. You set the payoffs, dropped the animals at some arbitrary mix, and selection alone carried them to a proportion that then held against any push. From 90% Hawks it fell; from 5% it climbed; both times it stopped at the same spot — the mix where a rare mutant of either type does no better than the crowd. That is an ESS you watched happen, and a model you’ve watched run sticks far better than one you’ve only read about.

In the lab above with V = 4 and C = 12, you started at 90% Hawks and selection drove the share DOWN to about 33%. Why couldn't a population of 90% Hawks hold?

Why this earns a place in the latticework

The ESS looks like a narrow piece of theoretical biology. It is actually one of the most portable tools you can carry, for three reasons.

  • It replaces “what should a rational agent do?” with “what can’t be invaded?” — a question you can ask even when there is no rational agent. Markets, ecosystems, immune systems, populations of software bots, social norms: none of them “reason,” but all of them settle into equilibria you can test for stability. Wherever outcomes depend on what everyone else is doing, the invasion test applies.
  • It explains stable mixtures, not just single winners. Plenty of models tell you which one option is best. The ESS tells you when the answer is genuinely “a blend” — why a population holds a stable ratio of aggressive to peaceful, risky to cautious, common to rare — and predicts the exact proportion. That’s a different and often more realistic shape of answer.
  • It’s honest about waste. An ESS is stable, not optimal. The Hawk–Dove population settles into a mix that burns real resources on pointless fights — and can’t get out, because any individual who unilaterally stopped fighting would be exploited. Knowing that stable ≠ good, and why a population gets trapped, is one of the sharpest lessons the model teaches.

The map of the course

Five teaching lessons build the model from the definition up to its limits, then one exam locks it in:

  1. The Invasion Test — the precise definition of an ESS and the two-line inequality that decides it: a strategy is stable if it’s a best reply to itself, and, when it merely ties, does strictly better against the mutant than the mutant does against itself. The exact bar a strategy must clear to be uninvadable.
  2. Hawk–Dove: The Flagship Game — the full worked model. Why pure Hawk is invadable whenever C > V, why the ESS is the mixed proportion V/C, and the real numbers behind the fighting-vs-displaying trade-off.
  3. ESS and Nash: The Extra Bar — how an ESS refines the Nash equilibrium (every ESS is Nash, not every Nash is an ESS), and the two faces of a mixed strategy: one animal randomising, versus a stable polymorphism of pure types.
  4. Frequency-Dependent Selection — the deep engine: when a strategy’s payoff depends on how common it is, rare types get an advantage. The one-to-one sex ratio, left- and right-mouthed scale-eating fish, and the rock–paper–scissors lizards.
  5. Cooperation, Waste, and Where the Model Lies — ESS meets the repeated dilemma (Tit-for-Tat is almost an ESS; Always Defect is stable too — a bistable world), replicator dynamics as the engine, and an honest catalogue of the model’s limits: stability isn’t optimality, not every game has a pure ESS, and real populations break the tidy assumptions.

Then a Final Exam — graded, one question at a time, one-way: once you answer, it locks. No back button, no retries, 70% to pass.

How to use this course

One rule does most of the work: guess before you peek. Commit to an answer on every exercise before you reveal the explanation — the small sting of being wrong is what welds the idea into memory. And keep coming back to the lab: slide the value and cost, drop the population somewhere new, and press run until “the equilibrium selection finds by itself” feels obvious in your hands rather than clever on the page.

Next up: The Invasion Test — where we turn the loose phrase “can’t be invaded” into an exact inequality you can check on any strategy at all.

Mark lesson as complete