The Invasion Test

An evolutionarily stable strategy (ESS) is a behaviour that, once nearly everyone in a population is doing it, cannot be pushed out by a rare newcomer doing something else. That is a lovely sentence. It is also, on its own, useless — because “cannot be pushed out” sounds like a vibe, and evolution does not run on vibes. It runs on payoffs, and payoffs are numbers.

So this lesson has exactly one job: take the phrase “cannot be invaded” and turn it into an inequality you can check. Two lines of arithmetic, a best-reply condition and a tie-breaker, and you will be able to look at any strategy and say — not guess, say — whether a mutant could ever get a foothold.

The setup: residents, mutants, and E(A, B)

Before any inequality, we need one piece of notation, and we will lean on it for the rest of the course. Write E(A, B) for the payoff earned by a player using strategy A when its opponent uses strategy B. First slot: you. Second slot: them. That is the whole grammar.

Now name the cast:

• I — the resident (or incumbent): the strategy almost everyone is already playing. Think of it as the local custom.
• J — a rare mutant: some newcomer strategy, J ≠ I, that has just appeared in a tiny sliver of the population.

The question of the entire field of evolutionary game theory, boiled down: if the population is basically all I with a whisper of J, does J grow or shrink? If it always shrinks — for every possible J — then I is an ESS. It is a fortress no single mutant type can storm.

Why “rare” is the whole game

An ESS is defined against rare mutants — formally, in the limit as the mutant fraction shrinks toward zero — and against one mutant type at a time. This is not a technicality to skim past; it is the load-bearing assumption. “Evolutionarily stable” means locally uninvadable: safe from any lone newcomer that shows up in small numbers. It does not promise the population is globally best, happiest, or most efficient. (Hold that thought — it will bite in the very first worked example.)

Deriving the test from population arithmetic

Here is the beautiful part: you do not have to memorise the ESS definition. You can derive it from one honest fitness calculation, and then read the definition straight off the arithmetic.

Let the population be a fraction (1 − ε) residents playing I and a small fraction ε mutants playing J, where ε (epsilon) is tiny. A randomly chosen opponent is a resident with probability (1 − ε) and a mutant with probability ε. So the expected fitness of each type is just a weighted average over who it might meet:

$W(I) = (1-\varepsilon)\,E(I,I) + \varepsilon\,E(I,J)$

$W(J) = (1-\varepsilon)\,E(J,I) + \varepsilon\,E(J,J)$

The resident I resists invasion exactly when it out-reproduces the mutant — when W(I) > W(J) for all small ε > 0. Let us actually look at that comparison as ε shrinks toward zero.

The dominant terms. As ε → 0, the (1 − ε) weights swamp the ε weights. So the comparison is decided first by the big terms: E(I, I) versus E(J, I). For I to have any hope, you need

$E(I,I) \ge E(J,I).$

Two cases fall out, and they are the two conditions:

• If that inequality is strict — E(I, I) > E(J, I) — then for small enough ε, W(I) > W(J) no matter what happens in the tiny ε terms. The resident wins outright. This is Condition 1.
• If it is an equality — E(I, I) = E(J, I), a tie on the big term — then the big terms cancel and the decision falls to the ε terms. There, I wins iff E(I, J) > E(J, J). This is Condition 2.

That is the entire ESS definition, and we just found it rather than declaring it.

Notice the shape: Condition 1 is about the common matchup (everyone-vs-resident), and Condition 2 is the fine print for when that common matchup is a draw, decided by the rare matchup (mutant-vs-mutant). Big term first; if it ties, small term breaks it.

When to reach for this derivation

Any time someone hands you an ESS “definition” as a pair of inequalities to memorise, you can regenerate it in thirty seconds from W(I) > W(J) and the phrase “big term first, then the tie-breaker.” Understanding why the two conditions exist — one for the dominant matchup, one for the rare one — is what lets you apply them without second- guessing which slot goes where.

Worked example A: Always Defect is an ESS in the one-shot PD

Time to stop admiring the definition and plug numbers in. Recall the one-shot prisoner’s dilemma payoffs from earlier courses, with T > R > P > S and the standard values T = 5, R = 3, P = 1, S = 0.

Let the resident be AllD (“Always Defect”). A whole population defects. Now a rare mutant AllC (“Always Cooperate”) appears. Run the test.

The relevant matchups, all against the sea of defectors:

Check Condition 1: is E(I, I) > E(J, I)? That is 1 > 0. Yes — strictly. A resident scoring 1 against the crowd beats a mutant scoring 0 against the same crowd. We never even need Condition 2; the strict best-reply condition already slams the door. AllD is an ESS.

Worked example B: Always Cooperate is NOT an ESS

Now the mirror image. Resident is AllC; everyone cooperates, life is lovely. A rare AllD mutant appears. Run the same two-row test:

Check Condition 1: is E(I, I) > E(J, I)? That is 3 > 5. No. In fact E(I, I) < E(J, I) — the mutant does strictly better against the residents than the residents do. Condition 1 fails hard.

Does Condition 2 rescue it? No — Condition 2 only even applies when there is a tie on the main matchup, E(I, I) = E(J, I). Here 3 ≠ 5, so there is no tie to break. Neither condition holds. AllC is not an ESS — a single defector earns a growth premium and the mutation spreads. The lovely all-cooperate world gets eaten from the inside.

Worked example C: when the tie-breaker actually earns its keep

Both examples so far were settled on line one. To see Condition 2 do real work, we need a matchup that ties. Picture a coordination-style convention — say, two ways of doing the same thing that pay equally well when you meet the locals. Here are payoffs to the row player:

Read off the four numbers we need. Resident I, mutant J:

• E(I, I) = 2 — resident meets resident.
• E(J, I) = 2 — mutant meets resident. Tie with E(I, I)!
• E(I, J) = 2 — resident meets mutant.
• E(J, J) = 0 — mutant meets mutant.

Condition 1: is E(I, I) > E(J, I)? That is 2 > 2 — false. It is a tie, not a strict win. So line one cannot save I, and a lazier definition (“just be a best reply to yourself”) would shrug and call it a day.

Condition 2: the tie sends us to the fine print. We need E(I, J) > E(J, J), i.e. 2 > 0. True! So I is an ESS after all — rescued entirely by the second line.

The story behind the numbers is delightful. The mutant J is a perfect impostor against the crowd: it scores 2 against residents, exactly like a resident would, so it drifts in without penalty. But J has a fatal flaw that only shows up when two mutants happen to meet each other — then they score a miserable 0 instead of 2. As mutants become even slightly more common, those mutant-on-mutant encounters start happening, and J bleeds fitness. The residents, meanwhile, still score 2 even against the mutants. So J can never gain traction. The rare self-matchup is the trap that springs on the invader.

When to reach for Condition 2

Whenever the main matchup ends in a draw. Coordination games, conventions, and “which-side-of-the-road” problems constantly produce ties on E(I, I) = E(J, I), because the whole point of a convention is that the alternative pays the same when everyone’s still doing the old thing. In exactly those cases, line one is silent and the tie-breaker is the only thing standing between “self-enforcing custom” and “one drift away from collapse.”

Pitfalls: how the invasion test gets misused

Three ways smart people trip over this test:

1. Asserting stability instead of earning it. “Obviously that behaviour is evolutionarily stable” is not an argument. You must name the payoffs and plug in. The whole value of the invasion test is that it replaces intuition with E(I, I) versus E(J, I). If you have not written those numbers down, you have not checked anything.

2. Confusing local uninvadability with global optimality. An ESS is safe from rare mutants, one type at a time. That is all. It says nothing about whether the population is efficient, fair, or happy — AllD is Exhibit A. Never upgrade “stable” to “best.”

3. Mistaking the Nash condition for the ESS condition. Best-reply-to-itself — E(I, I) ≥ E(J, I), with a weak inequality — is precisely the Nash equilibrium condition. It is necessary for an ESS but not sufficient. ESS demands more: either a strict win on the main matchup, or the Condition 2 tie-breaker on top. The full comparison of Nash and ESS is the next lesson; for now, just don’t treat “it’s Nash” as “it’s an ESS.”

When to reach for the invasion test

Any time you can write down payoffs that depend on who you’re up against — the opponent’s type — the invasion test tells you which behaviours, conventions, or strategies are self-enforcing. That covers an enormous range: animal contests (fight or flee?), signalling, market conventions, social norms, the side of the road you drive on. If you can fill in a payoff matrix, you can run E(I, I) versus E(J, I), add the tie-breaker if it ties, and pronounce a verdict on stability. The test is small; its reach is not.

Recap