The Value of a Real, Noisy Test (EVSI)

The crystal ball was a fantasy. Real tests — surveys, scans, pilots, references — are noisy: they’re right most of the time and wrong some of the time. So the real question isn’t “what’s a perfect answer worth?” (that’s EVPI, the ceiling we found last lesson: 160 on the launch decision). It’s “what’s this flawed test worth, and is that more than its price?” That number has a name — EVSI, the Expected Value of Sample Information — and computing it is where Bayesian updating finally earns its keep.

A test is just two conditional probabilities

Our market survey isn’t a crystal ball; it flashes one of two signals — a green light (“market looks good”) or a red flag (“market looks bad”) — and it’s sometimes wrong. Pin down “sometimes” with two numbers:

• Sensitivity — when the market really is good, how often does the survey correctly flash green? Say 80%.
• Specificity — when the market really is bad, how often does it correctly flash red? Say 80% too.

So the survey is right 80% of the time in each state and wrong the other 20%. It’s informative, but it’s no oracle. Now — exactly as in Bayesian updating — we turn those “probability of the signal given the state” numbers around into what we actually want: the probability of the state given the signal.

Step 1 — how often does each signal fire, and what does it mean?

Start from 100 imagined launches. 40 are truly good, 60 truly bad (the prior). The survey is 80% accurate each way:

Read the posteriors straight off the table:

• After a green light (44 of them), the market is truly good in 32: $32/44 \approx$ 73%. Your 40% prior jumped to 73%.
• After a red flag (56 of them), the market is truly good in just 8: $8/56 \approx$ 14%. Your prior fell from 40% to 14%.

Step 2 — would the answer change your action? (Yes — it flips)

Now the pivotal check from lesson one, applied to each signal. Plug each posterior back into the launch payoffs (+400 good, −300 bad):

After a green light (73% good): $EV(\text{Launch}) = 0.73 \times 400 - 0.27 \times 300 \approx 209$ That’s well above 0, so a green light says launch.

After a red flag (14% good): $EV(\text{Launch}) = 0.14 \times 400 - 0.86 \times 300 \approx -200$ Below 0, so a red flag says walk away.

The two signals lead to different actions — green→launch, red→walk. That’s the whole point: this test can flip your decision, so it clears the would-it-change-my- action bar and earns a positive value. (If both signals had said “walk away,” the test would be worthless, exactly like the surgeon’s scan.)

Step 3 — add it up: the pre-posterior expected value

Here’s the subtle move that trips people up. You must value the test before you know which signal you’ll get — because that’s when you decide whether to buy it. So average the value of acting-on-each-signal, weighted by how often each signal fires:

$EV(\text{with survey}) = \underbrace{0.44 \times 209}_{\text{green} \to \text{launch}} + \underbrace{0.56 \times 0}_{\text{red} \to \text{walk, banks }0} \approx 92$

With the survey you expect 92. Without it, acting on your prior, you walked away and banked 0 ($V_0 = 0$ from last lesson). So:

$\text{EVSI} = 92 - 0 = \mathbf{92}$

Step 4 — the only number that decides it: the net

EVSI is the test’s worth. The decision to buy it is itself an expected-value decision: compare the worth against the price. If the survey costs 30:

$\text{Net value} = \text{EVSI} - \text{price} = 92 - 30 = \mathbf{62}$

Positive, so buy the survey: it’s worth 92 in averted mistakes and costs only 30. But watch how fragile that is. If the same survey cost 100, the net is $92 - 100 = -8$ — now it’s an informative test you still shouldn’t buy, because it costs more than the decision it improves is worth. Worth and worth buying are different questions, and only the net answers the second.

See it move: the value-of-information lab

Everything above is one frozen frame. Drag the dials and watch the value breathe. Start where our example sits (prior 40% good, reliability 80%, price 30, payoffs +400 / −300) and notice the three bars: act now on the prior, buy the test (its information gain tinted on top, its price shaded off), and the crystal-ball ceiling. Then try to break it:

• Slide the prior toward 5% or 95% and watch the test’s value collapse to zero — the read-out flips to “this test cannot change your decision” because both signals now point the same way.
• Crank reliability up and down: more accuracy lifts EVSI toward the ceiling, but it can never punch through it.
• Push the price up until the net goes red: a worthwhile test becomes a waste.

Why EVSI is always below the ceiling

It’s worth seeing why the inequality EVSI ≤ EVPI can never break. Perfect information tells you the true state every time, so you act right every time. A noisy test sometimes lies — a false green light launches you into a bad market, a false red flag scares you off a good one. Those mistakes are exactly the value a perfect answer would have saved and a noisy one doesn’t. So a real test recovers only part of the ceiling, and the noisier it is, the smaller that part. Get the test up to 100% accurate and EVSI rises to meet EVPI exactly; drop it to a coin-flip 50% and EVSI falls to 0 — a test that’s right half the time tells you nothing.

You can now price a real, imperfect test end to end: turn its error rate into posteriors, check whether the answer flips your action, average across the signals to get EVSI, and subtract the price for the net. Next we step back from the arithmetic and ask the strategic question it all points to: when is information worth gathering at all? The answer is three conditions — and killing any one of them sends the value to zero.