Where the Model Lies

Every good model has a shadow: the situations where following it faithfully leads you off a cliff. Value of information is one of the sharpest tools in decision analysis — but sharp tools cut the hand that holds them wrong. Over five lessons you learned to price a question: to ask what an answer would be worth before paying for it, to compare EVSI against a test’s price, and to remember that a test is worth exactly $0 if it can’t change what you do.

This capstone is about the other direction — the ways the model quietly lies to you, or rather, the ways you lie to yourself while waving the model as cover. Let’s start with a gut check.

The model assumes a frictionless world. Reality has friction.

The clean value-of-information formula imagines a tidy universe: you have one decision, one payoff, a test that arrives instantly for a known price, and a version of you who obediently does whatever the answer recommends. Every one of those assumptions can break. When it does, the number on your spreadsheet is confidently wrong — which is the most dangerous kind of wrong. Let’s walk the six ways it happens.

Pitfall 1 — Information is never free, and delay is the hidden invoice

Ask someone the cost of a test and they’ll name the sticker price: the $30 survey, the $5,000 lab panel, the two-week pilot. That’s the visible cost. The one that bankrupts decisions is delay — the price of the time the test takes while the world keeps moving.

Think of a surfer waiting for the “perfect” wave. Each wave he lets pass to gather more information is a wave he doesn’t ride. While you run the two-week pilot, the competitor ships, the hiring candidate takes another offer, the market window narrows, the acquisition target gets a second bidder. A good decision now frequently beats a better-informed decision later, because “later” has its own body count.

Return to our running launch example: prior 40% the product is good, payoffs of +$400k if it’s good and −$300k if it’s bad (in thousands). The 80%-accurate survey has an EVSI of 92 and costs 30, for a tidy net of +62. But suppose running it burns a month, and in that month a rival launches something similar and shaves $150k off your upside whether the product is good or bad. Now the survey’s real price isn’t 30 — it’s 30 plus the erosion of the very payoff you’re trying to protect. The test can be net-positive on paper and net-negative in the world.

Pitfall 2 — Analysis paralysis: data past the decision threshold is procrastination with footnotes

Once you have enough information to pick an action, any further information that can’t flip that action is pure waste. Recall the change-my-action test: if every plausible next result points to the same move, you’ve already decided — you’re just refusing to admit it. Analysis paralysis is gathering more information past the point where it could change your decision: waste dressed as diligence.

You’ve seen the species. The “let’s just be sure” that was already sure. The fourth focus group after the first three agreed. The vanity study commissioned to confirm what the boss already wants to hear. Each extra week of “rigor” feels responsible, but if no realistic finding would change the call, the rigor is theatre — and it’s carrying Pitfall 1’s delay costs the whole time.

Here’s the tell: before you commission the next study, write down what you’d do for each possible result. If the answer is “the same thing regardless,” stop. You are not being careful. You are being slow.

Pitfall 3 — A flawless test of the wrong question

A test can be exquisitely precise and still worth zero, because accuracy on the wrong variable is worth exactly as much as accuracy on any other irrelevant fact: nothing. Value of information is defined against your decision — the payoff of your best choice with the info minus your best choice without it. If the measured variable doesn’t move that difference, the difference stays zero no matter how many decimal places your instrument reports.

This is the streetlight effect: searching for your keys under the lamp because that’s where the light is, not because that’s where you dropped them. Teams measure what’s cheap and clean — click-through rates, NPS to two decimals, server latency — while the decision actually turns on something awkward and unmeasured, like whether enterprise buyers trust the brand. A hyper-accurate reading of the easy variable is a beautifully calibrated answer to a question nobody asked.

Concretely: you’re deciding whether to greenlight a product, and the decision hinges on demand. You commission a study that measures manufacturing defect rates to ±0.1%. Impeccable. Useless — because within the range you’d actually see, defect rate doesn’t change the greenlight. The fix: start from the decision and work backward. Name the one or two variables whose values would actually flip your action, and test those, accepting a noisier answer on the right question over a razor-sharp answer to the wrong one.

Pitfall 4 — Base rates: how a rare target turns an accurate test into a coin toss

This is the subtle one, and it’s where people wildly overvalue tests. When the thing you’re testing for is rare, even a highly accurate test produces a flood of false positives — so a “positive” result moves your belief far less than the headline accuracy suggests. You met this in the Bayesian-updating lesson as false-positive dilution; here it becomes a pricing error.

Suppose you’re screening startups for a trait that only 1 in 100 genuinely has, using a test that’s 90% accurate both ways. Screen 1,000 companies: 10 truly have the trait, and the test flags about 9 of them. But of the 990 that don’t, it falsely flags 10% — about 99. So a “positive” points at roughly 9 real signals buried in 99 false alarms: your posterior is about 9 in 108, or ~8%. The test felt 90% reliable and delivered an 8% confidence. If you priced this test as if a positive nearly settled the question, you overpaid by an order of magnitude.

The deeper lesson: EVSI depends on the base rate, not just the test’s accuracy. A 90% test on a coin-flip question is worth a great deal; the same 90% test on a one-in-a-hundred question is worth a fraction of it. People forget the prior, anchor on the accuracy, and pay for signal they aren’t getting.

Pitfall 5 — VOI assumes you’ll actually act on the answer

The entire model rests on a quiet premise: that different answers lead to different actions by you, in reality. Information you’ll rationalise away is worth nothing. Information that arrives after you’ve already committed the budget, signed the lease, or announced the plan is worth nothing. If, deep down, you know you’re launching no matter what the survey says, then the survey’s value isn’t 92 or 30 — it’s $0, and the honest move is to skip it and own the decision.

We’re excellent at commissioning information for cover, then discarding it when it disagrees. The board that orders due diligence on a deal it has already emotionally closed. The founder who A/B tests but ships the version he liked regardless. That’s not decision-making; it’s evidence-shopping. And it’s worse than free, because it costs money and time (hello again, Pitfall 1) to manufacture a permission slip.

Pitfall 6 — The model’s own edge: clean numbers, messy world

Even used perfectly, VOI as usually taught assumes a single, well-defined decision with a known payoff distribution. Push past that and the clean number bends. Under deep uncertainty — where you can’t even list the outcomes, let alone their probabilities — an EVPI of “160” is false precision dressed as rigor. And sometimes the act of testing changes the outcome: floating a price to gauge demand tips off competitors; polling employees about a reorg starts the rumour that reshapes the answer; measuring a system perturbs it. In those cases the thing you measured isn’t the thing you’ll decide on.

So hold value of information as a discipline, not a decimal. Its enduring gift isn’t the exact figure — it’s the habit of asking “what would this answer be worth, and would it change what I do?” before reaching for your wallet.

Sort the tests: which ones could earn their keep?

Time to make the change-my-action test reflexive. For each scenario, decide whether the information could actually change the action — Worth testing — or whether it lands in the same place regardless, making it Worth nothing no matter its accuracy or price.

Pin down the vocabulary

Before the exam, make sure the five core terms are crisp. Match each to its precise definition.

Recap quiz: spot the trap

One more check

The healthy stance

Strip away the traps and a simple discipline remains. Value of information is worth reaching for when a decision is close (the options are genuinely competitive), high-stakes (being wrong is expensive), genuinely uncertain (you don’t already know the answer), and actionable (the result will change what you do, in time to do it). Miss any one of those and the smart move is usually to decide now and skip the study.

When those conditions are met, still do two things before you buy: price the test honestly — sticker price plus the cost of delay and any side effects of testing — and remember that worth ≠ worth-buying. An answer can be genuinely worth 92 and still be a bad purchase at a price of 100, or a delay of a month, or when you’d have acted the same anyway.

That’s the whole model in two sentences: information is only worth what it changes about your best decision, and you should pay for it only when that change, net of every cost including time, is worth more than the price. Everything else — EVPI as a ceiling, EVSI as the realistic value, the change-my-action test as the fast screen — is bookkeeping around that one idea.