The Knowledge Problem: Why the Perfect Planner's Data Never Arrives

Here’s a fantasy that never dies. Take the smartest person alive — or, in the modern remake, the biggest AI ever trained — hand them a dashboard showing everything: every factory’s output, every warehouse’s stock, every shopper’s budget. Then let them run the economy. Set every price, route every truck, decide how much bread each city bakes. No waste, no duplication, no chaos. Surely that beats millions of people muddling along, each seeing only their own tiny corner?

In 1945, Friedrich Hayek published a short paper — “The Use of Knowledge in Society” — that killed this fantasy so thoroughly that people have spent eighty years trying not to understand it. His point was not that the planner isn’t smart enough, or that the computer is too slow. It’s stranger and more fatal: the data the dashboard needs never arrives. The knowledge required to run the whole economy, Hayek wrote, “never exists in concentrated or integrated form, but solely as dispersed bits of incomplete and frequently contradictory knowledge which all the separate individuals possess.” There is no dashboard, because there is nothing to put on it.

The economic problem is not what you think

Ask most people what “the economic problem” is and you’ll get something like: we have limited resources and unlimited wants, so we need to allocate cleverly. Fine as far as it goes — but notice the hidden assumption. It treats the problem as if some single mind already knows all the resources, all the wants, and all the techniques, and merely has to do the arithmetic of matching them up. If that were the real problem, it genuinely would be a math exercise. Feed the data to a big enough computer, press Enter, done. Hayek conceded this openly: if we possessed all the relevant information, the remaining problem “is purely one of logic.”

But we never possess it. Nobody does. The real problem is one step earlier:

Read that last line twice, because it reframes everything. The question is not “who should do the calculation?” It’s “how can a calculation happen at all, when the inputs live in a hundred million heads and refuse to leave?” A society that solves this is doing something no planner — human or silicon — can replicate by collecting forms.

Three kinds of knowledge the planner can’t collect

Why can’t the fragments just be reported upward? Because they come in three flavors, and each one resists collection for a different reason.

1. Local knowledge — the knowledge of time and place

Local knowledge is knowledge of particular circumstances of time and place: not “steel production statistics” but this machine, this shipment, this Tuesday. The warehouse foreman who can hear that the number-three conveyor is about to fail. The greengrocer who knows her Tuesday regulars buy twice the tomatoes when the weather turns. The shipping agent who knows of a tramp steamer sitting half-empty in a nearby port and can make a living purely by matching stray cargo to stray capacity.

None of this is grand knowledge. It looks almost too trivial to count as “economic data.” But Hayek’s insight is that it’s valuable precisely because it’s fleeting and particular. The foreman’s hunch matters this week; the half-empty steamer sails tomorrow. By the time such a fact is written up, passed to a regional office, aggregated, and forwarded to the ministry, it isn’t knowledge anymore — it’s history. Reporting it up the chain doesn’t transmit it; it embalms it.

2. Tacit knowledge — we know more than we can tell

The philosopher Michael Polanyi gave the second flavor its name: tacit knowledge — skill and judgment you demonstrably have but cannot state. “We know more than we can tell.” You can ride a bike, but you cannot write instructions that would let a novice balance on the first try. A baker knows by feel when the dough has been kneaded enough; a veteran trader smells that a counterparty’s story is fishy before she can say why; a machinist knows the lathe is “off” from a sound no manual describes.

This is the flavor that breaks the reporting fantasy completely. Local knowledge is at least stale when reported; tacit knowledge can’t be put on the form at all. There is no box on any census that captures the baker’s hands. An economy runs on oceans of this stuff, and every drop of it is invisible to any center — not hidden, not withheld, but untellable.

3. Dispersed preferences — wants nobody has finished discovering

The third flavor is the sneakiest: what each person actually wants, right now, at the prices they actually face. You might think people at least know their own preferences and could just be surveyed. But you don’t fully know yours until you’re standing in the shop. Would you still buy coffee at $30 a bag? You’d find out — by choosing. People discover their own trade-offs partly by choosing at posted prices, which means the “data” a planner would survey doesn’t fully exist before the market process runs. Asking people what they’d hypothetically want, at prices that don’t exist yet, generates fiction with decimal points.

Worked example: Hayek’s tin story

Now the most famous three paragraphs in the paper — the tin example. Walk it slowly, because it shows something that should feel slightly impossible.

Somewhere in the world, one of two things happens: a major tin mine floods, or someone invents a profitable new use for tin. You — a user of tin, making, say, food cans in Rotterdam — will never learn which. Here’s what you see instead: the price of tin rises from $10 to $14 a kilo. That’s it. One number moved.

What do you do? You don’t need a committee meeting. At $14, some of your uses of tin no longer pay. So you economize where it costs you least: switch that one product line to aluminium, start buying recycled scrap, redesign the lid to use 8 grams instead of 11. Meanwhile a toothpaste-tube maker in Lyon does her own version of the same arithmetic, a solder factory in Osaka does theirs, and a scrap dealer in Lagos suddenly finds it worth driving further to collect old tin. Millions of adjustments, each tailored to local conditions that only the adjuster can see — and every one of them points the same direction: use less tin, and give up the least valuable uses first.

Count what the price accomplished. It transmitted, to every tin user on Earth, exactly the instruction each of them needed — “tin is now scarcer relative to the demands on it; economize accordingly” — without anyone learning the flood happened, without anyone knowing who else adjusted or by how much, and without any office anywhere computing anything. Hayek’s own summary: it is a marvel that in the tin shortage, “without an order being issued, without more than perhaps a handful of people knowing the cause,” tens of thousands of people move in the right direction. The number is the message. The cause stays unknown — and, crucially, doesn’t need to be known.

Why “more data” doesn’t fix it

Every generation rediscovers the planner fantasy with new hardware. Punch cards, mainframes, big data, and now foundation models — each time, the pitch is the same: the old planners failed because they couldn’t process enough information; we can. But the knowledge problem was never a processing problem, and it’s worth being precise about the four separate reasons the data never arrives, no matter how large the intake pipe.

Aggregation destroys the particulars. Statistics work by throwing away detail — that’s their job. “Regional conveyor uptime: 97.3%” is exactly the kind of number a dashboard shows, and it contains zero trace of the fact that the number-three conveyor in this warehouse will fail this week. The foreman’s knowledge doesn’t survive being averaged. What the center can see is precisely what the center doesn’t need; what it needs is precisely what aggregation deletes.

Reporting is slow, and the knowledge is time-sensitive. The half-empty steamer sails tomorrow. Local knowledge has a shelf life measured in days or hours; bureaucratic reporting has a latency measured in weeks. By arrival, the report describes a world that no longer exists.

Tacit knowledge can’t be reported at all. No latency fix, no better form, no interview technique extracts the baker’s hands or the trader’s nose. This fraction of the economy’s working knowledge — a huge fraction — is permanently off the dashboard, not because collection is hard but because telling is impossible.

Incentives corrupt what does get reported. You did the incentives course, so you know what happens when a principal pays for a number: the agent produces the number. Factory managers rewarded for meeting quotas report meeting quotas — Soviet factories famously hit “tons of nails” targets by making a few absurdly giant nails. The planner’s inputs aren’t just stale and gappy; they’re strategically bent by everyone whose career depends on what the dashboard shows. The measure becomes the target, and the data becomes advertising.

The punchline: if you can’t gather it, you must mobilize it in place

Here is where the lesson hands the baton to the rest of the course. Follow the logic one step past the wreckage of the dashboard fantasy:

1. Good allocation requires the dispersed fragments — the foreman’s hunch, the baker’s hands, the shopper’s Tuesday wants.
2. The fragments cannot be moved to a center — stale, untellable, corrupted, or not yet discovered.
3. Therefore any system that works must leave the knowledge where it is and let each person act on their own fragment…
4. …while some signal coordinates the millions of local actions so they add up instead of colliding.

You already met the signal, working incognito, in the tin story: the price. One number that condenses “everything everyone else knows and wants about tin” into the only fact you need. In the next lesson we’ll put it under the microscope — Hayek called the price system “a kind of machinery for registering change,” a telecommunications network that lets a single index number move a million hands.

When to reach for this model

Reach for the knowledge problem whenever someone proposes replacing a decentralized process with a dashboard, a committee, or a clever central optimizer — in a government, a company, or your own team. Before debating whether the center would decide wisely, ask the prior question: does the knowledge this plan needs exist anywhere in collectible form? Who currently holds it, is it tellable, how fast does it rot, and what will the holders’ incentives do to it on the way up? Very often the honest answer is that the center would be optimizing beautifully over data that is stale, bent, or imaginary — and the “messy” decentralized version was quietly doing something no center can: using knowledge where it lives.

Recap

So: the fragments can’t be gathered, yet somehow the city gets fed — which means something out there is already solving the problem the dashboard can’t. Next lesson we meet it properly: prices as signals — Hayek’s telecommunications system, where one number does the talking so that a hundred million fragments of knowledge never have to travel at all.