Tetlock’s Expert Prediction Study: What It Actually Found

Lychee University · Lesson 1

You have probably heard the claim: an expert predicting the future is about as accurate as a chimpanzee throwing darts.

It is a memorable image. It is also an incomplete account of Philip Tetlock’s research.

The short answer: Tetlock found limited average forecasting skill on many political and economic questions, with performance deteriorating over longer horizons. Some experts performed better than others. His findings do not justify dismissing every expert forecast. Tetlock explains those qualifications in this authorized excerpt from Superforecasting.

The memorable claim

“An expert is about as accurate as a chimp throwing darts.”

Catchy. Incomplete.

What it actually is

A chance benchmark

Scattered, not aimed. A forecast needs this kind of baseline before a confident voice means anything.

Sounds impressiveis not the same as beating chance

What was the Tetlock study?

Political psychologist Philip E. Tetlock recruited 284 professionals who offered commentary or advice on political and economic trends. He gathered forecasts, checked outcomes, and compared performance with non-experts and simple forecasting methods.

A scholarly review of his 2005 book, Expert Political Judgment: How Good Is It? How Can We Know?, reports 82,361 forecasts accumulated by 2003. This figure refers to forecasts, not necessarily that many distinct questions. Read the review by Adrian Tschoegl and J. Scott Armstrong, hosted by Wharton.

Who was in the study

0

professionals commenting on political and economic trends

What was collected by 2003

0

forecasts, checked against what later happened

Political and economic forecastingNot a test of Wall Street stock picking

The study concerned political and economic forecasting. Describing it as an experiment demonstrating that Wall Street traders cannot pick stocks changes the claim.

What does “dart-throwing chimpanzee” mean?

The chimp represents a chance-based forecasting benchmark. It is a metaphor, rather than evidence that an actual chimp competed against Tetlock’s participants.

Tetlock used the comparison himself. In a Wharton interview with Tetlock and Dan Gardner, they explain its role as a baseline for evaluating performance.

That distinction matters because beating a benchmark is more informative than sounding impressive. A forecast needs something meaningful to compete against.

What did the study actually find?

Longer horizons were harder

In his later account, Tetlock describes stronger performance on questions looking about one year ahead, with forecasts approaching chance performance three to five years out. This was a finding about the questions studied, rather than a universal expiration date for all predictions. Source: Tetlock and Gardner, Superforecasting.

Time ahead of the question

Closer questions left more room for skill

Stronger

About 1 year3–5 years

A finding about the questions Tetlock studied, not an expiration date for every prediction.

Thinking style mattered

Tetlock distinguished between people who relied heavily on one overarching explanation and those who considered multiple perspectives. Using Isaiah Berlin’s metaphor, he called them hedgehogs and foxes.

His original book emphasizes the advantage of flexible thinking, tolerance for uncertainty, and willingness to reconsider. Read the opening chapter from Princeton University Press.

Hedgehog

One big explanation

The story stays fixed, even when the question changes.

Fox

Politics

Economics

Base rates

New evidence

Several views. Room to change one of them.

A compelling performance is not an accuracy record

The same chapter describes tension between the qualities that make someone persuasive in public debate and those associated with sound judgment. Confidence alone is insufficient evidence of forecasting ability. Source: Expert Political Judgment, Chapter 1.

How should you evaluate a forecast?

Here is a practical exercise for this lesson. Before accepting a prediction, write down:

1. The event: What exactly has to happen?
2. The deadline: When will we know the outcome?
3. The probability: How likely does the forecaster think it is?
4. The benchmark: What would a simple alternative predict?
5. The result: Will someone return to score it?

Vague

Hard to check

• EventInflation will fall
• DeadlineEventually
• Probability—
• Benchmark—
• ResultNo one checks

Specific enough to score

Example

• EventCPI above 3%
• DeadlineDec 31, 2026
• Probability35%
• BenchmarkLast year’s rate
• ResultScore it later

For example, “inflation will fall eventually” leaves too much room for reinterpretation. A forecast specifying an inflation measure, a threshold, a deadline, and a probability is easier to evaluate.

One correct call still leaves an open question: was it skill or luck? Keep a record across many forecasts rather than choosing the most impressive anecdote.

What does this mean for prediction markets?

Apply the same questions when examining a Kalshi or Polymarket contract. Check its resolution rules, observation date, price history, and eventual outcome.

Kalshi or Polymarket contract

What has to happen, and when is it scored?

• ✓Resolution rules
• 2Observation date
• 3Price history
• 4Eventual outcome

Tetlock’s study still does not, by itself, show that a market outperformed experts.

Tetlock’s original study alone does not establish that a particular prediction market outperforms experts. That comparison requires matched questions, comparable forecast times, and a consistent scoring method.

To begin your own investigation, use our guide to download Kalshi trade history, or explore Polymarket historical data. Historical records let you inspect what a market implied before an outcome became known.

Frequently asked questions

Did Tetlock prove that experts are useless?

No. He explicitly rejected that interpretation. The findings varied across forecasters, questions, and time horizons. Read his explanation.

Was this the monkey stock-picking experiment?

No. This lesson concerns Tetlock’s political and economic forecasting research. The dart-throwing chimp serves as a metaphor for chance performance, not a stock portfolio selected by an animal.

Is Expert Political Judgment the same as Superforecasting?

No. Expert Political Judgment was published in 2005. Superforecasting, by Tetlock and Dan Gardner, followed in 2015 and discusses later research into improving forecasting, including the Good Judgment Project. See the publisher’s description.

Your takeaway

Before trusting a confident prediction, ask to see the record. Make your own forecasts specific enough that someone can check them later.