University · Lesson 9 · Prediction Market Mechanics
A 90% Prediction Lost. Was It Wrong?
Learn prediction market calibration with a simple example: why a 90% forecast can lose, how to compare probabilities with outcomes, and what to check.
Lychee University · Lesson 9
A market gives an event a 90% chance. The event doesn't happen. Someone posts the screenshot: “Prediction markets got it wrong again.”
The favored outcome lost. But a 90% forecast left room for that result.
The screenshot
90%
Event did not happen.
“Prediction markets got it wrong again.”
Room left by 90%
Nine of ten. The miss belongs in the forecast — it doesn’t, by itself, break calibration.
Calibration asks whether events assigned a given probability happen at roughly that frequency across many forecasts. If forecasts are well calibrated, events assigned 90% should happen about nine times in ten over a sufficiently large, relevant set. It doesn't tell you which tenth will fail.
What does a calibrated prediction market look like?
Imagine a hypothetical collection of binary markets. We record each forecast exactly one day before its event and later check the resolved outcome.
| Recorded probability | Number of forecasts | Events that happened |
|---|---|---|
| 50% | 100 | 50 |
| 70% | 100 | 70 |
| 90% | 100 | 90 |
50%
70%
90%
Ten of the 90% forecasts failed. In this constructed example, that is still calibrated.
The probabilities match the observed frequencies in this constructed example. Among the 90% forecasts, ten events failed to occur. Those losses are compatible with the group's calibration.
Drag the hit count for a hundred 90% forecasts on the toy reliability plot. Real data won't usually sit so neatly on the diagonal.
Observed rate
90%
Vs 90% forecast
0 pp
Close to the diagonal in this toy example. Real samples wobble — especially with fewer than 100 forecasts.
Sampling variation matters, especially with few forecasts. Ninety successes out of 100 is a useful observation, not proof that the forecasting process is perfectly calibrated. Metaculus explains calibration in terms of probabilities matching observed frequencies.
Can one result prove a forecast was good or bad?
One result gives you information, but cannot establish calibration.
A 99% forecast that fails deserves scrutiny: it expressed much more confidence than a 60% forecast. Yet neither one alone tells you whether a forecaster systematically overstates probabilities. You need the record, including ordinary successes and other failures.
Screenshot verdict
“They said 90%. It failed. Markets are broken.”
A 99% miss deserves more scrutiny than a 60% miss. Still, one outcome cannot establish whether probabilities are systematically too high.
The same applies in reverse. One successful 1% prediction doesn't establish exceptional forecasting skill. Highlighting only the most surprising win or loss selects the story after seeing the ending.
How is calibration different from picking winners?
Imagine a forecaster who assigns 51% to a hundred outcomes and gets all hundred right. It picked every winner, yet its 51% forecasts occurred far more often than stated.
Another forecaster assigns 90% to a hundred outcomes and gets ninety right. It picks fewer winners, but its probabilities match the observed frequency in this example.
Winner picker
Stated probability
51%
Observed rate
100%
Calibration gap
+49 pp
Perfect winner-picking in this example — and badly calibrated. Events at 51% happened 100% of the time.
A forecast can therefore have an impressive winner-picking record while describing uncertainty poorly. Calibration evaluates the probability attached to the prediction.
What should you check before comparing markets?
Start with the forecast time. A price recorded a month before an election answers a harder question than one recorded after most votes have been counted. Kalshi's calibration research explicitly examines time to resolution rather than treating every timestamp as equivalent. Read Kalshi's study summary.
1 month out
Harder
1 day out
Easier
Resolved
Outcome
A 65¢ price a month out is not the same claim as 65¢ after most votes are counted.
Then check the sample. Keep the market category, probability bands, number of observations, and treatment of related contracts visible. A hundred contracts tied to one election are not a hundred unrelated elections. Broad averages can also conceal differences between categories.
A hundred contracts tied to one election are not a hundred unrelated elections. Shared information can move them together.
For real prices, group nearby probabilities into stated ranges and report how many observations fall in each. Leave room for uncertainty rather than treating every deviation from the diagonal as a discovery.
How can you investigate it yourself?
Keep a dated record of the question, forecast probability, observation time, resolution criteria, and eventual result. Decide those rules before selecting examples.
- 1Question
- 2Forecast probability
- 3Observation time
- 4Resolution criteria
- 5Eventual result
Then ask: what happened across the other forecasts made with the same confidence?
For a deeper research application, explore Lychee's Kalshi political-market calibration study.
The next time a 90% favorite loses, ask a better follow-up: what happened across the other forecasts made with the same confidence?
Get started with Lychee.
Free to explore here · Jarvis included in every paid plan at launch
More reading
You Won 90% of Your Bets. Why Are You Still Losing?
misterrpink · 3 min read
· Research
When Do Prediction Markets Become Accurate? A Kalshi Political Market Lifecycle Analysis
misterrpink · 18 min read
· Research
Are 90% Prediction Markets Reliable? A Kalshi Political Calibration Study
misterrpink · 15 min read
· Research
Political Prediction Markets: What Historical Data Reveals About Election Forecasting, Polls, and Market Accuracy
misterrpink · 12 min read
Kalshi Shows 40¢. Why Did Your Order Average 42.8¢?
misterrpink · 3 min read
Polymarket Shows 65¢. Why Does Buying Cost 67¢?
misterrpink · 3 min read