Calibration
Also known as: probability calibration, reliability diagram, overconfidence.
Events forecast at p% happen about p% of the time. Calibration is separate from accuracy: an overconfident forecaster is not calibrated.
Synthetic dataExamples on this page use synthetic data created for illustration.
Not adviceThis is educational material. It is not investment, business or other professional advice.
Definition
A forecaster or model is calibrated when events it gives a probability of p% actually happen about p% of the time. Calibration is separate from accuracy: a calibrated forecaster can still be uninformative, and an accurate but overconfident one is not calibrated.
In plain terms
Check it by grouping forecasts by stated probability (all the “90%” calls together, all the “70%” calls) and comparing each group’s stated probability with how often the events actually happened. A chart of this is called a reliability diagram.
People and many machine-learning models tend to be overconfident: their 90% events happen less than 90% of the time. Practice with prompt feedback improves it, which is a main finding of research on forecasting tournaments, and models can be recalibrated after training. Calibration alone is not enough; a forecaster who always says “50%” is perfectly calibrated for a coin and useless otherwise. Scores such as the Brier Score reward calibration and sharpness together.
Example
100 forecasts of whether items will rise, with the stated chance and what happened:
| Stated probability | Forecasts | Events that happened | Observed frequency |
|---|---|---|---|
| 90% | 20 | 15 | 75% |
| 70% | 40 | 29 | 72.5% |
| 50% | 40 | 21 | 52.5% |
The 70% and 50% calls are well calibrated; the 90% calls are overconfident by 15 points. The fix is not to lower every forecast but to learn what the “90%” feeling is really worth.
Common misunderstandings
- “High accuracy means calibrated.” A model can be right most of the time and still state 99% when it deserves 80%.
- “A calibrated forecaster is a good one.” Always saying the base rate is calibrated and tells you nothing new.
- “Only machine-learning models need it.” Human forecasts are the classic case.
References
- Glenn W. Brier. Verification of Forecasts Expressed in Terms of Probability . Monthly Weather Review, 1950. DOI: 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
- Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger. On Calibration of Modern Neural Networks . Proceedings of the 34th International Conference on Machine Learning, 2017.
- Philip E. Tetlock, Dan Gardner. Superforecasting: The Art and Science of Prediction . Crown, 2015. ISBN 978-0804136693.