Overfitting

Definition

Overfitting happens when a model learns the noise and quirks of its training data instead of the underlying pattern. It scores very well on the data it was built from but poorly on new data, so its apparent accuracy is misleading.

In plain terms

A student who memorizes last year’s exam answers does well on last year’s exam and badly on this year’s. The student fitted the questions, not the subject.

Models do the same when they are flexible enough to bend around every point. The fix is not a cleverer model but an honest test: measure performance on data the model never saw while it was being built or tuned.

Overfitting also happens in the researcher, not just the model. Try many variations, keep the best-looking one and report only that, and you have fitted the noise by selection.

Example

The numbers below are synthetic and illustrative. The same 20 noisy points are fitted with curves of increasing flexibility, then each curve is tested on 20 new points.

The most flexible curve is nearly perfect on its own data and by far the worst on new data. Training error alone would have picked the wrong model.

Common misunderstandings

• “Overfitting only happens with complex models.” Any model can overfit if you tune it enough against the same test data.
• “More data removes the problem.” It helps, but repeated testing against the same held-out data lets information leak back in. See also Look-ahead Bias, which inflates results in a different way.
• “Low training error shows quality.” It shows only that the model fits what it has seen.