Evidlake
Menu

Holdout Set

Also known as: test set, hold-out, out-of-sample data.

Data set aside before any building begins and used once, at the end, to estimate performance on unseen data.

Updated Last reviewed

Synthetic dataExamples on this page use synthetic data created for illustration.

Not adviceThis is educational material. It is not investment, business or other professional advice.

Definition

A holdout set is a portion of the data set aside before any model building and used once, at the end, to estimate how the finished model performs on unseen data. It stays trustworthy only for as long as nobody tunes anything against it.

In plain terms

A common arrangement is three parts: training data to fit the model, validation data to choose between versions, and a test (holdout) set touched at the very end. The test score is an honest estimate only if the holdout played no part in any decision. Each time it is used to choose, the choice learns a little about it, and its score becomes optimistic.

For data ordered in time, the holdout should be the latest period, not a random sample, because the real future comes after the past. Writing down the split rule before the analysis starts is the simplest way to keep oneself honest.

Example

1,000 labelled items: 600 for training, 200 for validation, 200 held out.

StageScore
Training data95%
Validation data82%
Holdout, used once80%

Now suppose the team tries 50 variants and keeps the one that does best on the holdout. Its holdout score may read 85%, but on fresh data it is still about 80%: the five extra points were selected, not earned.

Common misunderstandings

  • “The test set can be used to tune, just a little.” Every use for a decision makes its score less honest.
  • “A random split is fine for time-ordered data.” It lets the model see the future of the period it is tested on.
  • “A larger holdout is always better.” It also leaves less for building; the split is a trade-off.
  • Cross-validation

    Estimating how well a method will do on new data by repeatedly leaving part of the data out, training on the rest, and rotating.

  • Overfitting

    When a model learns the noise in its training data instead of the real pattern, and so fails on new data.

  • Data Leakage

    Information that would not exist at prediction time finds its way into the data used to build or evaluate a model, so results look better than they can ever be in use.

References

  1. Trevor Hastie, Robert Tibshirani, Jerome Friedman. The Elements of Statistical Learning . Springer, 2009. ISBN 978-0387848570.
  2. Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Roth. The Reusable Holdout: Preserving Validity in Adaptive Data Analysis . Science, 2015. DOI: 10.1126/science.aaa9375