Evidlake
Menu

Data Leakage

Also known as: target leakage, information leakage, train-test contamination.

Information that would not exist at prediction time finds its way into the data used to build or evaluate a model, so results look better than they can ever be in use.

Updated Last reviewed

Synthetic dataExamples on this page use synthetic data created for illustration.

Not adviceThis is educational material. It is not investment, business or other professional advice.

Definition

Data leakage is when information that would not be available when a prediction is really made finds its way into the data used to build or evaluate a model. The results then look better than they can ever be in actual use, and often look spectacular.

In plain terms

Common forms: a feature that is really a consequence of the outcome (target leakage); preparation done on all the data before splitting, so the test set influenced training; the same item appearing on both sides of a split; and information from after the decision date (Look-ahead Bias is the time version).

One question catches most of it: “would I have this exact value at the moment the prediction is made?” Results that look too good, or a feature that tops the importance list without an obvious reason, are warning signs. Leakage has been found behind many published machine-learning results that later failed to reproduce.

Example

A model predicts whether a customer will cancel. One column is cancellation_reason.

FeatureAvailable when predicting?Verdict
Months as a customerYesFine
Support tickets last monthYesFine
cancellation_reasonNo: it is filled in only after someone cancelsLeak

With the leaking column the model scores 99% in testing, because “has a reason filled in” is just another way of saying “cancelled”. In use the column is empty for everyone and the model is useless.

Common misunderstandings

  • “Leakage is a competition problem, not a real-world one.” It is among the most common reasons a model works in testing and fails in use.
  • “Cross-validation prevents leakage.” Not if scaling, feature selection or gap-filling is done before the folds are made.
  • “A very high score means a very good model.” It more often means something is leaking.
  • Look-ahead Bias

    A study error where information that was not yet available at the time of a decision is used to evaluate that decision.

  • Cross-validation

    Estimating how well a method will do on new data by repeatedly leaving part of the data out, training on the rest, and rotating.

  • Purging and Embargo

    Removing training observations whose time span overlaps the test period, and skipping a short stretch after it, so no information leaks between the two.

  • Point-in-Time

    Reading data as it was known at a past moment — not as it looks today after later corrections and additions.

References

  1. Shachar Kaufman, Saharon Rosset, Claudia Perlich, Ori Stitelman. Leakage in Data Mining: Formulation, Detection, and Avoidance . ACM Transactions on Knowledge Discovery from Data, 2012. DOI: 10.1145/2382577.2382579
  2. Sayash Kapoor, Arvind Narayanan. Leakage and the Reproducibility Crisis in Machine-learning-based Science . Patterns, 2023. DOI: 10.1016/j.patter.2023.100804