Cross-validation

Definition

Cross-validation estimates how well a method will do on new data by repeatedly splitting the data: train on most of it, test on the part left out, and rotate until every part has been left out once. The scores are then averaged.

In plain terms

In k-fold cross-validation the data are cut into k parts (folds). Each fold takes a turn as the test set while the rest train the model; the k scores give an estimate and a sense of its spread. It uses small datasets more efficiently than a single holdout.

Two traps are common. Any preparation that learns from the data (scaling, selecting features, filling gaps) must happen inside each training fold, or information leaks (Data Leakage). And when observations are not independent, as with time series or repeated measurements of the same item, random folds mislead; use folds that respect the structure (Walk-forward Validation).

Example

5-fold cross-validation on 500 items (100 per fold), accuracy per fold:

Mean 0.80, standard deviation about 0.03: the estimate is “about 80%, give or take a few points”. A single split could easily have produced 0.77 or 0.84 and told a different story.

Common misunderstandings

• “The cross-validation score is the final, honest test.” If it was used to choose between many options, keep a separate holdout for the end.
• “Random folds are fine for any data.” Dependent data need folds that keep related observations and time order together.
• “Scaling the whole dataset first is harmless.” It lets the test fold influence the training; do it inside each fold.