Research & Validation

Loading or recomputing data for past periods after the fact — and recording when each value was really loaded, so history is not made to look earlier-known than it was.

Evaluating a rule or model by replaying it on historical data as if it had been used at the time. Informative only if the replay is honest.

A simple method, such as repeating the last value, against which a new method is compared. A result means something only relative to it.

Data that records two times for every fact: when it was true in the world, and when the system learned it.

Grouping items or people by a shared starting point, such as the month they first appeared, and following each group over time.

A change over time in the relationship between inputs and the outcome, so that a model or rule that once worked slowly stops working.

A method that turns texts or images into countable categories using explicit coding rules, so that claims about content can be checked and repeated.

Estimating how well a method will do on new data by repeatedly leaving part of the data out, training on the rest, and rotating.

Information that would not exist at prediction time finds its way into the data used to build or evaluate a model, so results look better than they can ever be in use.

Turning raw data into inputs a model or rule can use, such as ratios, rates of change and counts over windows. Each must be computable from what was known at the time.

A ranking of kinds of evidence by how well their design guards against error: reviews and randomised experiments above observational studies, anecdotes at the bottom.

Once an outcome is known, believing it was predictable and that you knew it all along. It makes the past look more certain than it was.

Data set aside before any building begins and used once, at the end, to estimate performance on unseen data.

Records that reach a pipeline after the period they describe has already been processed or reported. They are normal, so a pipeline needs a rule for them.

A study error where information that was not yet available at the time of a decision is used to evaluate that decision.

The more tests, rules or variants you try on the same data, the more likely some look significant by chance. The number of tries has to be counted.

When a model learns the noise in its training data instead of the real pattern, and so fails on new data.

How surprising the data would be if the null hypothesis were true. It is not the probability that the hypothesis is true.

Reading data as it was known at a past moment — not as it looks today after later corrections and additions.

Writing down, before seeing results, the question, the data and exactly how it will be analysed, and keeping that record unchanged.

Removing training observations whose time span overlaps the test period, and skipping a short stretch after it, so no information leaks between the two.

Human assessments of how well each result answers a query, recorded as grades. They are the answer key against which ranking methods are scored.

A result is reproducible when someone else, with the same data and code, gets the same result. It needs the data version, code, settings and environment to be recorded.

Evaluating a forecasting method by training on data up to a date, testing on the period just after, then moving the date forward and repeating.