Selection Bias

Definition

Selection bias arises when the units that end up in your data differ systematically from the group you want to describe, because of how they were chosen or retained. Results then hold for the selected group and may fail for everyone else.

In plain terms

It comes in many forms, and several have their own names: the frame leaves people out (Sampling Frame), people choose to take part (self-selection, such as customers who write reviews), the survivors are kept and the failures vanish (Survivorship Bias, Attrition Bias), or units are selected because of the outcome (a “top 100” list).

It cannot be repaired by adding more data from the same process. The first question to ask of any table is “how did these rows get here?”, and the second is “who is not in it?”.

Example

A marketplace has 1,000 buyers of a product. A random survey of 200 buyers finds average satisfaction of 3.9 stars. Only 50 buyers volunteered a rating on the product page, mostly delighted or furious.

Averaging the published ratings overstates the satisfaction of the buyers as a whole by half a star, and nothing in the published data shows it.

Common misunderstandings

• “A large dataset avoids selection bias.” A large dataset of a selected group is a precise description of that group.
• “If it looks random it is unselected.” Selection often leaves no visible trace in the data itself.
• “It only matters in surveys.” Any table built from a filtered, ranked or voluntary process is exposed.