Effect Size

Definition

An effect size is a number describing how big a difference or relationship is in practical terms, rather than whether it is statistically detectable. It can be a plain difference, a ratio or a standardised measure such as Cohen’s d, and should be reported together with its uncertainty.

In plain terms

Significance and size are different questions. A very large sample can make a trivial difference “statistically significant”; a small sample can fail to detect a large one. Reading only p-values confuses “we can detect it” with “it matters”.

Cohen’s d expresses a difference between two averages in units of the spread of the data: d of 0.2 is conventionally called small, 0.5 medium and 0.8 large, though what counts as meaningful depends on the field and the cost of acting on it.

Example

Two comparisons of average ratings, with a standard deviation of 0.8:

Test A proves a difference too small to act on; test B cannot tell whether a worthwhile difference exists. Only the effect size and its interval say which situation you are in.

Common misunderstandings

• “Significant means important.” It means detectable; importance depends on the size of the effect.
• “Not significant means no effect.” The sample may simply be too small to see it.
• “Standardised effect sizes need no context.” What is a small effect for one purpose may be decisive for another.