Cohen's Kappa

Definition

Cohen’s kappa measures how much two raters agree beyond what chance alone would produce. It is 1 for perfect agreement, 0 for agreement no better than chance, and can be negative. It corrects the misleadingly high raw agreement that arises when one category is very common.

In plain terms

Raw agreement flatters: if 90% of items are “not relevant”, two raters who guess “not relevant” for everything agree 90% of the time. Kappa subtracts the agreement expected by chance, calculated from how often each rater uses each label: κ = (observed − expected) / (1 − expected).

Verbal scales such as “substantial agreement” are conventions, not rules. For more than two raters, related measures (Fleiss’ kappa, Krippendorff’s alpha) are used. Low agreement means the instructions are unclear or the task is subjective, and it limits how well any model can score against those labels.

Example

Two raters mark 100 documents as relevant or not:

Observed agreement = (20 + 65) / 100 = 0.85. A says “relevant” 25% of the time, B 30%, so chance agreement = 0.25 × 0.30 + 0.75 × 0.70 = 0.60. Kappa = (0.85 − 0.60) / (1 − 0.60) = 0.625: 85% raw agreement, but only a moderate-to-substantial real agreement.

Common misunderstandings

• “Kappa is accuracy.” It measures agreement between raters, not correctness against a truth.
• “The scale labels are rules.” The words attached to ranges of kappa are conventions; judge by what your use needs.
• “High agreement means the labels are right.” Two raters can share the same misunderstanding.