Relevance Judgments

Definition

Relevance judgments are human assessments of how well each result answers a query, recorded as grades such as not relevant, partly relevant and highly relevant. They form the answer key against which search and ranking methods are scored.

In plain terms

A test collection is a set of queries, a set of documents and the judgments that link them. Evaluation campaigns such as TREC have used this design for decades. Judgments need written guidelines (what does each grade mean, for which kind of user), more than one judge for some of the items, and a check of how often judges agree (see Cohen's Kappa).

Nobody can judge every document for every query, so a sample of results is judged, often those returned by several systems (pooling). Results nobody judged are not known to be irrelevant, and treating them as such penalises systems that find new good results.

Example

Query “refund policy”; five results; grades 0 (not relevant), 1 (partly), 2 (fully). Two judges:

They agree exactly on 3 of 5 (60%) and are never more than one grade apart. The two disagreements point at a guideline gap: when is a page that mentions refunds only in passing “partly” relevant?

Common misunderstandings

• “Relevance is objective.” It depends on the user and the need behind the query; guidelines make judgments consistent, not objective.
• “Click data can replace judgments.” Clicks are shaped by what was shown first (see Position Bias).
• “An unjudged result is an irrelevant result.” It is an unknown one.