Mean Reciprocal Rank
Abbreviation: MRR.
The average, over many queries, of one divided by the position of the first correct result. It rewards finding one right answer early.
Synthetic dataExamples on this page use synthetic data created for illustration.
Definition
Mean reciprocal rank averages, over many queries, one divided by the position of the first correct result. It rewards finding a right answer early and ignores everything after it, so it suits tasks where the user needs a single good result.
In plain terms
For each query, find the highest-ranked correct result. If it is first, the query scores 1; second, 1/2; third, 1/3; not found at all, 0. The mean over all queries is the MRR.
It is easy to compute and to explain, and it is a natural fit for question answering or “find the right page”. It says nothing about the second correct answer and so is a poor choice when the user wants several good results; for that, a measure such as Normalized Discounted Cumulative Gain fits better.
Example
Four queries and the position of the first correct result:
| Query | First correct result at | Reciprocal rank |
|---|---|---|
| 1 | position 1 | 1.000 |
| 2 | position 3 | 0.333 |
| 3 | position 2 | 0.500 |
| 4 | not found | 0 |
MRR = (1 + 0.333 + 0.5 + 0) / 4 = 0.458.
Common misunderstandings
- “MRR measures the quality of the whole list.” It looks only at the first correct result.
- “Queries with no correct result can be left out.” They count as zero; dropping them flatters the score.
- “Position 2 is exactly half as good as position 1 for a user.” That is the assumption built into the measure, not a fact about people.
References
- Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze. Evaluation in information retrieval, in Introduction to Information Retrieval . Cambridge University Press, 2008.
- Nick Craswell. Mean Reciprocal Rank . Encyclopedia of Database Systems, Springer, 2009. DOI: 10.1007/978-0-387-39940-9_488