TF-IDF
Also known as: term frequency–inverse document frequency, tf*idf.
A weight for a word in a document: how often it occurs there, multiplied by how rare it is across all documents.
Synthetic dataExamples on this page use synthetic data created for illustration.
Definition
TF-IDF weights a word’s importance in a document by multiplying how often it occurs there (term frequency) by how rare it is across all documents (inverse document frequency). Words common everywhere score low; words frequent here but rare elsewhere score high.
In plain terms
If every document contains “the”, seeing “the” tells you nothing. If only a few contain “heat pump”, seeing it tells you a lot. TF-IDF captures that: the inverse document frequency of a word is the logarithm of how many documents there are divided by how many contain the word, so rare words get large weights.
Summing the weights of the query’s words that appear in a document gives a simple relevance score. It is the ancestor of more refined formulas such as BM25, and shares their blind spot: it only sees the words, not their meaning.
Example
Four documents (N = 4) and the query “cheap paris flights”. The weight of a word is log₁₀(N ÷ documents containing it).
| Word | Documents containing it | Weight |
|---|---|---|
| paris | 3 | 0.125 |
| cheap | 2 | 0.301 |
| flights | 2 | 0.301 |
| Document | Query words it contains | Score |
|---|---|---|
| d1 “cheap flights to paris” | cheap, flights, paris | 0.301 + 0.301 + 0.125 = 0.727 |
| d3 “cheap hotels in paris” | cheap, paris | 0.426 |
| d4 “flights to rome” | flights | 0.301 |
| d2 “paris travel guide for first time visitors” | paris | 0.125 |
“paris” appears almost everywhere, so matching it counts for little; matching the rarer “cheap” and “flights” decides the order.
Common misunderstandings
- “A high TF-IDF means the word is important to the topic.” It means the word is distinctive for that document within this collection.
- “There is one TF-IDF formula.” Many variants exist (log bases, smoothing, length normalisation); results differ.
- “It understands synonyms.” A document about “automobiles” scores zero for the query “cars”.
References
- Karen Spärck Jones. A Statistical Interpretation of Term Specificity and Its Application in Retrieval . Journal of Documentation, 1972. DOI: 10.1108/eb026526
- Gerard Salton, Christopher Buckley. Term-weighting Approaches in Automatic Text Retrieval . Information Processing & Management, 1988. DOI: 10.1016/0306-4573(88)90021-0
- Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze. Tf-idf weighting, in Introduction to Information Retrieval . Cambridge University Press, 2008.