Position Bias

Definition

Position bias is the tendency of people to click results simply because they appear higher on the page, whatever their quality. Click counts therefore reflect the ranking as well as relevance, and treating clicks as relevance judgments reinforces whatever order was shown.

In plain terms

Eye-tracking and click studies find that attention falls off steeply down a list. A result in position 1 is seen by almost everyone; a result in position 8 by few. So a high click count may mean “was shown first”.

This creates a feedback loop: results that rank high get clicks, clicks raise their rank, and better results lower down are never discovered. Ways out include experiments that swap positions, statistical click models that estimate the effect of position, weighting clicks by how likely they were to be seen, and relying on human judgments (see Relevance Judgments).

Example

The same result is shown to two equal groups of 100 visitors, in position 1 for one group and in position 5 for the other (synthetic numbers):

The result is identical, so the fivefold difference is the effect of position. Judging this result’s quality by its clicks in a live ranking would credit it with a quality that belongs to its place.

Common misunderstandings

• “More clicks means more relevant.” Clicks measure attention and relevance together.
• “Position bias only concerns the top of a list.” It acts throughout, and also depends on layout, images and ads.
• “It is too small to matter.” Differences of several times between positions are common.