Designing Scores You Can Explain: Rules, Weights, Unknowns and Reasons

Teams often need to rank candidates: which items deserve attention, which sources to trust first, which leads to follow. A score is the natural tool, and an opaque score is the natural way to start an argument: “why is this one on top?” If the answer is “the model said so”, the argument is lost, because nobody can inspect the reasoning, contest an input, or learn what to change. This article describes a simple design in which every score arrives with its reasons (Explainable Scoring).

The ingredients

A transparent scoring scheme has four parts, applied in this order:

Hard rules that exclude or require something outright (“not out of stock”).
Criteria with weights that add up to a score (Weighted Scoring Model).
A proper place for unknowns, so missing data are neither hidden nor treated as bad (Three-valued Logic).
Reasons returned with every result.

A worked example

Rank three items. The scheme, in plain words:

Rule: the item must be in stock. If it is known to be out of stock, exclude it. If stock is unknown, keep it but mark it.
Price rating: 1 − price / 40, so cheaper is better (price 15 gives 0.625).
Reviews rating: review count / 1,000, capped at 1.
Delivery rating: 1.0 for two days or fewer, 0.6 for up to five days, 0.2 beyond.
Weights: price 0.5, reviews 0.3, delivery 0.2.

Item	Price	Reviews	Delivery	In stock	Result
A	15	600	2 days	yes	0.5 × 0.625 + 0.3 × 0.6 + 0.2 × 1.0 = 0.69
B	22	1,200	4 days	unknown	0.5 × 0.45 + 0.3 × 1.0 + 0.2 × 0.6 = 0.65, “possible: stock unknown”
C	12	80	7 days	no	Excluded: out of stock (no score)

Look at what the output contains beyond the numbers. C is not given a low score; it is excluded by a named rule, and the reason says so. B is ranked, but flagged because one fact is missing; an alternative design would put it in a “needs data” list, and either is better than quietly treating “unknown” as “no” (which would silently exclude B) or as “yes” (which would silently trust it). A reader can verify A’s score in seconds.

Return the reasons

For every item, return a small record with: the final score or status; each criterion’s rating, weight and contribution; which rules fired; which inputs were missing; and the version of the scheme and of the input data. For A:

Criterion	Weight	Rating	Contribution
Price	0.5	0.625	0.313
Reviews	0.3	0.60	0.180
Delivery	0.2	1.00	0.200
Score			0.69

Now someone can ask useful questions: “what if reviews counted for more?”, “why is delivery weighted so low?”, “what would raise this item’s rank?” An explanation also has to be faithful: it must describe the calculation that actually happened, not a story generated afterwards (Explainable Scoring).

Weights are judgments: show the sensitivity

The scheme above ranks A above B. Change the weights to price 0.3, reviews 0.5, delivery 0.2 and the scores become A 0.69 (0.3 × 0.625 + 0.5 × 0.6 + 0.2 × 1.0 = 0.69) and B 0.76 (0.3 × 0.45 + 0.5 × 1.0 + 0.2 × 0.6): the order flips. Neither weighting is wrong; they express different priorities. The honest practice is to say which weights were used and why, and to report whether the ranking changes under reasonable alternatives. A ranking that flips with every small change in weights should be presented as a rough grouping, not a precise order.

Version the scheme and test it with fixed cases

Version the rules and weights like code. A score without the version of the scheme that produced it cannot be reproduced (Reproducibility).
Keep a small set of “golden” cases: items with known, hand-checked expected scores and reasons. Run them after every change to the scheme. If A’s score moves from 0.69 to 0.70, a person should have meant it.
Change one thing at a time, and record why.

Check the scheme against human judgments

A tidy, explainable score can still be a wrong one. Test it. Have people grade a sample of items for the purpose the score serves, on a stated scale, with written guidelines and a check that two people agree (Relevance Judgments, Cohen's Kappa). Then measure how well the score’s ranking matches the grades, using a measure that rewards putting the best first (Normalized Discounted Cumulative Gain). Compare with a plain baseline, such as sorting by one criterion (Baseline). If the elaborate scheme does not beat the baseline, the elaboration is not paying for itself.

Where simple rules fall short, and what to do

Rules and weights are easy to inspect but limited: they cannot capture subtle interactions, and they depend on the criteria you thought of. A learned model may rank better. If you use one, keep the explainable scheme as the benchmark and insist on the same discipline for the model: reasons with each result, versioned data, tests against judgments. Where decisions are consequential, there is a respectable argument for choosing the interpretable scheme even at some cost in accuracy, because a mistake you can see is a mistake you can fix (Rudin 2019).

A short checklist

Hard rules first; each exclusion names its rule.
Criteria rated on a common scale, with written weights that add to one.
Unknowns handled explicitly: flagged, held back, or listed separately, never silently turned into yes or no.
Reasons returned with every result, faithful to the calculation.
Sensitivity to the weights reported.
Rules versioned; golden cases rerun after every change.
Scores checked against human judgments and against a plain baseline.