Gradium × TensorTonic Voice Challenge is open for registration.Register now
Modules
20/31
Bradley-Terry

Contents

Bradley-Terry Model

How pairwise comparisons become scores, and why reward models and DPO are built on them.

Last updated 20 November 2026

What is the Bradley-Terry model?

The Bradley-Terry model is a probability model for pairwise comparisons. It gives every item a single score and predicts the probability that item i is preferred over item j from the difference between their scores, using the sigmoid function. It is named after Ralph Bradley and Milton Terry, who published it in 1952, and it is the model behind the reward models used in RLHF and the DPO loss.

P(i≻j)=esiesi+esj=σ(si−sj)=11+e−(si−sj)P(i \succ j) = \frac{e^{s_i}}{e^{s_i} + e^{s_j}} = \sigma(s_i - s_j) = \frac{1}{1 + e^{-(s_i - s_j)}}

Here s_i and s_j are the scores of the two items, and σ is the sigmoid function. Two equal scores give a probability of 0.5, a higher score for i pushes the probability towards 1, and a lower one pushes it towards 0. Only the difference between the scores matters, so adding the same constant to every score leaves every prediction unchanged.

What does a Bradley-Terry calculation look like?

Suppose player A has a score of 1.2 and player B has a score of 0.4. The difference is 0.8, so the model predicts that A beats B with probability σ(0.8), which is about 0.69, and that B beats A with probability about 0.31. The two probabilities always add up to 1.

P(A≻B)=σ(1.2−0.4)=11+e−0.8≈0.69P(A \succ B) = \sigma(1.2 - 0.4) = \frac{1}{1 + e^{-0.8}} \approx 0.69

The same calculation works for any kind of item. The items can be chess players, search results, products or two answers written by a language model, as long as each comparison has a winner and a loser.

How do you fit Bradley-Terry scores from comparisons?

You fit Bradley-Terry scores by maximum likelihood, choosing the scores that make the observed winners as probable as possible. For a list of comparisons where w beat l, the negative log-likelihood is a sum of binary cross-entropy terms on the score differences, so fitting the model is the same as training a logistic regression on pairs.

L(s)=−∑(w, l)log⁡σ(sw−sl)\mathcal{L}(s) = -\sum_{(w,\, l)} \log \sigma(s_w - s_l)

The gradient for each comparison is simple. The winner's score moves up by 1 − σ(s_w − s_l) and the loser's score moves down by the same amount, so comparisons the model already predicts well barely change the scores, while surprising results change them the most. The loss is convex, so gradient descent finds the best scores, and those scores are finite as long as no group of items has won every comparison against the rest. The cross-entropy lesson explains why this log loss comes from maximum likelihood.

The fitted scores combine every comparison at once. In the NumPy example below, A beats C in two of three games, but the model predicts that A beats C with probability 0.72, because A also beat B and B also beat C, and those results add evidence about A against C.

How do RLHF reward models use the Bradley-Terry model?

In RLHF, a reward model reads a prompt and an answer and outputs one score. It is trained on pairs where a labeler preferred one answer over another, and the Bradley-Terry model turns each pair into a loss on the difference between the chosen answer's score and the rejected answer's score.

LRM=−log⁡σ(r(x,y+)−r(x,y−))\mathcal{L}_{\mathrm{RM}} = -\log \sigma\big(r(x, y^+) - r(x, y^-)\big)

Here r(x, y⁺) is the score for the chosen answer and r(x, y⁻) is the score for the rejected one. When the two scores are equal, the loss is log 2, about 0.693. Training pushes the chosen score above the rejected one, and after training the reward model can score new answers that no labeler has seen, which PPO then uses to improve the language model. Because only differences are trained, a reward score on its own means nothing in absolute terms, which is the same constant-shift property as in the basic model.

How does DPO build on the Bradley-Terry model?

DPO keeps the Bradley-Terry form and removes the separate reward model. It replaces each answer's reward with β times the log-ratio between the probability the trained model gives that answer and the probability a frozen reference model gives it, so the language model itself plays the role of the reward model.

LDPO=−log⁡σ(βlog⁡πθ(y+∣x)πref(y+∣x)−βlog⁡πθ(y−∣x)πref(y−∣x))\mathcal{L}_{\mathrm{DPO}} = -\log \sigma\left(\beta \log \frac{\pi_\theta(y^+\mid x)}{\pi_{\mathrm{ref}}(y^+\mid x)} - \beta \log \frac{\pi_\theta(y^-\mid x)}{\pi_{\mathrm{ref}}(y^-\mid x)}\right)

The structure is the same as the reward model loss: a sigmoid of a difference between the chosen and the rejected answer, inside a negative log. The DPO lesson walks through the reference model, the role of β and one full training step.

How do you implement the Bradley-Terry model in NumPy?

The code below computes a win probability, the reward model loss for a batch of two pairs, and fitted scores for three items from nine comparisons. The loss uses np.logaddexp(0, −d), which equals −log σ(d) but stays accurate when the score difference d is large.

import numpy as np

def win_probability(s_i, s_j):
    return 1 / (1 + np.exp(-(s_i - s_j)))

def reward_model_loss(r_chosen, r_rejected):
    return np.mean(np.logaddexp(0, -(r_chosen - r_rejected)))

def fit_scores(comparisons, n_items, lr=0.1, steps=2000):
    s = np.zeros(n_items)
    for _ in range(steps):
        grad = np.zeros(n_items)
        for winner, loser in comparisons:
            p = win_probability(s[winner], s[loser])
            grad[winner] += 1 - p
            grad[loser] -= 1 - p
        s += lr * grad
        s -= s.mean()
    return s

print(round(win_probability(1.2, 0.4), 3))
print(round(reward_model_loss(np.array([2.0, 0.5]), np.array([1.0, 1.5])), 3))

comparisons = [(0, 1), (0, 1), (1, 0), (0, 2), (0, 2), (2, 0), (1, 2), (1, 2), (2, 1)]
scores = fit_scores(comparisons, 3)
print(np.round(scores, 2))
print(round(win_probability(scores[0], scores[2]), 2))
0.69
0.813
[ 0.47  0.   -0.47]
0.72

The first line matches the worked example. In the loss, the first pair has the chosen answer ahead by 1 and the second has it behind by 1, so the losses are about 0.313 and 1.313 and their mean is 0.813. The fitted scores are centered at zero, with A at 0.47, B at 0 and C at −0.47.

Frequently asked questions

What is the difference between the Bradley-Terry model and Elo?

Elo uses the same idea as the Bradley-Terry model. Both give each player a single rating and turn the rating difference into a win probability with a logistic curve. Elo writes the curve in base 10 with a scale of 400 rating points and updates ratings one game at a time, while the Bradley-Terry model is usually fit to all comparisons at once by maximum likelihood.

What are the assumptions of the Bradley-Terry model?

The Bradley-Terry model assumes that each item has one fixed strength, that the outcome of a comparison depends only on the difference between the two strengths, and that comparisons are independent of each other. The basic model also has no ties, although extensions such as the Rao-Kupper and Davidson models add them.

Why are Bradley-Terry scores only defined up to a constant?

The win probability depends only on the difference between two scores, so adding the same number to every score leaves every prediction unchanged. Implementations fix this by setting one score to zero or by keeping the scores centered at a mean of zero, which is what the code on this page does.

Why do reward models in RLHF use the Bradley-Terry model?

Human labelers find it easier and more consistent to say which of two answers is better than to give each answer an absolute score. The Bradley-Terry model turns those pairwise judgments into a training loss, so a reward model can learn a single score per answer from comparisons alone.

Practice this

On TensorTonic you implement the pieces of this model from scratch and test them in the browser.

The Cracking RL study plan continues with reward model, PPO, DPO and GRPO problems. Related posts in this series: RLHF, DPO and PPO.