A model that outputs 80% confidence should be correct about 80% of the time. Expected Calibration Error (ECE) measures how far a model's confidence is from its actual accuracy, averaged across probability bins.
Given true binary labels, predicted probabilities, and a number of equal-width bins, compute the ECE.
Partition predictions into equal-width bins over [0, 1). For each non-empty bin:
ECE=m=1∑Mn∣Bm∣acc(Bm)−conf(Bm)where:
acc(Bm)=∣Bm∣1i∈Bm∑yiconf(Bm)=∣Bm∣1i∈Bm∑p^iA prediction p falls into bin index floor(p * n_bins), except p = 1.0 which goes into the last bin.
Return the ECE as a float.
Input:
y_true = [1, 0, 1, 0] y_pred = [0.9, 0.9, 0.9, 0.9] n_bins = 5
Output:
0.4
All samples fall in the same bin. acc = 0.5, conf = 0.9. ECE = |0.5 - 0.9| = 0.4. The model is overconfident.
Input:
y_true = [0, 0, 1, 1, 0, 1, 1, 1] y_pred = [0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9] n_bins = 2
Output:
0.125
Bin 0 ([0, 0.5)): acc=0.5, conf=0.25, diff=0.25. Bin 1 ([0.5, 1.0)): acc=0.75, conf=0.75, diff=0.0. ECE = (4/8)*0.25 + (4/8)*0.0 = 0.125
Be careful with the bin assignment when a prediction equals exactly 1.0.
Each bin's contribution to ECE must be weighted by the fraction of samples it contains.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number
A model that outputs 80% confidence should be correct about 80% of the time. Expected Calibration Error (ECE) measures how far a model's confidence is from its actual accuracy, averaged across probability bins.
Given true binary labels, predicted probabilities, and a number of equal-width bins, compute the ECE.
Partition predictions into equal-width bins over [0, 1). For each non-empty bin:
ECE=m=1∑Mn∣Bm∣acc(Bm)−conf(Bm)where:
acc(Bm)=∣Bm∣1i∈Bm∑yiconf(Bm)=∣Bm∣1i∈Bm∑p^iA prediction p falls into bin index floor(p * n_bins), except p = 1.0 which goes into the last bin.
Return the ECE as a float.
Input:
y_true = [1, 0, 1, 0] y_pred = [0.9, 0.9, 0.9, 0.9] n_bins = 5
Output:
0.4
All samples fall in the same bin. acc = 0.5, conf = 0.9. ECE = |0.5 - 0.9| = 0.4. The model is overconfident.
Input:
y_true = [0, 0, 1, 1, 0, 1, 1, 1] y_pred = [0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9] n_bins = 2
Output:
0.125
Bin 0 ([0, 0.5)): acc=0.5, conf=0.25, diff=0.25. Bin 1 ([0.5, 1.0)): acc=0.75, conf=0.75, diff=0.0. ECE = (4/8)*0.25 + (4/8)*0.0 = 0.125
Be careful with the bin assignment when a prediction equals exactly 1.0.
Each bin's contribution to ECE must be weighted by the fraction of samples it contains.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number