Expected Calibration Error
Expected Calibration Error
A model that outputs 80% confidence should be correct about 80% of the time. Expected Calibration Error (ECE) measures how far a model's confidence is from its actual accuracy, averaged across probability bins.
Given true binary labels, predicted probabilities, and a number of equal-width bins, compute the ECE.
Formula
Partition predictions into equal-width bins over [0, 1). For each non-empty bin:
ECE=m=1∑Mn∣Bm∣acc(Bm)−conf(Bm)where:
acc(Bm)=∣Bm∣1i∈Bm∑yiconf(Bm)=∣Bm∣1i∈Bm∑p^iA prediction p falls into bin index floor(p * n_bins), except p = 1.0 which goes into the last bin.
Return the ECE as a float.
Examples
Input:
y_true = [1, 0, 1, 0] y_pred = [0.9, 0.9, 0.9, 0.9] n_bins = 5
Output:
0.4
All samples fall in the same bin. acc = 0.5, conf = 0.9. ECE = |0.5 - 0.9| = 0.4. The model is overconfident.
Input:
y_true = [0, 0, 1, 1, 0, 1, 1, 1] y_pred = [0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9] n_bins = 2
Output:
0.125
Bin 0 ([0, 0.5)): acc=0.5, conf=0.25, diff=0.25. Bin 1 ([0.5, 1.0)): acc=0.75, conf=0.75, diff=0.0. ECE = (4/8)*0.25 + (4/8)*0.0 = 0.125
Hint 1
Be careful with the bin assignment when a prediction equals exactly 1.0.
Hint 2
Each bin's contribution to ECE must be weighted by the fraction of samples it contains.
Requirements
- Use equal-width binning over [0, 1)
- Skip empty bins in the summation
- Handle the edge case where a prediction is exactly 1.0
- Weight each bin's contribution by its proportion of total samples
Constraints
- 1 <= len(y_true) == len(y_pred) <= 10000
- y_true[i] is 0 or 1
- 0.0 <= y_pred[i] <= 1.0
- 2 <= n_bins <= 100
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number
Expected Calibration Error
Expected Calibration Error
A model that outputs 80% confidence should be correct about 80% of the time. Expected Calibration Error (ECE) measures how far a model's confidence is from its actual accuracy, averaged across probability bins.
Given true binary labels, predicted probabilities, and a number of equal-width bins, compute the ECE.
Formula
Partition predictions into equal-width bins over [0, 1). For each non-empty bin:
ECE=m=1∑Mn∣Bm∣acc(Bm)−conf(Bm)where:
acc(Bm)=∣Bm∣1i∈Bm∑yiconf(Bm)=∣Bm∣1i∈Bm∑p^iA prediction p falls into bin index floor(p * n_bins), except p = 1.0 which goes into the last bin.
Return the ECE as a float.
Examples
Input:
y_true = [1, 0, 1, 0] y_pred = [0.9, 0.9, 0.9, 0.9] n_bins = 5
Output:
0.4
All samples fall in the same bin. acc = 0.5, conf = 0.9. ECE = |0.5 - 0.9| = 0.4. The model is overconfident.
Input:
y_true = [0, 0, 1, 1, 0, 1, 1, 1] y_pred = [0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9] n_bins = 2
Output:
0.125
Bin 0 ([0, 0.5)): acc=0.5, conf=0.25, diff=0.25. Bin 1 ([0.5, 1.0)): acc=0.75, conf=0.75, diff=0.0. ECE = (4/8)*0.25 + (4/8)*0.0 = 0.125
Hint 1
Be careful with the bin assignment when a prediction equals exactly 1.0.
Hint 2
Each bin's contribution to ECE must be weighted by the fraction of samples it contains.
Requirements
- Use equal-width binning over [0, 1)
- Skip empty bins in the summation
- Handle the edge case where a prediction is exactly 1.0
- Weight each bin's contribution by its proportion of total samples
Constraints
- 1 <= len(y_true) == len(y_pred) <= 10000
- y_true[i] is 0 or 1
- 0.0 <= y_pred[i] <= 1.0
- 2 <= n_bins <= 100
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number