TensorTonicTensorTonic
Problems
Study PlansProjectsNewInterviewPricingFeedback
Problems
Loading...
1 / 1

Expected Calibration Error

Metrics & Evaluation
Medium

Expected Calibration Error measures the gap between predicted confidence and observed accuracy. Split probabilities into equal-width bins over [0,1][0,1][0,1]. A probability of exactly one belongs to the final bin.

For each nonempty bin, compute its mean target and mean predicted probability:

acc⁡(Bm)=1∣Bm∣∑i∈Bmyi\operatorname{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}y_iacc(Bm​)=∣Bm​∣1​i∈Bm​∑​yi​ conf⁡(Bm)=1∣Bm∣∑i∈Bmpi\operatorname{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}p_iconf(Bm​)=∣Bm​∣1​i∈Bm​∑​pi​

Then compute:

ECE⁡=∑m=1M∣Bm∣n∣acc⁡(Bm)−conf⁡(Bm)∣\operatorname{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|ECE=m=1∑M​n∣Bm​∣​∣acc(Bm​)−conf(Bm​)∣

Here, MMM is the number of bins, nnn is the number of predictions, yiy_iyi​ is a binary target, and pip_ipi​ is its predicted probability. Return ECE as a Python float.

Loading visualization...

Examples

Input: y_true = [1, 0, 1, 0], y_pred = [0.9, 0.9, 0.9, 0.9], n_bins = 5

Output: 0.4

Explanation: All predictions share one bin whose accuracy is 0.5 and mean confidence is 0.9.

Input: y_true = [0, 0, 1, 1, 0, 1, 1, 1], y_pred = [0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9], n_bins = 2

Output: 0.125

Hint 1

Use min(int(probability * n_bins), n_bins - 1) for the bin index.

Hint 2

Accumulate targets, probabilities, and counts separately for each bin.

Requirements

  • Assign every prediction to one equal-width bin
  • Place probability one in the final bin
  • Ignore empty bins
  • Return the weighted absolute calibration gap as a float

Constraints

  • The target and probability lists have equal nonzero length
  • Targets are zero or one
  • Probabilities lie between zero and one
  • The number of bins is a positive integer
Try Similar Problems
Isotonic CalibrationRoc CurveClassification MetricsLog Loss Per SampleConfusion Matrix Norm

Sign in to take notes on this problem

Case 1
Case 2

Accepts: array

Accepts: array

Accepts: number

You must run your code first.
PrevNext

Expected Calibration Error

Metrics & Evaluation
Medium

Expected Calibration Error measures the gap between predicted confidence and observed accuracy. Split probabilities into equal-width bins over [0,1][0,1][0,1]. A probability of exactly one belongs to the final bin.

For each nonempty bin, compute its mean target and mean predicted probability:

acc⁡(Bm)=1∣Bm∣∑i∈Bmyi\operatorname{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}y_iacc(Bm​)=∣Bm​∣1​i∈Bm​∑​yi​ conf⁡(Bm)=1∣Bm∣∑i∈Bmpi\operatorname{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}p_iconf(Bm​)=∣Bm​∣1​i∈Bm​∑​pi​

Then compute:

ECE⁡=∑m=1M∣Bm∣n∣acc⁡(Bm)−conf⁡(Bm)∣\operatorname{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|ECE=m=1∑M​n∣Bm​∣​∣acc(Bm​)−conf(Bm​)∣

Here, MMM is the number of bins, nnn is the number of predictions, yiy_iyi​ is a binary target, and pip_ipi​ is its predicted probability. Return ECE as a Python float.

Loading visualization...

Examples

Input: y_true = [1, 0, 1, 0], y_pred = [0.9, 0.9, 0.9, 0.9], n_bins = 5

Output: 0.4

Explanation: All predictions share one bin whose accuracy is 0.5 and mean confidence is 0.9.

Input: y_true = [0, 0, 1, 1, 0, 1, 1, 1], y_pred = [0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9], n_bins = 2

Output: 0.125

Hint 1

Use min(int(probability * n_bins), n_bins - 1) for the bin index.

Hint 2

Accumulate targets, probabilities, and counts separately for each bin.

Requirements

  • Assign every prediction to one equal-width bin
  • Place probability one in the final bin
  • Ignore empty bins
  • Return the weighted absolute calibration gap as a float

Constraints

  • The target and probability lists have equal nonzero length
  • Targets are zero or one
  • Probabilities lie between zero and one
  • The number of bins is a positive integer
Try Similar Problems
Isotonic CalibrationRoc CurveClassification MetricsLog Loss Per SampleConfusion Matrix Norm

Sign in to take notes on this problem

Case 1
Case 2

Accepts: array

Accepts: array

Accepts: number

You must run your code first.