Cohen's Kappa
Cohen's Kappa
Raw accuracy can be misleading when evaluating agreement between two raters (or a model vs ground truth) because it ignores agreement that happens purely by chance. Cohen's Kappa adjusts for this.
Given two lists of labels from two raters, compute Cohen's Kappa coefficient.
Formula
κ=1−pepo−pewhere:
po=nnumber of agreements pe=k∑ncount of label k by rater 1×ncount of label k by rater 2The sum is over all distinct labels appearing in either rater's annotations.
If both raters assign the same label to every sample (making the denominator zero), return 1.0.
Return the kappa score as a float.
Examples
Input:
rater1 = [0, 1, 0, 1] rater2 = [0, 1, 0, 1]
Output:
1.0
Perfect agreement. p_o = 1.0, p_e = 0.5, kappa = (1.0 - 0.5) / (1 - 0.5) = 1.0
Input:
rater1 = [0, 0, 1, 1] rater2 = [0, 1, 1, 0]
Output:
0.0
Agreement is exactly at chance level. p_o = 0.5, p_e = 0.5, kappa = 0.0
Hint 1
p_e uses label frequencies from each rater separately, not combined frequencies.
Hint 2
Collect all distinct labels from both raters before computing p_e.
Requirements
- Compute observed agreement from matching labels at each position
- Compute expected agreement using label frequencies from each rater independently
- Support any number of distinct labels (not just binary)
- Handle the degenerate case where p_e = 1.0
Constraints
- 1 <= len(rater1) == len(rater2) <= 10000
- Labels are integers
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: array
Cohen's Kappa
Cohen's Kappa
Raw accuracy can be misleading when evaluating agreement between two raters (or a model vs ground truth) because it ignores agreement that happens purely by chance. Cohen's Kappa adjusts for this.
Given two lists of labels from two raters, compute Cohen's Kappa coefficient.
Formula
κ=1−pepo−pewhere:
po=nnumber of agreements pe=k∑ncount of label k by rater 1×ncount of label k by rater 2The sum is over all distinct labels appearing in either rater's annotations.
If both raters assign the same label to every sample (making the denominator zero), return 1.0.
Return the kappa score as a float.
Examples
Input:
rater1 = [0, 1, 0, 1] rater2 = [0, 1, 0, 1]
Output:
1.0
Perfect agreement. p_o = 1.0, p_e = 0.5, kappa = (1.0 - 0.5) / (1 - 0.5) = 1.0
Input:
rater1 = [0, 0, 1, 1] rater2 = [0, 1, 1, 0]
Output:
0.0
Agreement is exactly at chance level. p_o = 0.5, p_e = 0.5, kappa = 0.0
Hint 1
p_e uses label frequencies from each rater separately, not combined frequencies.
Hint 2
Collect all distinct labels from both raters before computing p_e.
Requirements
- Compute observed agreement from matching labels at each position
- Compute expected agreement using label frequencies from each rater independently
- Support any number of distinct labels (not just binary)
- Handle the degenerate case where p_e = 1.0
Constraints
- 1 <= len(rater1) == len(rater2) <= 10000
- Labels are integers
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: array