Standard deviation in inter-rater fidelity

15 Views Asked by At

When dealing with Cohen's and Fleiss' kappa to measure inter-rater fidelity, one can use this formula to estimate the standard deviation: $$SD(\kappa)=\sqrt{\frac{P_o(1-P_o)}{(1-P_e)^2}}$$ where $P_o$ is the proportion of observed agreement and $P_e$ is the expected agreement.

However, if the observed data has a worst-than-random agreement, that is $P_o<0$, the given formula for $SD(\kappa)$ will not work as it is not a real number value.

My question is, is there a workaround to this problem?

I'm guessing that just by taking the absolute value inside the square root is not a valid approach... (?)

Thanks for your answers and comments!

1

There are 1 best solutions below

0
On BEST ANSWER

The question arose as a lack of attention, taking $\kappa$ as the value of $P_o$, I realized this thanks to the comments.

Thank you for your help!