Hello,
I have a question regarding Cohen’s Kappa analysis on ELAN.
I was wondering why “kappa_ipf” is NOT computed for specific annotations in an .eaf file, while it is computed for those including unlinked/unmatched annotations.
What is the reason behind this difference?
Thank you for your time, and I look forward to your reply.
Valerie Fan
Hi Valerie,
kappa_ipf is the ipf variant of kappa , Cohen’s kappa computed using the iterative proportional fitting (IPF) procedure from Holle & Rein’s EasyDIAg method, and that procedure is specifically designed to incorporate unmatched annotations.
In the label “Overall results (incl. unlinked/unmatched annotations)”, is based on the EasyDIAg IPF method and corresponds to kappa_ipf in the table. It treats unlinked/unmatched annotations as real disagreements and includes them in the contingency table via the IPF procedure.
The second one, labeled “Overall results (excl. unlinked/unmatched annotations)”, corresponds to kappa (excl.). Here all unlinked/unmatched annotations are left out before computing kappa, so this value only reflects agreement on those annotations that could be matched across raters.
In short, kappa_ipf = kappa including unlinked/unmatched annotations , and kappa (excl.) = kappa excluding them (only overlapping annotations are considered)