Does this system calculate inter-rater reliability? such as Kappa?
The current version of ELAN doesn’t have such functionality built-in. So users have to export the data to some other tool for that kind of calculations.
We have been working on such function recently though; it is almost ready for inclusion in the next release.
So far we integrated an algorithm for “modified (Cohen’s) kappa” as proposed by Holle and Rein and implemented in the EasyDIAg tool. And we’ve integrated a library named Staccato which calculates a Degree of Organization (proposed by Thomann) strictly based on the segmentation (ignoring the label of annotations).
And we still have the old compare annotators function that calculates the ratio of overlap and total extent of overlapping annotations.
-Han.