Calculating inter-rater reliability in realtime

Clicking to calculate inter-rater reliability and checking the result in txt files takes some time and is not user-friendly for my students. Could it be inter-rater reliability tab with results shown while editing the file with two tiers, one for each rater?

Also, I’ve tried to replicate calculating Kappa and ratio of overlap with Python based on the elan source code and easydiag source code in matlab but I can’t get the same results. Is there more detailed explanation of the algorithm for calculating inter-rater reliability, I mean more detailed then in Holle and Rein 2015 paper?

Calculating inter-rater reliability is thought of as something which is performed after the annotation process and is usually performed on multiple files (even though it can be run on a single document) and almost always requires several configuration steps. So, I don’t really see this as a function available in a separate tab while editing the file.

I don’t think there is more detailed documentation available on the Kappa algorithm than the Holle and Rein paper. We have tried to implement it in as much as possible the same way in ELAN and tested it on a (limited) number of data sets.
So, in your Python case the differences in the results are not minor or trivial (let’s say on the level of rounding effects)?