Dear ELAN users,
I have a question regarding the Fleiss’ Kappa.
If I go to File > Multiple File Processing > Calculate Inter Annotator Reliability and I click on “by calculating Fleiss’ Kappa” (…) it tells me to select the tiers to compare in the corresponding files. After I select the tiers, I am not able to click on “finish”, as the button is not clickable. The only clickable options are “cancel” and “previous”. Why is that? Could someone help me with this issue?
Best regards and thanks in advance!
Ulya
Dear Ulya,
The Next and Finish buttons are disabled/enabled based on choices in the current and previous steps. There are many combinations possible of where tiers are and how they should be combined and when I just tried again it seems that for the Fleiss’ Kappa calculation not all possible combinations are correctly implemented. Resulting in either disabled buttons or errors in the final calculation step. This is something we have to fix.
You may have run into this problem but maybe there is another explanation. If you want you may send an example (set of) file(s), with a description of how you wish to compare, to me (han.sloetjes AT mpi.nl) so that I can confirm if the problem is due to a bug or to inconsistent settings.
-Han
As a follow up on this: it appeared that especially the combination of Fleiss’ kappa and manual tier selection is not correctly implemented. In this particular case, tier matching based on suffixes turned out to be (after a file exchange) a working alternative, but that may not always be the case.
We’ll try to fix this problem.
