Cohen's Kappa separated annotations

divya.kanekal
Thu, 20 Feb 2025 16:02:54 +0000

Hi Valerie,

With the above image it is more clear now what is the calculation you want to perform.

When you perform the kappa calculation on the above scenario, you will get the overall (all tiers) tiers result as well as results per pair combination if you select “Also generate and export agreement per tier pair”.
That is with 2 sets of 2 combined tiers, There will be 6 tier combinations , along with overall result ( all tiers )

You can choose paring of tiers to compare “based on prefix or suffix” to select the tiers by annotators. Then once you select the first tier by annotator the corresponding tier will be automatically selected. It is possible to calculate the kappa value with this scenario. But it is best if you carry out the calculation and see if indeed that is the output that you desired. Let me know.

Best Regards
Divya