Cohen's Kappa separated annotations

valerie
Wed, 07 May 2025 20:57:16 +0000

Hi Divya,

It’s been a while since I last contacted you, I hope you’re doing well!

Regarding where we left off and to provide some context, last time I asked whether I am able to conduct kappa analysis on 2 sets of symbolically associated tiers in which you showed me the method in a Zoom meeting we had.

However, I ran into another problem with this analysis. In the .txt file that was generated, I noticed that the number of pairs of tiers in the comparison is 6, but not 2. Please correct me if I’m wrong, but I believe this is due to the comparison of all possible pairwise relationships across all 4 individual tiers.

Is there any way that I can specifically choose what tiers I want to compare in the kappa analysis? For example, in this screenshot below:

Can I manually select “JP_annot” and “ML_annot” as 1 comparison, and then “sub_JP_annot” and “sub_ML_annot” as 1 comparison, resulting in 2 comparisons done within 1 kappa analysis and generating 1 .txt file with kappa values reflecting both comparisons at the same time?

Thank you for your assistance and I look forward to your reply.

Kind regards,
Valerie Fan