Hello, my name is Valerie and I have 2 questions about the flexibility of Cohen’s Kappa analysis.
First, can 1 annotation (i.e. annot_A_and_B) be separated into 2 annotations (i.e. annot_A, annot_B) within the same tier at the same period?
If yes, is it possible to compute Cohen’s Kappa values for the 2 annotations simultaneously? (i.e. kappa of union(annot_A, annot_B)). In other words, is it possible to compute 1 kappa value for 2 annotations simultaneously?
Thank you for your time and assistance!
Hello Valerie,
If I understand your question correctly, you want to have two annotations on a single time interval on the same tier ?
In ELAN that is not possible. A time interval or a segment is equivalent to one annotation.
The selection of tiers for calculating Kappa values always asks for two different tiers.Hence I am afraid it is not possible to perform the scenario you have described.
Best,
Hi Divya,
Thank you for the clarification.
Since the above scenario is inapplicable, I would like to suggest another solution.
Referring to the attached image, Scenario A is the inapplicable solution you described. As for the other solution (scenario B), I was thinking of splitting “annot_A_and_B” into 2 separate tiers as indicated by the image so 1 tier contains “annot_A”, 1 tier contains “annot_B”. Then, running kappa analysis by manually selecting the 2 tiers into 1 so there is 1 kappa value computed per 2 tiers.
May I ask if this is possible?
If this message is unclear, please let me know if you need more clarification.
Thank you,
Valerie Fan
Hi Valerie,
yes with the described scenario (scenario B) you will be able to choose these two tiers for calculation and hence can compute the kappa value.
Hi Divya,
Thank you again for the reply!
I wanted to clarify my question further as I think my question is still unclear according to your response.
According to the image below, I was actually referring to running 1 kappa analysis on 2 sets of 2 tiers.
Each set will include “annot_A” and “annot_B”, separating “annot_A_and_B” from 1 into 2 tiers, so essentially comparing agreement between 2 sets of 2 combined tiers.
Please let me know if this is possible, and again if you need more clarification.
Thank you,
Valerie Fan
Hi Valerie,
With the above image it is more clear now what is the calculation you want to perform.
When you perform the kappa calculation on the above scenario, you will get the overall (all tiers) tiers result as well as results per pair combination if you select “Also generate and export agreement per tier pair”.
That is with 2 sets of 2 combined tiers, There will be 6 tier combinations , along with overall result ( all tiers )
You can choose paring of tiers to compare “based on prefix or suffix” to select the tiers by annotators. Then once you select the first tier by annotator the corresponding tier will be automatically selected. It is possible to calculate the kappa value with this scenario. But it is best if you carry out the calculation and see if indeed that is the output that you desired. Let me know.
Best Regards
Divya
Hi Divya,
I apologize for the late response, thank you again for your assistance.
I think I understand the navigation of tier selection within the Cohen’s Kappa analysis window. However I am a little confused on how you achieved the subtiers for “A_code” and “B_code” as specified in your screenshot below.
Will the corresponding tier only be selected when subtiers are created?
I apologize for the inconvenience, but may I also request a step-by-step procedure to achieve the complete analysis, if possible?
Thank you for your time, and I look forward to your reply.
Kind regards,
Valerie Fan
Hi Valerie,
No worries at all.
As I understood from your requirement, you want to have the two annotations having same time intervals hence I created the sub-tiers having a symbolic association with parent tiers “A_code” and “B_code” .
I created the tier types subA_code and subB_code having stereotype ‘symbolic association’.
Then I created tier subA_code with tier type subA_code and choosing A_code tier as a parent. Same steps for tier subB_code.
The corresponding tiers get selected based on the different suffix/prefix basis. After creation of the tiers and their annotations, the calculation steps and screenshots are as attached in previous response.
Best Reagrds
Divya
Hi Divya,
Thank you again for the response.
For clarification, how exactly did you create subtiers having a symbolic association with parent tiers?
I was trying to add a subtier and set its Parent Tier to another tier but I don’t think the output was what was referred in your response.
May I ask for specific procedures (menu selection steps?) I need to do to conduct this?
Thank you again for your assistance,
Valerie Fan
Hi Valerie,
If you could let me know where exactly you encounter the issue, during creation of sub tiers ? when you mention output , do you mean annotations ?
Best,
Hi Divya,
It’s been a while since I last contacted you, I hope you’re doing well!
Regarding where we left off and to provide some context, last time I asked whether I am able to conduct kappa analysis on 2 sets of symbolically associated tiers in which you showed me the method in a Zoom meeting we had.
However, I ran into another problem with this analysis. In the .txt file that was generated, I noticed that the number of pairs of tiers in the comparison is 6, but not 2. Please correct me if I’m wrong, but I believe this is due to the comparison of all possible pairwise relationships across all 4 individual tiers.
Is there any way that I can specifically choose what tiers I want to compare in the kappa analysis? For example, in this screenshot below:
Can I manually select “JP_annot” and “ML_annot” as 1 comparison, and then “sub_JP_annot” and “sub_ML_annot” as 1 comparison, resulting in 2 comparisons done within 1 kappa analysis and generating 1 .txt file with kappa values reflecting both comparisons at the same time?
Thank you for your assistance and I look forward to your reply.
Kind regards,
Valerie Fan
Hi Valerie,
I am doing well, Thank you. Hope you are doing well too.
You are right. With 2 sets of symbolically associated tiers, kappa is calculated with all the tier combinations hence there is 6 tier combinations.
I am afraid there is no way to manually select only the two combinations.
