When we have two nominal variables with the same values (usually two raters’ observations or scores using the same codes), you can compute Cohen’s kappa to check the reliability or agreement between the measures. Kappa has a few assumptions about the underlying nature of the data: 1) Participants are independent of each other, 2) the raters, reporters, or observers providing the data do so independently of one another, and 3) the rating categories are mutually exclusive and exhaustive. Data typically are nominal, but they can be ordinal. If they are normally distributed data, other measures are preferable. Imagine that the ethnicity variables were based on school records. Then, a new variable was obtained by asking students to self-report their ethnicity. The question is, how reliable is the interobserver classification of ethnicity?
What is the reliability coefficient for the ethnicity codes (based on school records) and ethnicity reported by the student?
To compute the kappa:
- Click on Analyze → Descriptive Statistics→ Crosstabs...
- Click on Reset to clear the previous entries.
- Move ethnicity (ethnic) to the Rows box and ethnicity reported by students (ethnic 2) to the Columns
- Click on Kappa in the Statistics dialog box.
- Click on Continue to go back to the Crosstabs dialog window.
- Then click on Cells and request the Observed cell counts and Total under percentages.
- Click on Continue and then OK. Compare your syntax and output to Output 7.5.
Output 7.5: Cohen’s Kappa With Nominal Data
CROSSTABS
/TABLES=ethnic BY ethnic2
/FORMAT= AVALUE TABLES
/STATISTICS=KAPPA
/CELLS= COUNT TOTAL
/COUNT ROUND CELL.
Crosstabs
Interpretation of Output 7.5
The Case Processing Summary table shows that 71 students have data on both variables; however, 6 students disagree with the school records, as indicated by boxes off the diagonal. The Crosstabulation table of ethnicity and ethnicity reported by student also shows the cases where the school records and the student self-reports are in agreement; they are on the diagonal and circled. There are 65 (40 + 11 + 8 + 6) students with such agreement or consistency.
The Symmetric Measures table shows that Kappa is .86. This indicates good reliability because such measures should be high ( > .70) and positive. Statistical significance is not relevant for reliability measures.
Example of How to Write About Problem 7.5
Results
Kappa was used to investigate the reliability coefficient for the ethnicity codes (based on school records) and ethnicity reported by the student (kappa = .86). This indicates that there is high reliability between the students’ reports and the school records.
Source: Morgan George A, Leech Nancy L., Gloeckner Gene W., Barrett Karen C.
(2012), IBM SPSS for Introductory Statistics: Use and Interpretation, Routledge; 5th edition; download Datasets and Materials.
Some genuinely great information, Glad I discovered this.