PSY 350 Lecture Notes - Lecture 15: Psy, Inter-Rater Reliability
Document Summary
The correlation between two (or more) observers" scores for the same set of observations. If two raters are scoring things in the same way, their ratings will be highly and positively correlated. A low (or negative) correlation means that the raters are seeing things very differently. We find this by having two raters observe and score the same group of participants. Might just be a subset of the whole sample. But big enough to calculate a meaningful correlation. Higher than this if we are making high-stakes decisions. You could calculate the correlations among all possible pairs of raters, then report the mean r and the range of r"s. If they"re all above . 70, that"s "acceptable reliability" for most research purposes. You could calculate the intraclass correlation coefficient (icc) This tells us how much of the variance in your overall scores is due to the raters, rather than due to the participants.
