PSYC 2030U Lecture Notes - Lecture 2: Inter-Rater Reliability, Predictive Validity, Content Validity
Chapter 3
Video Notes: Reliability and Validity in Assessment and Diagnosis
- Most important measurements for assessment tools
- Reliability – measurement of consistency of score or responses
o Test-retest reliability – consistency of test results across time by testing
and retesting individuals to compare individuals
▪ Ex: Give Bob a test for OCD, chronic condition, if we retest him a
couple days later, will the test remain the same?
o Interrater reliability – across raters, if doctors use different manuals
across diagnosing Bob, there would be low interrater reliability if Dr A
diagnoses him for ADHD and Dr B diagnosis him for OCD
- Validity – test measures what it is supposed to
o Ex: IQ test actually measure intelligence?
o Content validity – content of the test matches its objectives
▪ IQ test – include questions that measure Bob’s processing speed,
not valid to include love interests for example
o Predictive validity – test can make accurate predictions
▪ If Bob’s IQ scores are high does that mean he will be successful in
school? If so, it has high predictive validity.
o Construct validity – extent to which constructs are measured accurately
▪ How accurate will it be to determine depression across thousands
of patients
- Which question do you think is most valid for a test that assesses one’s mood?
o Have you felt more depressed than usual?
- Which of the following examples implies good interrater reliability?
o Ten different doctors using the same assessment tool concluded that John
has Bipolar Disorder.
Chapter 3 Quiz (Practice).
1.) Patient was oriented to time and place, showed appropriate affect, and could do
simple calculations. Short- and long-term memories were intact – Mental status
exam
2.) Assessment is shown what it is intended to measure – validity
3.) Which is a part of the intellectual functioning component of a mental status
exam?
a. Noting a person’s vocabulary
4.) Compared to informal observation, what is an advantage of formal observation?
a. Formal observations are more reliable, due to the focus on behaviours that
are both observable and measurable.
5.) What is used to assess results on the MMPIT?
a. The correspondence between an individual’s pattern of answer and that of
people diagnosed with a specific disorder.
6.) In an attempt to predict which children would succeed in school, what form of
assessment did Binet develop?
a. An intelligence test, based on an intelligence quotient.
7.) Marguerite is tested for brain damage. Although her test results indicate that she
has no brain damage, in reality she has a brain tumour. Her tests indicate a false
negative, which of the following is the MOST likely outcome?
a. False negative – negative when it is actually positive
b. May not get the treatment she needs.
8.) How does the CAT scan neuroimaging technique work?
a. It uses X-rays to portray brain structures.
9.) What instrument measures brain wave activity by recording the electrical activity
of the brain?
a. EEG (electroencephalogram)
10.) What is the primary basis for prognosis?
a. Information about the course of the disorder in other individuals.
11.) What is the difference between the classical and dimensional approaches
to nosology?
a. The classical approach notes the presence of symptoms and the
dimensional approach notes the degree of severity of symptoms.
12.) What system of nosology is the DSM-5 based on?
a. The prototypical system of nosology.
13.) What is the purpose of psychophysiological assessment techniques?
a. To measure bodily changes, such as heart rate and anxiety, that are
associated with psychological processes.
14.) The DSM-5 is similar to the DSM-IV-R in which of the following ways?
a. The categories are very similar
15.) As a clinician, what should you do to begin an initial assessment?
a. Ask broad, open-ended questions.
16.) What is the purpose of standardizing diagnostic techniques?
a. To make techniques consistent and comparable.
17.) What is the primary purpose of structures and semi-structured interviews?
a. To collect necessary information in a standardized way to allow for
comparison with similar individuals.
18.) A young woman intently observes a handsome young man. He notices her
watching him and he promptly turns away and blushes. What is this an example
of?
a. Reactivity
19.) On which scale of the MMPI would you expect a remorseless, aggressive,
and irresponsible person to get a high score?
a. Psychopathic deviation scale
20.) Which of the following is a problem associated with a false positive result
on a psychological test
a. Got diagnosed but does not really have it
b. May call for unnecessary and expensive further testing
21.) Which technique would be appropriate if you wished to look for the
location of possible damage in a client’s brain?
a. PET scan
22.) Using the diagnosis “depressed” allows you, as a clinician to determine an
effective treatment and give an accurate prognosis. Which form of validity does
the diagnosis have?
a. Criterion validity
23.) What does “comorbidity” refer to?
a. More than one disorder diagnosed for the same individual.
24.) Why might a mixed-anxiety depression diagnosis be considered for DSM
inclusion?
a. Because symptoms are often seen in primary care settings.
25.) Emily has been evaluated by three different professionals, each of whom
offers a different diagnosis. She wonders whether those who are academically
qualified work with any unified standards. What do the diagnoses lack?
a. Reliability
Chapter 3 Textbook Notes
Introduction
- Clinical assessment – systemic evaluation of factors (psychological, biological,
and social) in an individual that is presenting with a possible psychological
disorder.
- Diagnosis – determines whether the individual meets the criteria for a
psychological disorder as per the DSM-5
Assessing Psychological Disorders
- Funnel – we take so much information from a client and narrow it down into its
most important constituents.
- Reliability, validity, and standardization are important concepts that help aid in
assessment techniques.
o Reliability – the degree to which a measurement is consistent
▪ Ex: You would be mad if you went to many different doctors to get
a different diagnosis, therefore their method of treatment would be
reliably low.
o Inter-rater reliability – Conducting research and having two or more
raters get similar scores.
o Test-retest reliability – if you take a test for the first time, you would
expect to get similar results the second time you take it.
o Validity – whether the assessment measures what it is supposed to.
▪ Concurrent/descriptive validity – comparing results of one
assessment measure with the results of others to determine the
validity of the first measure
• If we have a previous version of an IQ test that was the
practically the same as a new IQ test, it would have
concurrent validity.
▪ Predictive validity – how well the assessment tells you what will
happen in the future
• IQ tests predict how well a student will succeed in school
depending on the score.
Document Summary
Video notes: reliability and validity in assessment and diagnosis. Validity test measures what it is supposed to: ex: iq test actually measure intelligence, content validity content of the test matches its objectives. Iq test include questions that measure bob"s processing speed, not valid to include love interests for example: predictive validity test can make accurate predictions. If so, it has high predictive validity: construct validity extent to which constructs are measured accurately, how accurate will it be to determine depression across thousands. Which of the following examples implies good interrater reliability: ten different doctors using the same assessment tool concluded that john has bipolar disorder. Patient was oriented to time and place, showed appropriate affect, and could do simple calculations. Short- and long-term memories were intact mental status exam. Assessment is shown what it is intended to measure validity. Which is a part of the intellectual functioning component of a mental status exam: noting a person"s vocabulary.
