Week 3: Main Post



Week 3: Main Post

'Reliability' of any research is the degree to which it gives an accurate score across a range of measurement. It can thus be viewed as being 'repeatability' or 'consistency'. In summary (Thompson, 2006):

Inter-rater: Different people, same test.

Test-retest: Same people, different times.

Parallel-forms: Different people, same time, different test.

Internal consistency: Different questions, same construct.

Inter-Rater Reliability

When multiple people are giving assessments of some kind or are the subjects of some test, then similar people should lead to the same resulting scores. It can be used to calibrate people, for example those being used as observers in an experiment.

Inter-rater reliability thus evaluates reliability across different people.

Two major ways in which inter-rater reliability is used are (a) testing how similarly people categorize items, and (b) how similarly peoplescore items.

This is the best way of assessing reliability when you are using observation, as observer bias very easily creeps in. It does, however, assume you have multiple observers, which is not always the case (Anderson, 1994).

Inter-rater reliability is also known as inter-observer reliability or inter-coder reliability.

Examples

Two people may be asked to categorize pictures of animals as being dogs or cats. A perfectly reliable result would be that they both classify the same pictures in the same way.

Observers being used in assessing prisoner stress are asked to assess several 'dummy' people who are briefed to respond in a programmed and consistent way. The variation in results from a standard gives a measure of their reliability.

In a test scenario, an IQ test applied to several people with a true score of 120 should result in a score of 120 for everyone. In practice, there will be usually be some variation between people (Thompson, 2006).

Test-Retest Reliability

An assessment or test of a person should give the same results whenever you apply the test.

Test-retest reliability evaluates reliability across time.

Reliability can vary with the many ...