An Ideal Measurement

Abstract

When one undertakes the measurement of a latent trait, what are the desirable properties one would like to have for the measures?

Appendices

Hands-on Practices

Task 1

Use simulation to generate raw scores for students on an easy test and a hard test.

• Q1. Plot the two test scores on a graph

• Q2. Apply a logistic transformation to the raw scores as follows:

• Step 1: Compute percentage correct from the raw scores (raw score divided by possible maximum score). Let p denote percentage correct.

• Step 2: Compute transformed score by applying transformation, log(p/(1 − p)), where log is the natural logarithm. The ratio, p/(1 − p), is referred to as an “odds”. The results from the transformation of log(p/(1 − p)) are said to be in the “log of odds unit” (abbreviated as “logit”)

• Step 3: Plot the two transformed scores on a graph

Discuss the shapes of the two graphs in terms of measurement invariance. Which graph is closer to a straight line?

Note: This hands-on practice is to demonstrate IRT as viewed as a transformation of the raw scores. However, the actual mathematical modelling of IRT is at the individual item and individual person level, not at the test score level. In IRT software programs, often logistic transformations applied to the test scores or to item scores (percentage of students getting an item right), as shown in this hands-on practice, are used to provide initial values of person and item parameters.

Task 2

Investigate the relationship between raw scores and transformed logit scores. For example, if a test has a maximum score of 30, plot raw scores (between 0 and 30) against transformed scores. What are your observations in terms of the distances between raw scores and between logit scores? Is the relationship between raw scores and logit scores a linear one? If not, is there a range between which the relationship is approximately linear?

Discussion Points

• (1) For what purposes of measurement would raw scores be sufficient? For what purposes of measurement should IRT be applied?

• (2) Based on the presentation in Chaps. 5 and 6, what do you think are the differences between classical test theory and item response theory?

• (3) The illustration of the principles for estimating ability (as shown from Figs. 6.8, 6.9, 6.10 and 6.11) relies on a response pattern that shows more items correct for easy items, and fewer items correct for difficult items. In this way one can identify the region where there are about equal numbers of correct and incorrect items. What happens if there is no clear pattern of item responses, such as a random scattering of incorrect items over the low to high scale, so that there is no clear region where the student’s ability might be?

Exercises

• Q1. As percentages, raw scores have a minimum of 0 and a maximum of 100. What is the minimum and maximum of logits? (logit is defined as in the Hands-on Practice section).

• Q2. When percentage (p) is 50%, what is the value of the transformed logit?

• Q3. Consider two raw scores expressed in percentages, p1 and p2, where p2 is greater than p1. Let t1 and t2 denote the transformed logit scores of p1 and p2 respectively. Which of the following option(s) do you think are appropriate in relation to the relative magnitude of t1 and t2?

  t1 is greater than t2

  t2 is greater than t1

  One cannot say which is larger, as it depends on whether t1 and t2 are positive or negative

  One cannot say which is larger, as it depends on whether p1 and p2 are below or above average

• Q4. The following shows the response pattern of a student. Can you estimate the student’s ability?