Using Item Response Theory as a Tool in Educational Measurement

Abstract

Item response theory (IRT) and classical test theory (CTT) are invaluable tools for the construction of assessment instruments and the measurement of student proficiencies in educational settings. However, the advantages of IRT over CTT are not always clear. This chapter uses an example item analysis to contrast IRT and CTT. It is hoped that the readers can gain a deeper understanding of IRT through comparisons of similarities and differences between IRT and CTT statistics. In particular, this chapter discusses item properties such as the difficulty and discrimination power of items, as well as person ability measures contrasting the weighted likelihood estimates and plausible values in non-technical ways. The main advantage of IRT over CTT is outlined through a discussion on the construction of a developmental scale on which individual students are located. Further, some limitations of both IRT and CTT are brought to light to guide the valid use of IRT and CTT results. Lastly, the IRT software program, ConQuest (Wu et al. ACERConQuest version 2: Generalised item response modelling software. Australian Council for Educational Research, Camberwell, 2007), is used to run the item analysis to illustrate some of the program’s functionalities.

Appendices

Appendices

1.1 Appendix A.1 Mathematics Test

1.2 Appendix A.2 Recoding of the Item Responses

Question	New code  =  student response
1. Place value	1 = 6 ones
	2 = 6 tens
	3 = 6 hundreds
	4 = 6 thousands
	9 = missing response
2. Time	1 = 25 (minutes)
	2 = 30 (minutes)
	3 = 35 (minutes)
	4 = 40 (minutes)
	0 = all other responses
	9 = missing response
3. Stamp	1 = 1 cm
	2 = 2 cm
	3 = 3 cm
	4 = 4 cm
	9 = missing response
4. Map	1 = 12 (km)
	0 = all other responses
	9 = missing response
5. Multiplication	1 = 1720
	0 = all other responses
	9 = missing response
6. Floor plan	1 = 50 (m)
	2 = 54 (m)
	3 = 56 (m)
	0 = all other responses
	9 = missing response
7. Sports graph	1 = netball
	2 = football
	3 = cricket
	4 = athletics
8. Transport graph	1 = 4 (students)
	2 = 5 (students)
	0 = all other responses
	9 = missing response
9. Lollies	1,2,3,4 according to the order of the 4 response options.
	9 = missing response
10. Spinner	1 = 1
	2 = 2 3 = 3
	4 = 4
	5 = 5
	9 = missing response
11. Shape fraction	1 = 1/2
	2 = 1/3
	3 = 1/4
	4 = 1/5
	9 = missing response
12. Number sentence	1 = 2
	2 = 3
	3 = 36
	0 = all other responses
	9 = missing response
13. Gingerbread man	1 = ($) 5
	2 = ($) 7.5
	3 = ($) 8.5
	4 = ($) 10
	0 = all other responses
	9 = missing response
14. Party pies	1 = 3 (boxes)
	2 = 4 (boxes)
	3 = 5 (boxes)
	0 = all other responses
	9 = missing response
15. Cubes	1 = 14
	2 = 15
	3 = 16
	4 = 17
	0 = all other responses
	9 = missing response