Diagnosing Preservice Teachers’ Understanding of Statistics and Probability: Developing a Test for Cognitive Assessment

  • Muhammet AricanEmail author
  • Okan Kuzu


This study investigates preservice middle school mathematics teachers’ understanding of statistics and probability and provides a cognitive diagnostic assessment of their strengths and weaknesses on these subjects. A statistical reasoning test that included 15 multiple-choice and 5 open-ended items was developed from the perspective of the log-linear cognitive diagnosis model, which is a general form of the cognitive diagnosis models. The statistical reasoning test was applied to 456 preservice teachers from 4 universities in 3 different regions of Turkey. The collected data were analyzed using the Mplus 6.12 software, and diagnostic feedback on the preservice teachers’ responses was provided based on the findings. The analysis suggested that although many preservice teachers were able to represent and interpret the given data, most experienced difficulty in drawing inferences about populations based on samples, selecting and using appropriate statistical methods, and understanding and applying the basic concepts of probability. In addition, preservice teachers had difficulty answering open-ended items. Implications for teaching are also discussed.


Cognitive assessment Diagnostic classification models Preservice teacher education Statistics and probability Test design 



Parts of this study were presented at the 2017 European Educational Research Conference held in Copenhagen/Denmark.

Authors’ Contributions

This study was conducted in equal collaboration between the two authors.


The authors disclose receipt of the following financial support for the research, authorship, and publication of this study: This study was supported by the Ahi Evran University Scientific Research Projects Coordination Unit. Project Number EGT.A3.16.014.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that there are no conflicts of interest.

Supplementary material

10763_2019_9985_MOESM1_ESM.docx (152 kb)
ESM 1 (DOCX 151 kb)


  1. Ball, D. L., Lubienski, S. T., & Mewborn, D. S. (2001). Research on teaching mathematics: The unsolved problem of teachers’ mathematical knowledge. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 433–456). Washington, DC: American Educational Research Association.Google Scholar
  2. Batanero, C., & Díaz, C. (2010). Training teachers to teach statistics: What can we learn from research? Statistique et Enseignement, 1(1), 5–20.Google Scholar
  3. Batanero, C., & Díaz, C. (2012). Training school teachers to teach probability: Reflections and challenges. Chilean Journal of Statistics, 3(1), 3–13.Google Scholar
  4. Batanero, C., Godino, J. D., & Roa, R. (2004). Training teachers to teach probability. Journal of Statistics Education, 1(1), 5–20. Retrieved February 3, 2018, from
  5. Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., ... Tsai, Y. (2010). Teachers’ mathematical knowledge, cognitive activation in the classroom, and student progress. American Educational Research Journal, 47(1), 133–180.Google Scholar
  6. Bradshaw, L. (2015). PARCC diagnostic assessments for mathematics comprehension: A diagnostic classification model approach. Paper presented at the Council of Chief State School Officers (CCSSO) 2015 National Conference on Student Assessment (NCSA) in San Diego, California.Google Scholar
  7. Bradshaw, L., Izsák, A., Templin, J., & Jacobson, E. (2014). Diagnosing teachers’ understandings of rational numbers: Building a multidimensional test within the diagnostic classification framework. Educational Measurement: Issues and Practice, 33(1), 2–14.CrossRefGoogle Scholar
  8. Brown, A., & Silver, E. (1989). Data organization and interpretation. In M. M. Lindquist (Ed.), Results from the fourth mathematics assessment of the national assessment of educational progress (pp. 28–34). Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  9. Burgess, T. A. (2007). Investigating the nature of teacher knowledge needed and used in teaching statistics (Unpublished doctoral dissertation). Massey University, Auckland, New Zealand.Google Scholar
  10. Choi, K. M., Lee, Y. S., & Park, Y. S. (2015). What CDM can tell about what students have learned: An analysis of TIMSS eighth grade mathematics. Eurasia Journal of Mathematics, Science & Technology Education, 11(6), 1563–1577.Google Scholar
  11. Common Core State Standards Initiative. (2010). The common core state standards for mathematics. Washington, DC: Author.Google Scholar
  12. Curcio, F. R. (1987). Comprehension of mathematical relationships expressed in graphs. Journal for Research in Mathematics Education, 18(5), 382–393.CrossRefGoogle Scholar
  13. de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362.CrossRefGoogle Scholar
  14. de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.CrossRefGoogle Scholar
  15. delMas, R., Garfield, J., & Ooms, A. (2005, July). Using assessment items to study students’ difficulty with reading and interpreting graphical representations of distributions. Paper presented at the Fourth Forum on Statistical Reasoning, Thinking, and Literacy (SRTL–4), Auckland, New Zealand.Google Scholar
  16. delMas, R., Garfield, J., Ooms, A., & Chance, B. (2007). Assessing students’ conceptual understanding after a first course in statistics. Statistics Education Research Journal, 6(2), 28–58.Google Scholar
  17. Dereli, A. (2009). The mistakes and misconceptions in probability of eighth grade students (Unpublished Master’s Thesis). Eskisehir Osmangazi University, Turkey.Google Scholar
  18. DiBello, L. V., Stout, W. F., & Roussos, L. A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood–based classification techniques. In P. Nichols, S. Chipman, & R. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361–390). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  19. Dogan, E., & Tatsuoka, K. (2008). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS–R. Educational Studies in Mathematics, 68(3), 263–272.CrossRefGoogle Scholar
  20. Fennema, E., & Franke, M. L. (1992). Teachers’ knowledge and its impact. In D. A. Grouws (Ed.), Handbook of research on mathematics teaching and learning (pp. 147–164). New York, NY: Macmillan.Google Scholar
  21. Franklin, C., Kader, G., Mewborn, D. S., Moreno, J., Peck, R., Perry, M., & Scheaffer, R. (2007). Guidelines for assessment and instruction in statistics education (GAISE) report: A pre–K–12 curriculum framework. Alexandria, VA: American Statistical Association. Retrieved February 8, 2018, from
  22. Franklin, C., & Mewborn, D. (2006). The statistical education of PreK–12 teachers: A shared responsibility. In G. Burrill (Ed.), NCTM 2006 Yearbook: Thinking and reasoning with data and chance (pp. 335–344). Reston, VA: NCTM.Google Scholar
  23. Garfield, J., & Ben-Zvi, D. (2007). How students learn statistics revisited: A current review of research on teaching and learning statistics. International Statistical Review, 75(3), 372–396.CrossRefGoogle Scholar
  24. Garfield, J., & Ben–Zvi, D. (2008). Developing students’ statistical reasoning: Connecting research and teaching practice. Berlin, Germany: Springer Science & Business Media.Google Scholar
  25. Garfield, J., & Ahlgren, A. (1988). Difficulties in learning basic concepts in probability and statistics: Implications for research. Journal for research in Mathematics Education, 19(1), 44–63.Google Scholar
  26. Groth, R. E. (2007). Toward a conceptualization of statistical knowledge for teaching. Journal for Research in Mathematics Education, 38(5), 427–437.Google Scholar
  27. Groth, R. E., & Bergner, J. A. (2006). Preservice elementary teachers’ conceptual and procedural knowledge of mean, median, and mode. Mathematical Thinking and Learning, 8(1), 37–63.CrossRefGoogle Scholar
  28. Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333.CrossRefGoogle Scholar
  29. Hartz, S. (2002). A Bayesian framework for the Unified Model for assessing cognitive abilities: Blending theory with practice (Unpublished doctoral dissertation). University of Illinois at Urbana–Champaign.Google Scholar
  30. Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnostics. Applied Psychological Measurement, 29(4), 262–277.CrossRefGoogle Scholar
  31. Henson, R., Roussos, L., Douglas, J., & He, X. (2008). Cognitive diagnostic attribute–level discrimination indices. Applied Psychological Measurement, 32(4), 275–288.CrossRefGoogle Scholar
  32. Henson, R., Templin, J., & Willse, J. (2009). Defining a family of cognitive diagnosis models using log–linear models with latent variables. Psychometrika, 74(2), 191–210.CrossRefGoogle Scholar
  33. Hill, H., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42(2), 371–406.CrossRefGoogle Scholar
  34. Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers’ mathematics knowledge for teaching. The Elementary School Journal, 105(1), 11–30.CrossRefGoogle Scholar
  35. Im, S., & Park, H. J. (2010). A comparison of US and Korean students’ mathematics skills using a cognitive diagnostic testing method: Linkage to instruction. Educational Research and Evaluation, 16(3), 287–301.CrossRefGoogle Scholar
  36. Jones, G. A. (2005). Exploring probability in school: Challenges for teaching and learning. New York, NY: Springer.CrossRefGoogle Scholar
  37. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272.CrossRefGoogle Scholar
  38. Leavy, A. M. (2010). The challenge of preparing preservice teachers to teach informal inferential reasoning. Statistics Education Research Journal, 9(1), 46–67.Google Scholar
  39. Lee, H. S., & Hollebrands, K. F. (2011). Characterising and developing teachers’ knowledge for teaching statistics with technology. In C. Batanero, G. Burrill, & C. Reading (Eds.), Teaching statistics in school mathematics-challenges for teaching and teacher education (pp. 359–369). Dordrecht, Netherlands: Springer.Google Scholar
  40. Lee, Y. S., Park, Y. S., & Taylan, D. (2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the US national sample using the TIMSS 2007. International Journal of Testing, 11(2), 144–177.CrossRefGoogle Scholar
  41. Li, K. Y., & Shen, S. M. (1992). Students’ weaknesses in statistical projects. Teaching Statistics, 14(1), 2–8.CrossRefGoogle Scholar
  42. Makar, K., & Rubin, A. (2009). A framework for thinking about informal statistical inference. Statistics Education Research Journal, 8(1), 82–105.Google Scholar
  43. Mathews, D., & Clark, J. (2003). Successful students’ conceptions of mean, standard deviation, and the central limit theorem. Unpublished paper. Retrieved July 4, 2018 from
  44. Muthen, L. K., & Muthen, B. O. (1998–2011). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthen & Muthen.Google Scholar
  45. O’Connell, A. A. (1999). Understanding the nature of errors in probability problem–solving. Educational Research and Evaluation, 5(1), 1–21.CrossRefGoogle Scholar
  46. Panackal, A. A., & Heft, C. S. (1978). Cloze technique and multiple choice technique: Reliability and validity. Educational and Psychological Measurement, 38(4), 917–932.CrossRefGoogle Scholar
  47. R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved January 3, 2018, from
  48. Ranjbaran, F., & Alavi, S. M. (2017). Developing a reading comprehension test for cognitive diagnostic assessment: A RUM analysis. Studies in Educational Evaluation, 55, 167–179.CrossRefGoogle Scholar
  49. Ravand, H., & Robitzsch, A. (2015). Cognitive diagnostic modeling using R. Practical Assessment, Research & Evaluation, 20(11), 1–12.Google Scholar
  50. Rupp, A., & Templin, J. (2008). Effects of Q–matrix misspecification on parameter estimates and misclassification rates in the DINA model. Educational and Psychological Measurement, 68(1), 78–98.CrossRefGoogle Scholar
  51. Rupp, A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.Google Scholar
  52. Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference chi–square test statistic. Psychometrika, 75(2), 243–248.CrossRefGoogle Scholar
  53. Sen, S., & Arican, M. (2015). A diagnostic comparison of Turkish and Korean students’ mathematics performances on the TIMSS 2011 assessment. Journal of Measurement and Evaluation in Education and Psychology, 6(2), 238–253.Google Scholar
  54. Shaughnessy, J. M. (2007). Research on statistics learning and reasoning. In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 957–1009). Reston, VA: The National Council of Teachers of Mathematics.Google Scholar
  55. Stohl, H. (2005). Probability in teacher education and development. In G. A. Jones (Ed.), Exploring probability in school: Challenges for teaching and learning (pp. 345–366). Boston, MA: Springer.CrossRefGoogle Scholar
  56. Tatsuoka, K. (1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational Statistics, 10(1), 55–73.CrossRefGoogle Scholar
  57. Templin, J. (2008). Test construction item discrimination. Lecture presented at the Diagnostic Modelling Seminar at the University of Georgia, Athens. Retrieved February 2, 2018, from
  58. Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251–275.CrossRefGoogle Scholar
  59. Templin, J., & Henson, R. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305.CrossRefGoogle Scholar
  60. Toker, T., & Green, K. (2012). An application of cognitive diagnostic assessment on TIMMS–2007 8th grade mathematics items. Paper presented at the annual meeting of the American Educational Research Association, Vancouver, British Columbia, Canada.Google Scholar
  61. von Davier, M. (2005). A general diagnostic model applied to language testing data. ETS Research Report. Princeton, NJ: Educational Testing Service.Google Scholar
  62. Watson, J. M. (2006). Statistical literacy at school: Growth and goals. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  63. Werner, C., & Schermelleh-Engel, K. (2010). Deciding between competing models: Chi–square difference tests. In Introduction to structural equation modeling with LISREL (pp. 1–3). Frankfurt, Germany: Goethe University.Google Scholar
  64. Zawojewski, J. S., & Heckman, D. S. (1997). What do students know about data analysis, statistics, and probability? In P. A. Kenny & E. A. Silver (Eds.), Results from the sixth mathematics assessment of the National Assessment of Educational Progress (pp. 195–223). Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  65. Zawojewski, J. S., & Shaughnessy, J. M. (2000). Mean and median: Are they really so easy? Mathematics Teaching in the Middle School, 5(7), 436–440.Google Scholar

Copyright information

© Ministry of Science and Technology, Taiwan 2019

Authors and Affiliations

  1. 1.Department of Mathematics and Science EducationKırşehir Ahi Evran UniversityKırşehirTurkey

Personalised recommendations