# Diagnosing Preservice Teachers’ Understanding of Statistics and Probability: Developing a Test for Cognitive Assessment

## Abstract

This study investigates preservice middle school mathematics teachers’ understanding of statistics and probability and provides a cognitive diagnostic assessment of their strengths and weaknesses on these subjects. A statistical reasoning test that included 15 multiple-choice and 5 open-ended items was developed from the perspective of the log-linear cognitive diagnosis model, which is a general form of the cognitive diagnosis models. The statistical reasoning test was applied to 456 preservice teachers from 4 universities in 3 different regions of Turkey. The collected data were analyzed using the Mplus 6.12 software, and diagnostic feedback on the preservice teachers’ responses was provided based on the findings. The analysis suggested that although many preservice teachers were able to represent and interpret the given data, most experienced difficulty in drawing inferences about populations based on samples, selecting and using appropriate statistical methods, and understanding and applying the basic concepts of probability. In addition, preservice teachers had difficulty answering open-ended items. Implications for teaching are also discussed.

## Keywords

Cognitive assessment Diagnostic classification models Preservice teacher education Statistics and probability Test design## Notes

### Acknowledgements

Parts of this study were presented at the 2017 European Educational Research Conference held in Copenhagen/Denmark.

### Authors’ Contributions

This study was conducted in equal collaboration between the two authors.

### Funding

The authors disclose receipt of the following financial support for the research, authorship, and publication of this study: This study was supported by the Ahi Evran University Scientific Research Projects Coordination Unit. Project Number *EGT.A3.16.014*.

### Compliance with Ethical Standards

### Conflict of Interest

The authors declare that there are no conflicts of interest.

## Supplementary material

## References

- Ball, D. L., Lubienski, S. T., & Mewborn, D. S. (2001). Research on teaching mathematics: The unsolved problem of teachers’ mathematical knowledge. In V. Richardson (Ed.),
*Handbook of research on teaching*(4th ed., pp. 433–456). Washington, DC: American Educational Research Association.Google Scholar - Batanero, C., & Díaz, C. (2010). Training teachers to teach statistics: What can we learn from research?
*Statistique et Enseignement, 1*(1), 5–20.Google Scholar - Batanero, C., & Díaz, C. (2012). Training school teachers to teach probability: Reflections and challenges.
*Chilean Journal of Statistics, 3*(1), 3–13.Google Scholar - Batanero, C., Godino, J. D., & Roa, R. (2004). Training teachers to teach probability.
*Journal of Statistics Education, 1*(1), 5–20. Retrieved February 3, 2018, from http://www.amstat.org/publications/jse/v12n1/batanero.html. - Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., ... Tsai, Y. (2010). Teachers’ mathematical knowledge, cognitive activation in the classroom, and student progress.
*American Educational Research Journal, 47*(1), 133–180.Google Scholar - Bradshaw, L. (2015).
*PARCC diagnostic assessments for mathematics comprehension: A diagnostic classification model approach*. Paper presented at the Council of Chief State School Officers (CCSSO) 2015 National Conference on Student Assessment (NCSA) in San Diego, California.Google Scholar - Bradshaw, L., Izsák, A., Templin, J., & Jacobson, E. (2014). Diagnosing teachers’ understandings of rational numbers: Building a multidimensional test within the diagnostic classification framework.
*Educational Measurement: Issues and Practice, 33*(1), 2–14.CrossRefGoogle Scholar - Brown, A., & Silver, E. (1989). Data organization and interpretation. In M. M. Lindquist (Ed.),
*Results from the fourth mathematics assessment of the national assessment of educational progress*(pp. 28–34). Reston, VA: National Council of Teachers of Mathematics.Google Scholar - Burgess, T. A. (2007).
*Investigating the nature of teacher knowledge needed and used in teaching statistics*(Unpublished doctoral dissertation). Massey University, Auckland, New Zealand.Google Scholar - Choi, K. M., Lee, Y. S., & Park, Y. S. (2015). What CDM can tell about what students have learned: An analysis of TIMSS eighth grade mathematics.
*Eurasia Journal of Mathematics, Science & Technology Education, 11*(6), 1563–1577.Google Scholar - Common Core State Standards Initiative. (2010).
*The common core state standards for mathematics*. Washington, DC: Author.Google Scholar - Curcio, F. R. (1987). Comprehension of mathematical relationships expressed in graphs.
*Journal for Research in Mathematics Education, 18*(5), 382–393.CrossRefGoogle Scholar - de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications.
*Journal of Educational Measurement, 45*(4), 343–362.CrossRefGoogle Scholar - de la Torre, J. (2011). The generalized DINA model framework.
*Psychometrika, 76*(2), 179–199.CrossRefGoogle Scholar - delMas, R., Garfield, J., & Ooms, A. (2005, July).
*Using assessment items to study students’ difficulty with reading and interpreting graphical representations of distributions.*Paper presented at the Fourth Forum on Statistical Reasoning, Thinking, and Literacy (SRTL–4), Auckland, New Zealand.Google Scholar - delMas, R., Garfield, J., Ooms, A., & Chance, B. (2007). Assessing students’ conceptual understanding after a first course in statistics.
*Statistics Education Research Journal, 6*(2), 28–58.Google Scholar - Dereli, A. (2009).
*The mistakes and misconceptions in probability of eighth grade students*(Unpublished Master’s Thesis). Eskisehir Osmangazi University, Turkey.Google Scholar - DiBello, L. V., Stout, W. F., & Roussos, L. A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood–based classification techniques. In P. Nichols, S. Chipman, & R. Brennan (Eds.),
*Cognitively diagnostic assessment*(pp. 361–390). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar - Dogan, E., & Tatsuoka, K. (2008). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS–R.
*Educational Studies in Mathematics, 68*(3), 263–272.CrossRefGoogle Scholar - Fennema, E., & Franke, M. L. (1992). Teachers’ knowledge and its impact. In D. A. Grouws (Ed.),
*Handbook of research on mathematics teaching and learning*(pp. 147–164). New York, NY: Macmillan.Google Scholar - Franklin, C., Kader, G., Mewborn, D. S., Moreno, J., Peck, R., Perry, M., & Scheaffer, R. (2007).
*Guidelines for assessment and instruction in statistics education (GAISE) report: A pre–K–12 curriculum framework*. Alexandria, VA: American Statistical Association. Retrieved February 8, 2018, from http://www.amstat.org/asa/education/Guidelines-for-Assessment-and-Instruction-in-Statistics-Education-Reports.aspx. - Franklin, C., & Mewborn, D. (2006). The statistical education of PreK–12 teachers: A shared responsibility. In G. Burrill (Ed.),
*NCTM 2006 Yearbook: Thinking and reasoning with data and chance*(pp. 335–344). Reston, VA: NCTM.Google Scholar - Garfield, J., & Ben-Zvi, D. (2007). How students learn statistics revisited: A current review of research on teaching and learning statistics.
*International Statistical Review, 75*(3), 372–396.CrossRefGoogle Scholar - Garfield, J., & Ben–Zvi, D. (2008).
*Developing students’ statistical reasoning: Connecting research and teaching practice*. Berlin, Germany: Springer Science & Business Media.Google Scholar - Garfield, J., & Ahlgren, A. (1988). Difficulties in learning basic concepts in probability and statistics: Implications for research.
*Journal for research in Mathematics Education, 19*(1), 44–63.Google Scholar - Groth, R. E. (2007). Toward a conceptualization of statistical knowledge for teaching.
*Journal for Research in Mathematics Education, 38*(5), 427–437.Google Scholar - Groth, R. E., & Bergner, J. A. (2006). Preservice elementary teachers’ conceptual and procedural knowledge of mean, median, and mode.
*Mathematical Thinking and Learning, 8*(1), 37–63.CrossRefGoogle Scholar - Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment.
*Applied Measurement in Education, 15*(3), 309–333.CrossRefGoogle Scholar - Hartz, S. (2002).
*A Bayesian framework for the Unified Model for assessing cognitive abilities: Blending theory with practice*(Unpublished doctoral dissertation). University of Illinois at Urbana–Champaign.Google Scholar - Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnostics.
*Applied Psychological Measurement, 29*(4), 262–277.CrossRefGoogle Scholar - Henson, R., Roussos, L., Douglas, J., & He, X. (2008). Cognitive diagnostic attribute–level discrimination indices.
*Applied Psychological Measurement, 32*(4), 275–288.CrossRefGoogle Scholar - Henson, R., Templin, J., & Willse, J. (2009). Defining a family of cognitive diagnosis models using log–linear models with latent variables.
*Psychometrika, 74*(2), 191–210.CrossRefGoogle Scholar - Hill, H., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement.
*American Educational Research Journal, 42*(2), 371–406.CrossRefGoogle Scholar - Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers’ mathematics knowledge for teaching.
*The Elementary School Journal, 105*(1), 11–30.CrossRefGoogle Scholar - Im, S., & Park, H. J. (2010). A comparison of US and Korean students’ mathematics skills using a cognitive diagnostic testing method: Linkage to instruction.
*Educational Research and Evaluation, 16*(3), 287–301.CrossRefGoogle Scholar - Jones, G. A. (2005).
*Exploring probability in school: Challenges for teaching and learning*. New York, NY: Springer.CrossRefGoogle Scholar - Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory.
*Applied Psychological Measurement, 25*(3), 258–272.CrossRefGoogle Scholar - Leavy, A. M. (2010). The challenge of preparing preservice teachers to teach informal inferential reasoning.
*Statistics Education Research Journal, 9*(1), 46–67.Google Scholar - Lee, H. S., & Hollebrands, K. F. (2011). Characterising and developing teachers’ knowledge for teaching statistics with technology. In C. Batanero, G. Burrill, & C. Reading (Eds.),
*Teaching statistics in school mathematics-challenges for teaching and teacher education*(pp. 359–369). Dordrecht, Netherlands: Springer.Google Scholar - Lee, Y. S., Park, Y. S., & Taylan, D. (2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the US national sample using the TIMSS 2007.
*International Journal of Testing, 11*(2), 144–177.CrossRefGoogle Scholar - Li, K. Y., & Shen, S. M. (1992). Students’ weaknesses in statistical projects.
*Teaching Statistics, 14*(1), 2–8.CrossRefGoogle Scholar - Makar, K., & Rubin, A. (2009). A framework for thinking about informal statistical inference.
*Statistics Education Research Journal, 8*(1), 82–105.Google Scholar - Mathews, D., & Clark, J. (2003).
*Successful students’ conceptions of mean, standard deviation, and the central limit theorem.*Unpublished paper. Retrieved July 4, 2018 from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.636.8870&rep=rep1&type=pdf. - Muthen, L. K., & Muthen, B. O. (1998–2011).
*Mplus user’s guide*(6th ed.). Los Angeles, CA: Muthen & Muthen.Google Scholar - O’Connell, A. A. (1999). Understanding the nature of errors in probability problem–solving.
*Educational Research and Evaluation, 5*(1), 1–21.CrossRefGoogle Scholar - Panackal, A. A., & Heft, C. S. (1978). Cloze technique and multiple choice technique: Reliability and validity.
*Educational and Psychological Measurement, 38*(4), 917–932.CrossRefGoogle Scholar - R Core Team. (2017).
*R: A language and environment for statistical computing*. Vienna, Austria: R Foundation for Statistical Computing. Retrieved January 3, 2018, from http://www.R-project.org/. - Ranjbaran, F., & Alavi, S. M. (2017). Developing a reading comprehension test for cognitive diagnostic assessment: A RUM analysis.
*Studies in Educational Evaluation, 55*, 167–179.CrossRefGoogle Scholar - Ravand, H., & Robitzsch, A. (2015). Cognitive diagnostic modeling using R.
*Practical Assessment, Research & Evaluation, 20*(11), 1–12.Google Scholar - Rupp, A., & Templin, J. (2008). Effects of Q–matrix misspecification on parameter estimates and misclassification rates in the DINA model.
*Educational and Psychological Measurement, 68*(1), 78–98.CrossRefGoogle Scholar - Rupp, A., Templin, J., & Henson, R. A. (2010).
*Diagnostic measurement: Theory, methods, and applications*. New York, NY: Guilford Press.Google Scholar - Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference chi–square test statistic.
*Psychometrika, 75*(2), 243–248.CrossRefGoogle Scholar - Sen, S., & Arican, M. (2015). A diagnostic comparison of Turkish and Korean students’ mathematics performances on the TIMSS 2011 assessment.
*Journal of Measurement and Evaluation in Education and Psychology, 6*(2), 238–253.Google Scholar - Shaughnessy, J. M. (2007). Research on statistics learning and reasoning. In F. K. Lester (Ed.),
*Second handbook of research on mathematics teaching and learning*(pp. 957–1009). Reston, VA: The National Council of Teachers of Mathematics.Google Scholar - Stohl, H. (2005). Probability in teacher education and development. In G. A. Jones (Ed.),
*Exploring probability in school: Challenges for teaching and learning*(pp. 345–366). Boston, MA: Springer.CrossRefGoogle Scholar - Tatsuoka, K. (1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach.
*Journal of Educational Statistics, 10*(1), 55–73.CrossRefGoogle Scholar - Templin, J. (2008).
*Test construction item discrimination*. Lecture presented at the Diagnostic Modelling Seminar at the University of Georgia, Athens. Retrieved February 2, 2018, from https://jonathantemplin.com/files/dcm/ersh9800f08/ersh9800f08_lecture11.pdf. - Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates.
*Journal of Classification, 30*(2), 251–275.CrossRefGoogle Scholar - Templin, J., & Henson, R. (2006). Measurement of psychological disorders using cognitive diagnosis models.
*Psychological Methods, 11*(3), 287–305.CrossRefGoogle Scholar - Toker, T., & Green, K. (2012).
*An application of cognitive diagnostic assessment on TIMMS–2007 8th grade mathematics items*. Paper presented at the annual meeting of the American Educational Research Association, Vancouver, British Columbia, Canada.Google Scholar - von Davier, M. (2005).
*A general diagnostic model applied to language testing data. ETS Research Report*. Princeton, NJ: Educational Testing Service.Google Scholar - Watson, J. M. (2006).
*Statistical literacy at school: Growth and goals*. Mahwah, NJ: Lawrence Erlbaum.Google Scholar - Werner, C., & Schermelleh-Engel, K. (2010). Deciding between competing models: Chi–square difference tests. In
*Introduction to structural equation modeling with LISREL*(pp. 1–3). Frankfurt, Germany: Goethe University.Google Scholar - Zawojewski, J. S., & Heckman, D. S. (1997). What do students know about data analysis, statistics, and probability? In P. A. Kenny & E. A. Silver (Eds.),
*Results from the sixth mathematics assessment of the National Assessment of Educational Progress*(pp. 195–223). Reston, VA: National Council of Teachers of Mathematics.Google Scholar - Zawojewski, J. S., & Shaughnessy, J. M. (2000). Mean and median: Are they really so easy?
*Mathematics Teaching in the Middle School, 5*(7), 436–440.Google Scholar