Toward Designing and Developing Likert Items to Assess Mathematical Problem Solving

  • James A. Mendoza ÁlvarezEmail author
  • Kathryn Rhoads
  • R. Cavender Campbell
Part of the ICME-13 Monographs book series (ICME13Mo)


Access to science, technology, engineering, and mathematics (STEM) careers relies heavily on student success in their foundational university mathematics courses. For future STEM majors, there is little debate over the procedural skills that are needed in algebra to continue in STEM, but what are the mathematical problem solving (MPS) skills needed to persist in calculus and beyond? There are currently few efficient tools that can be used to explore research questions such as this one. In response, our project goal is to develop Likert MPS items that can be machine scored and used as an efficient tool in exploring students’ MPS practices. The items link a student’s MPS to five key domains of MPS derived from the research literature; items are designed for entry-level university students and do not require content knowledge beyond secondary school algebra. In this chapter, we report on the Likert item development, including the revision and validation processes. As part of the development, we explored the following research questions: (a) How are student scores on MPS items linked to their MPS practices during interviews, and (b) How are student scores on MPS items linked to their course performance? Participants were over 1500 students enrolled in College Algebra or Calculus at a large, urban university in the United States. Results illustrated trends in student practices within each MPS domain; these trends are informative for future item development and research. Results also suggest that students may not be developing many MPS skills in their foundational university mathematics courses.


Mathematical problem solving Mathematics assessment College algebra Calculus placement 



Partial support for this work was provided by the National Science Foundation (NSF) Improving Undergraduate STEM Education (IUSE) program under Award No. 1544545. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. We also thank Dr. Lauren Coursey and Andrew Kercher for research assistance and reviewers for their feedback.


  1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.Google Scholar
  2. Bryk, A. S., & Treisman, U. (2010). Make math a gateway, not a gatekeeper. Chronicle of Higher Education, 56(32), B19–B20.Google Scholar
  3. Cai, J., Wang, N., Moyer, J. C., Wang, C., & Nie, B. (2011). Longitudinal investigation of the curriculum effect: An analysis of student learning outcomes from the LieCal project. International Journal of Educational Research, 50, 117–136.CrossRefGoogle Scholar
  4. Campbell, R. C. (2014). Reconciling secondary inservice teachers’ definitions of problem solving with those found in the research literature (Unpublished master’s thesis). The University of Texas at Arlington, Arlington, TX.Google Scholar
  5. Campbell, R. C. (2017, January). Mathematical problem solving practices: A comparison of a student in college algebra to a student in calculus. Presentation at the 2017 Joint Meetings of the American Mathematical Society and the Mathematical Association of America, Atlanta, GA.Google Scholar
  6. Carlson, M. P., & Bloom, I. (2005). The cyclic nature of problem solving: An emergent multidimensional problem solving framework. Educational Studies in Mathematics, 58, 45–75.CrossRefGoogle Scholar
  7. Carlson, M., Oehrtman, M., & Engelke, N. (2010). The precalculus concept assessment: A tool for assessing students’ reasoning abilities and understandings. Cognition and Instruction, 28(2), 113–145.CrossRefGoogle Scholar
  8. Chapman, O. (1999). Inservice teacher development in mathematical problem solving. Journal for Mathematics Teacher Education, 2, 121–142.CrossRefGoogle Scholar
  9. Charles A. Dana Center. (2007). Algebra II assessments: Performance tasks aligned to the Texas standards. Austin, TX: Author.Google Scholar
  10. Dawkins, P. C., & Epperson, J. A. M. (2014). The development and nature of problem-solving among first-semester calculus students. International Journal of Mathematical Education in Science and Technology, 45(6), 839–862.CrossRefGoogle Scholar
  11. Eisenberg, T., & Dreyfus, T. (1991). On the reluctance to visualize in mathematics. In W. Zimmermann & S. Cunningham (Eds.), Visualization in teaching and learning mathematics (pp. 25–37). Washington, DC: Mathematical Association of America.Google Scholar
  12. Engle, R. A. (2012). The productive disciplinary engagement framework: Origins, key concepts, and continuing developments. In D. Y. Dai (Ed.), Design research on learning and thinking in educational settings: Enhancing intellectual growth and functioning (pp. 161–200). London: Taylor & Francis.Google Scholar
  13. Engle, R. A., & Conant, F. R. (2002). Guiding principles for fostering productive disciplinary engagement: Explaining an emergent argument in a community of learners classroom. Cognition and Instruction, 20(4), 399–483.CrossRefGoogle Scholar
  14. Fike, D. S., & Fike, R. (2008). Mathematics: Gatekeeper for Hispanic success in health care professions. Hispanic Health Care International, 6(4), 185–191.CrossRefGoogle Scholar
  15. Fink, A. (2013). How to conduct surveys: A step-by-step guide (5th ed.). Los Angeles, CA: SAGE Publications.Google Scholar
  16. Garofalo, J., & Lester, F. K. (1985). Metacognition, cognitive monitoring, and mathematical performance. Journal for Research in Mathematics Education, 16(3), 163–176.CrossRefGoogle Scholar
  17. Gonzales, G. M. (2017). Investigating undergraduates’ mathematical problem solving: Alignment between interview practices and likert assessments (Unpublished master’s thesis). The University of Texas at Arlington, Arlington, TX.Google Scholar
  18. Green, P. (2016). An analysis of a University’s College Algebra curriculum: Searching for evidence of mathematical problem solving (Unpublished master’s thesis). The University of Texas at Arlington, Arlington, TX.Google Scholar
  19. Jarrett, E. (2000). Evaluating the persistence and performance of ‘successful’ precalculus students in subsequent mathematics courses (Master’s thesis). Lubbock, TX: Texas Tech University.Google Scholar
  20. Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65–94.CrossRefGoogle Scholar
  21. Kieran, C. (2007). Learning and teaching algebra at the middle school through college levels: Building meaning for symbols and their manipulation. In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (Vol. 1–2, pp. 707–762). Charlotte, NC: Information Age Publishing.Google Scholar
  22. Kubiszyn, T., & Borich, G. (2003). Educational testing and measurement: Classroom application and practice (7th ed.). New York, NY: Wiley.Google Scholar
  23. Kwek, M. L. (2015). Using problem posing as a formative assessment tool. In F. M. Singer, N. F. Ellerton, & J. Cai (Eds.), Mathematical problem posing from research to effective practice (pp. 273–292). New York, NY: Springer.CrossRefGoogle Scholar
  24. Lappan, G., & Phillips, E. (1998). Teaching and learning in the connected mathematics project. In L. Leutzinger (Ed.), Mathematics in the middle (pp. 83–92). Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  25. Lester, F. (2013). Thoughts about research on mathematical problem-solving instruction. The Mathematics Enthusiast, 10(1&2), 245–278.Google Scholar
  26. Lester, F., & Cai, J. (2014). Can mathematical problem solving be taught? Preliminary answers from 30 years of research. In P. Felmer, E. Pehkonen, & J. Kilpatrick (Eds.), Posing and solving mathematical problems advances and new perspectives (pp. 117–135). New York, NY: Springer.Google Scholar
  27. Maciejewski, W., & Merchant, S. (2016). Mathematical tasks, study approaches, and course grades in undergraduate mathematics: A year-by-year analysis. International Journal of Mathematical Education in Science and Technology, 47(3), 373–387.CrossRefGoogle Scholar
  28. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: SAGE Publications.Google Scholar
  29. National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics [Online version]. Retrieved from
  30. National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010). Common core state standards for mathematics. Washington, DC: Author.Google Scholar
  31. Oregon Department of Education, Office of Assessment and Evaluation. (2000). Mathematical problem solving scoring guide. Salem, OR: Author.Google Scholar
  32. Pearson Education Inc. (2010). Partnership for assessment of readiness for college and careers mathematics practice tests. Retrieved from
  33. Peters, H. (2017). Mathematical problem solving in a College Algebra Course curriculum (Unpublished master’s thesis). The University of Texas at Arlington, Arlington, TX.Google Scholar
  34. Phan, D. V. (2017). A snapshot of the alignment of University Students’ mathematical problem solving practices to a Likert-scale assessment of mathematical problem solving (Master’s thesis). The University of Texas at Arlington, Arlington, TX.Google Scholar
  35. Pineda, E. Y. (2016). Mathematical problem solving: Exploring undergraduate students’ written work and self-rated use of problem-solving domains (Unpublished master’s thesis). The University of Texas at Arlington, Arlington, TX.Google Scholar
  36. Polya, G. (1957). How to solve it. Princeton, NJ: Princeton University Press.Google Scholar
  37. Presmeg, N. C. (2006). Research on visualization in learning and teaching in mathematics: Emergence from psychology. In A. Gutierrez & P. Boero (Eds.), Handbook of research on the psychology of mathematics education: Past, present and future (pp. 205–235). Rotterdam, The Netherlands: Sense Publishers.Google Scholar
  38. Pugalee, D. (2004). A comparison of verbal and written descriptions of students’ problem solving processes. Educational Studies in Mathematics, 55, 27–47.CrossRefGoogle Scholar
  39. Rhoads, K., Epperson, J. A. M., & Campbell, R. C. (2017a). Undergraduates representing and connecting in mathematical problem solving. In T. A. Olson & L. Venenciano (Eds.), Proceedings of the 44th Annual Meeting of the Research Council on Mathematics Learning (pp. 129–136). Fort Worth, TX: RCML.Google Scholar
  40. Rhoads, K., Epperson, J. A. M., & Campbell, R. C. (2017b, January). The role of justifying in entry-level undergraduates’ mathematical problem solving. Presentation at the 2017 Joint Meetings of the American Mathematical Society and the Mathematical Association of America, Atlanta, GA.Google Scholar
  41. Santos-Trigo, M. (1998). Instructional qualities of a successful mathematical problem-solving class. International Journal for Math, Science, and Technology, 29(5), 631–646.CrossRefGoogle Scholar
  42. Santos-Trigo, M. (2014). Problem solving in mathematics education. In S. Lerman (Ed.), Encyclopedia of mathematics education (pp. 496–501). The Netherlands: Springer.Google Scholar
  43. Schoenfeld, A. H. (1985). Mathematical problem solving. Orlando, FL: Academic Press.Google Scholar
  44. Schoenfeld, A. H. (1988). Problem solving in context(s). In R. Charles & E. Silver (Eds.), The teaching and assessing of mathematical problem solving (pp. 82–92). Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  45. Schoenfeld, A. H. (1992). Learning to think mathematically: Problem solving, metacognition, and sense-making in mathematics. In D. Grouws (Ed.), Handbook for research on mathematics teaching and learning (pp. 334–370). New York: MacMillan.Google Scholar
  46. Schoenfeld, A. H. (2013). Reflections on problem solving theory and practice. The Mathematics Enthusiast, 10(1), 9–34.Google Scholar
  47. Schoenfeld, A. H. (2014). What makes for powerful classrooms, and how can we support teachers in creating them? A story of research and practice. Productively Intertwined. Educational Researcher, 43(8), 404–412.CrossRefGoogle Scholar
  48. Selden, A., Selden, J., Hauk, S., & Mason, A. (2000). Why can’t calculus students access their knowledge to solve non-routine problems. CBMS Issues in Mathematics Education, 8, 128–153.CrossRefGoogle Scholar
  49. Silver, E. A. (1985). Teaching and learning mathematical problem solving: Multiple research perspectives. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  50. Silver, E., & Cai, J. (1996). An analysis of arithmetic problem posing by middle school students. Journal for Research in Mathematics Education, 27, 521–539.CrossRefGoogle Scholar
  51. Smarter Balanced Assessment Consortium. (2014). Practice and training tests. Retrieved from
  52. Szetela, W., & Nicol, C. (2002). Evaluating problem solving in mathematics. Educational Leadership, 10, 42–45.Google Scholar
  53. Texas Education Agency. (2012). Texas essential knowledge and skills for mathematics. Retrieved from
  54. Turner, J. M. (2017). A tale of two techniques: A case study of mathematical problem-solving practices in college algebra students (Unpublished master’s thesis). The University of Texas at Arlington: Arlington, TX.Google Scholar
  55. Van Harpen, X., & Presmeg, N. (2015). An investigation of high school students’ mathematical problem posing in the United States and China. In F. M. Singer, N. F. Ellerton, & J. Cai (Eds.), Mathematical problem posing from research to effective practice (pp. 293–308). New York, NY: Springer.CrossRefGoogle Scholar
  56. Wilson, J. W., Fernandez, M. L., & Hadaway, N. (1993). Mathematical problem solving. In P. S. Wilson (Ed.), Research ideas for the classroom: High school mathematics (pp. 57–78). New York, NY: MacMillan.Google Scholar
  57. Yee, S. P., & Bostic, J. D. (2014). Developing a contextualization of students’ mathematical problem solving. The Journal of Mathematical Behavior, 36, 1–19.CrossRefGoogle Scholar
  58. Yin, R. K. (2009). Case study research: Design and methods. Thousand Oaks, CA: Sage.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • James A. Mendoza Álvarez
    • 1
    Email author
  • Kathryn Rhoads
    • 1
  • R. Cavender Campbell
    • 2
  1. 1.ArlingtonUSA
  2. 2.ViennaUSA

Personalised recommendations