, Volume 50, Issue 4, pp 675–686 | Cite as

Computer-based assessment of mathematics into the twenty-first century: pressures and tensions

  • Kees Hoogland
  • Dave Tout
Original Article


In recent decades, technology has influenced various aspects of assessment in mathematics education: (1) supporting the assessment of higher-order thinking skills in mathematics, (2) representing authentic problems from the world around us to use and apply mathematical knowledge and skills, and (3) making the delivery of tests and the analysis of results through psychometric analysis more sophisticated. We argue that these developments are not pushing mathematics education in the same direction, however, which creates tensions. Mathematics education—so essential for educating young people to be creative and problem solving agents in the twenty-first century—is at risk of focusing too much on assessment of lower order goals, such as the reproduction of procedural, calculation based, knowledge and skills. While there is an availability of an increasing amount of sophisticated technology, the related advances in measurement, creation and delivery of automated assessments of mathematics are however being based on sequences of atomised test items. In this article several aspects of the use of technology in the assessment of mathematics education are exemplified and discussed, including in relation to the aforementioned tension. A way forward is suggested by the introduction of a framework for the categorisation of mathematical problem situations with an increasing sophistication of representing the problem situation using various aspects of technology. The framework could be used to reflect on and discuss mathematical assessment tasks, especially in relation to twenty-first century skills.


CBAM Twenty-first century skills Higher-order thinking Assessment framework Mathematical assessment tasks Mathematics assessment Technology 


  1. Ashton, H. S., Beevers, C. E., Korabinski, A. A., & Youngson, M. A. (2006). Incorporating partial credit in computer-aided assessment of Mathematics in secondary education. British Journal of Educational Technology, 37(1), 93–119. Scholar
  2. Australian Association of Mathematics Teachers, & Australian Industry-Group. (2014). Identifying and Supporting Quantitative Skills of 21st Century WorkersFinal Report. Retrieved from
  3. Baird, J. -A., Andrich, D., Hopfenbeck, T. N., & Stobart, G. (2017a). Assessment and learning: fields apart? Assessment in Education: Principles, Policy & Practice, 24(3), 317–350. Scholar
  4. Baird, J. -A., Andrich, D., Hopfenbeck, T. N., & Stobart, G. (2017b). Metrology of education. Assessment in Education: Principles, Policy & Practice, 24(3), 463–470. Scholar
  5. Baker Dearing Educational Trust. (2017). From school work to real work: how education fails students in the real world. London: Baker Dearing Educational Trust.Google Scholar
  6. Bardini, C. (2015). Computer-based assessment of mathematics in PISA 2012. In K. Stacey & R. Turner (Eds.), Assessing mathematical literacy: The PISA experience (pp. 173–188). New York: Springer.Google Scholar
  7. Bennett, R. E. (1998). Reinventing assessment: Speculations on the future of large-scale educational testing. Princeton, NJ: ETS.Google Scholar
  8. Bennett, R. E. (2010a). Cognitively based assessment of, for, and as learning: a preliminary theory of action for summative and formative assessment. Measurement: Interdisciplinary Research and Perspectives, 8(91), 70–91.Google Scholar
  9. Bennett, R. E. (2010b). Technology for large-scale assessment. In P. Peterson, E. Baker & B. McGaw (Eds.), International encyclopedia of education (vol. 8, (3rd ed.), pp. 48–55). Oxford: Elsevier.CrossRefGoogle Scholar
  10. Bennett, R. E. (2015). The changing nature of educational assessment. Review of Research in Education, 39(1), 370–407. Scholar
  11. Bennett, R. E., Persky, H., Weiss, A., & Jenkins, F. (2010). Measuring problem solving with technology: A demonstration study for NAEP. Journal of Technology, Learning, and Assessment, 8(8). Retrieved from
  12. Biesta, G. J. J. (2010). What is education for? Good education in an age of measurement: Ethics, politics, democracy. London: Taylor & Francis.Google Scholar
  13. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. Scholar
  14. Bokhove, C., & Drijvers, P. (2010). Digital tools for algebra education: criteria and evaluation. International Journal of Computers for Mathematical Learning, 15(1), 45–62. Scholar
  15. Bower, M., Howe, C., McCredie, N., Robinson, A., & Grover, D. (2014). Augmented reality in education—cases, places and potentials. Educational Media International, 51(1), 1–15. Scholar
  16. Burkhardt, H., & Schoenfeld, A. (2018). Assessment in the service of learning: Challenges and opportunities or Plus ça Change, Plus c’est la même Chose. ZDM. Scholar
  17. Coben, D., Hall, C., Hutton, M., Rowe, D., Weeks, K., & Wolley, N. (2010). Benchmark assessment of numeracy for nursing: Medication dosage calculation at point of registration. Edinburgh: NHS Education for Scotland.Google Scholar
  18. Corcoran, T., Mosher, F. A., & Rogat, A. (2009). Learning Progressions in Science: An Evidence-based Approach to Reform (Research Report #RR-63). Retrieved from Philadelphia, PA:
  19. Csapó, B., Molnár, G., & Tóth, K. R. (2009). Comparing paper-and-pencil and online assessment of reasoning skills. A pilot study for introducing electronic testing in large-scale assessment in Hungary. In F. Scheuermann & J. Björnsson (Eds.), The transition to computer-based assessment. New approaches to skills assessment and implications for large-scale testing. Luxemburg: Office for Official Publications of the European Communities.Google Scholar
  20. Csapó, B., Ainley, J., Bennett, R. E., Latour, T., & Law, N. (2012). Technological Issues for computer-based assessment. In P. Griffin, B. McGaw & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 143–230). Dordrecht: Springer.CrossRefGoogle Scholar
  21. Daro, P., Mosher, F. A., & Corcoran, T. (2011). Learning trajectories in mathematics: A foundation for standards, curriculum, assessment, and instruction. CPRE Research Report #RR-68. Philadelphia: Consortium for Policy Research in Education. Scholar
  22. Davis, R., Maher, C., & Noddings, N. (Eds.). (1990). Constructivist views on the teaching and learning of mathematics. Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  23. European Commission. (2017). Commission Staff Working Document, Accompanying the document Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions A renewed EU agenda for higher education SWD(2017)264 Retrieved from
  24. Forman, S. L., & Steen, L. A. (1999). Beyond eighth grade: Functional mathematics for life and work. Berkeley, CA: National Centre for Research in Vocational Education.Google Scholar
  25. Geiger, V., Goos, M., & Forgasz, H. (2015). A rich interpretation of numeracy for the 21st century: A survey of the state of the field. ZDM—Mathematics Education, 47(4), 531–548. Scholar
  26. Gravemeijer, K., Stephan, M., Julie, C., Lin, F.-L., & Ohtani, M. (2017). What mathematics education may prepare students for the society of the future? International Journal of Science and Mathematics Education. Scholar
  27. Greiff, S., & Kyllonen, P. (2016). Contemporary assessment challenges: The measurement of 21st century skills. Applied Measurement in Education, 29(4), 243–244. Scholar
  28. Griffin, P., & Care, E. (2015). Assessment and teaching of 21st Century Skills—methods and approach. New York: Springer.Google Scholar
  29. Heritage, M. (2008). Learning Progression: Supporting instruction and formative assessment. Retrieved from Los Angeles, CA:
  30. Hoogland, K. (2016). Images of numeracy: Investigating effects of visual representations of problem situations in contextual mathematical problem solving. (PhD-thesis). Eindhoven: Technical University Eindhoven.Google Scholar
  31. Hoogland, K., Pepin, B., Bakker, A., de Koning, J., & Gravemeijer, K. (2016). Representing contextual mathematical problems in descriptive or depictive form: Design of an instrument and validation of its uses. Studies in Educational Evaluation, 50, 22–32. Scholar
  32. Hoogland, K., Pepin, B., de Koning, J., Bakker, A., & Gravemeijer, K. (2018). Word problems versus image-rich problems: an analysis of effects of task characteristics on students’ performance on contextual mathematics problems. Research in Mathematics Education, 20(1), 37–52. Scholar
  33. Hoyles, C., Noss, R., Kent, P., & Bakker, A. (2010). Improving Mathematics at Work The Need for Techno-Mathematical Literacies. London and New York: Routledge.Google Scholar
  34. Hoyles, C., Wolf, A., Molyneux-Hodgson, S., & Kent, P. (2002). Mathematical skills in the workplace. Retrieved from London, UK:
  35. Kent, P., Bakker, A., Hoyles, C., & Noss, R. (2011). Measurement in the workplace: the case of process improvement in manufacturing industry. ZDM—The International Journal on Mathematics Education, 43(5), 747. Scholar
  36. Leighton, J. P., & Gierl, M. J. (2007). Cognitive diagnostic assessment for education: Theory and applications. New York, NY: Cambridge University Press.CrossRefGoogle Scholar
  37. Livne, N. L., Livne, O. E., & Wight, C. A. (2007). Can automated scoring surpass hand grading of students’ constructed responses and error patterns in mathematics? MERLOT Journal of Online Learning and Teaching, 3(3), 295–306.Google Scholar
  38. Madison, B. L., & Steen, L. A. (2003). Quantitative Literacy: Why Numeracy Matters for Schools and Colleges. Princeton, NJ: National Council on Education and the Disciplines.Google Scholar
  39. Masters, G. N. (2013). Reforming Educational Assessment: Imperatives, principles and challenges—Australian education review; no. 57. Camberwell: ACER.Google Scholar
  40. Ministry of Education Singapore—Curriculum Planning and Development Division. (2012). Mathematics Syllabus: Primary One to Five. Retrieved from
  41. Mullis, I. V. S., & Martin, M. O. (Eds.). (2013). Timss 2015 Assessment Frameworks. Chestnut Hill, MA: TIMSS & PIRLS International Study Center.Google Scholar
  42. National Council of Teachers of Mathematics (NCTM). (2000). Principles and standards for school mathematics Retrieved from Reston, VA:
  43. National Council of Teachers of Mathematics (NCTM). (2017). Catalyzing Change in High School Mathematics. Retrieved from Reston, VA:
  44. OECD. (2013). PISA 2012 results: Excellence through equity. Giving every student the chance to succeed (volume II). Paris: OECD Publishing.CrossRefGoogle Scholar
  45. OECD. (2014). PISA 2012 results: Creative problem solving. Students’ skills in tackling real-life problems (volume V). Paris: OECD Publishing.CrossRefGoogle Scholar
  46. OECD. (2016). Global competency for an inclusive world. Paris: OECD Publishing.Google Scholar
  47. OECD. (2017). PISA 2015 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. Retrieved from Paris, France:
  48. Palm, T. (2006). Word problems as simulations of real-world situations: A proposed framework. For the Learning of Mathematics, 26(1), 42–47. Scholar
  49. Palm, T. (2009). Theory of Authentic Task Situations. In L. Verschaffel, B. Greer, W. V. Dooren & S. Mukhopadhyay (Eds.), Words and worlds—modelling verbal descriptions of situations (pp. 3–20). Rotterdam: Sense.Google Scholar
  50. Parshall, C. G., Spray, J. A., Kalohn, J. C., & Davey, T. (2002). Practical considerations in computer-based testing. New York, NY: Springer.CrossRefGoogle Scholar
  51. Partnership for 21st Century Skills. (2016). Framework for 21st Century Learning. Retrieved from Washington, DC:
  52. Pead, D. A. (2010). On Computer-based Assessment of Mathematics (Ph.D.-thesis). Nottingham: University of Nottingham.Google Scholar
  53. PIAAC Numeracy Expert Group. (2009). PIAAC Numeracy: A conceptual framework. Retrieved from Paris, France:
  54. Poggio, J., Glasnapp, D. R., Yang, X., & Poggio, A. J. (2005). A Comparative evaluation of score results from computerized and paper-and-pencil mathematics testing in a large scale state assessment program. The Journal of Technology, Learning, and Assessment, 3(6).
  55. PwC. (2015). A smart move: Future-proofing Australias workforce by growing skills in science, technology, engineering and maths. Retrieved from Sydney Australia:
  56. Schoenfeld, A. H. (2017). On learning and assessment. Assessment in Education: Principles, Policy & Practice, 24(3), 369–378. Scholar
  57. Schwab, K. (2016). The Fourth Industrial Revolution: what it means, how to respond. Retrieved from
  58. Shute, V. J., Leighton, J. P., Jang, E. E., & Chu, M.-W. (2016). Advances in the Science of Assessment. Educational Assessment, 21(1), 34–59. Scholar
  59. Stacey, K., & Wiliam, D. (2013). Technology and Assessment in Mathematics. In M. A. Clements, A. J. Bishop, C. Keitel, J. Kilpatrick & F. K. S. Leung (Eds.), Third international handbook of mathematics education (pp. 721–752). New York: Springer.Google Scholar
  60. Steen, L. A. (2003). Data, shapes, symbols: Achieving balance in school mathematics. In B. L. Madison & L. A. Steen (Eds.), Quantitative literacy: Why numeracy matters for schools and colleges. Princeton, NJ: National Council on Education and the Disciplines.Google Scholar
  61. Straesser, R. (2015). Numeracy at work”: A discussion of terms and results from empirical studies. ZDM—Mathematics Education, 47(4), 665–674. Scholar
  62. Taubman, P. M. (2009). Teaching By Numbers: Deconstructing the Discourse of Standards and Accountability in Education. New York: Routledge.Google Scholar
  63. Tout, D., Coben, D., Geiger, V., Ginsburg, L., Hoogland, K., Maguire, T., et al. (2017). Review of the PIAAC numeracy assessment framework: Final report. Camberwell, Australia: Australian Council for Educational Research (ACER).Google Scholar
  64. Tout, D., & Spithill, J. (2015). The challenges and complexities of writing items to test mathematical literacy. In K. Stacey & R. Turner (Eds.), Assessing mathematical literacy: The PISA experience (pp. 145–171). Basel: Springer International Publishing.Google Scholar
  65. Verschaffel, L., Greer, B., Dooren, W. Van, & Mukhopadhyay, S. (Eds.). (2009). Words and worlds - Modelling verbal descriptions of situations. Rotterdam: Sense.Google Scholar
  66. Voogt, J., & Pareja Roblin, N. N. (2012). A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies. Journal of Curriculum Studies, 44(3), 299–321. Scholar
  67. Wake, G. (2015). Preparing for workplace numeracy: A modelling perspective. ZDM—Mathematics Education, 47(4), 675–689. Scholar
  68. Wiliam, D. (2005). Assessment for learning: Why no profile in US policy? In J. Gardner (Ed.), Assessment and learning (pp. 169–183). London: Sage.Google Scholar
  69. Wiliam, D. (2011). What is assessment for learning? Studies in Educational Evaluation, 37(1), 3–14. Scholar
  70. Wiliam, D., & Leahy, S. (2015). Embedding formative assessment: Practical techniques for K-12 classrooms. West Palm Beach, FL: Learning Sciences International.Google Scholar
  71. Zevenbergen, R. (2004). Technologizing numeracy: intergenerational differences in working mathematically in New Times. Educational Studies in Mathematics, 56(1), 97–117. Scholar

Copyright information

© FIZ Karlsruhe 2018

Authors and Affiliations

  1. 1.HU University of Applied Sciences UtrechtUtrechtThe Netherlands
  2. 2.Australian Council for Educational ResearchMelbourneAustralia

Personalised recommendations