Advertisement

Measuring the Impact of Emerging Technologies in Education: A Pragmatic Approach

  • Mutlu Cukurova
  • Rosemary Luckin
Reference work entry
Part of the Springer International Handbooks of Education book series (SIHE)

Abstract

The evaluation of emerging technologies is important for their impacts to be effectively integrated into learning and teaching settings to bring the best benefit to learners and teachers. Educators, learners, parents, and policymakers alike, therefore, need reliable methodologies for evaluating the effectiveness of such emerging technologies. However, the impact evaluations of technology in education are challenging. This challenge is more significant for emerging technologies, as change is the essence of emerging educational technologies. Therefore, the value of traditional impact evaluations in education requires being reconsidered within this context. Here, we present a pragmatic approach to measuring the impact of emerging technologies in education which focuses on the suitability of the proposed evaluation methods and the types of evidence rather than on the hierarchy of these methods and evidence types. The approach has two main steps. First one is the creation of a clear theory of change to identify outcome measure(s) and assumptions that are behind the expected impact of the emerging technology intervention. Second is the identification of the type of evidence and methods to generate it that are the most appropriate for the current innovation stage of the emerging technology.

Keywords

Evidence-informed practice Emerging technologies Impact evaluation Pragmatism Evidence 

References

  1. Ainley, J., Enger, L., & Searle, D. (2008). Students in a digital age: Implications of ICT for teaching and learning. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 63–80). New York: Springer.CrossRefGoogle Scholar
  2. Bakker, M., Heuvel-Panhuizen, M., & Robitzsch, A. (2015). Longitudinal data on the effectiveness of mathematics mini-games in primary education. British Journal of Educational Technology, 46(5), 999–1004.CrossRefGoogle Scholar
  3. Beraza, I., Pina, A., & Demo, B. (2010, November). Soft & hard ideas to improve interaction with robots for kids & teachers. In Workshop proceedings of conference on simulation, modelling, and programming for autonomous robots, (pp. 549–557). Darmstadt (Germany).Google Scholar
  4. Biesta, G. (2007). Why “what works” won’t work: Evidence-based practice and the democratic deficit in educational research. Educational Theory, 57(1), 1–22.CrossRefGoogle Scholar
  5. Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141–178.CrossRefGoogle Scholar
  6. Buck, S., & McGee, J. (2015). Why government needs more randomized controlled trials: Refuting the myths. Houston: Laura and John Arnold Foundation. www.arnoldfoundation.org.Google Scholar
  7. Cobb, P., Confrey, J., DiSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.CrossRefGoogle Scholar
  8. Coburn, C. E., Russell, J. L., Kaufman, J. H., & Stein, M. K. (2012). Supporting sustainability: Teachers’ advice networks and ambitious instructional reform. American Journal of Education, 119(1), 137–182.CrossRefGoogle Scholar
  9. Cole, K. C., Van Tilburg, D., Burch-Vernon, A., & Riccio, D. C. (1996). The importance of context in the US preexposure effect in CTA: Novel versus latently inhibited contextual stimuli. Learning and Motivation, 27(4), 362–374.CrossRefGoogle Scholar
  10. Condie, R., & Munro, B. (2007). The impact of ICT in schools: Landscape review. UCL Institute of Education. London: United KingdomGoogle Scholar
  11. Cox, M. J. (2005). Educational conflict: The problems in institutionalizing new technologies in education. In G. Kouzelis, M. Pournari, M. Stoeppler, & V. Tselfes (Eds.), Knowledge in the new technology (pp. 139–165). Frankfurt: Peter Lang.Google Scholar
  12. Cox, M. J. (2008). Researching IT in education. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 965–981). New York: Springer.Google Scholar
  13. Cox, M. J., & Abbott, C. (Eds.). (2004). ICT and attainment – A review of the research literature. Full report. Coventry/London: Becta/DfES. http://www.becta.org.uk/research/index.cfm.Google Scholar
  14. Cox, M. J., & Marshall, G. (2007). Effects of ICT: Do we know what we should know? Education and Information Technologies, 12, 59–70.CrossRefGoogle Scholar
  15. Cox, M. J., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2003). ICT and pedagogy: A review of the research literature (ICT in schools research and evaluation series, Vol. 18) London: UCL Institute of Education.Google Scholar
  16. Cox, M., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2004). An investigation of the research evidence relating to ICT pedagogy. A report to DfES. Retrieved from http://dera.ioe.ac.uk/1601/1/becta_2003_attainmentpedagogy_queensprinter.pdf
  17. Cress, U., Moskaliuk, J., & Jeong, H. (Eds.). (2016). Mass collaboration and education (Vol. 16). Cham: Springer.Google Scholar
  18. Crompton, H., Burke, D., & Gregory, K. H. (2017). The use of mobile learning in PK-12 education: A systematic review. Computers & Education, 110, 51–63.CrossRefGoogle Scholar
  19. Cukurova, M., Luckin, R., & Baines, E. (2017). The significance of context for the emergence and implementation of research evidence: The case of collaborative problem-solving. Oxford Review of Education, 1–16. https://www.tandfonline.com/doi/abs/10.1080/03054985.2017.1389713.
  20. Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solving from students’ physical interactions. Computers & Education, 116, 93–109.CrossRefGoogle Scholar
  21. De Bruin, C. L. (2015). Conceptualizing effectiveness in disability research. International Journal of Research & Method in Education.  https://doi.org/10.1080/1743727X.2015.1033391.
  22. Department for International Development. (2017). Public Bulleting Board 7749: Education technology to improve learning for all. Retrieved from https://supplierportal.dfid.gov.uk/selfservice/pages/public/viewPublicNotice.cmd?bm90aWNlSWQ9Njc3MjA%3D.
  23. Dunleavy, M., & Dede, C. (2014). Augmented reality teaching and learning. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 735–745). New York: Springer.Google Scholar
  24. Fullan, M. (1983). Evaluating program implementation: What can be learned from Follow Through. Curriculum Inquiry, 13(2), 215–227.CrossRefGoogle Scholar
  25. Fullan, M. (2001). Leading in a culture of change. San Francisco: Jossey-Bass.Google Scholar
  26. Greany, T., & Maxwell, B. (2017). Evidence-informed innovation in schools: Aligning collaborative research and development with high quality professional learning for teachers. International Journal of Innovation in Education, 4(2–3), 147–170.CrossRefGoogle Scholar
  27. Gulson, K. N., & Symes, C. (2007). Spatial theories of education: Policy and geography matters. New York: Routledge.Google Scholar
  28. Hattie, J. (2008). Visible learning: A synthesis of over 800 meta analyses related to achievement. New York: Routledge.Google Scholar
  29. Hoeken, H. (2001). Anecdotal, statistical, and causal evidence: Their perceived and actual persuasiveness. Argumentation, 15(4), 425–437.Google Scholar
  30. Katterfeldt, E., Cukurova, M., Spikol, D., & Cuartielles, D. (2018). Physical computing with plug-and-play toolkits: Key recommendations for collaborative learning implementations. International Journal of Child-Computer Interaction, 1–23. https://www.sciencedirect.com/science/article/pii/S2212868917300351.
  31. Kelly, A. E., Baek, J. Y., Lesh, R. A., & Bannan-Ritland, B. (2008). Enabling innovations in education and systematizing their impact. In A. E. Kelly, R. A. Lesh, & J. Y. Baek (Eds.), Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching (pp. 3–18). Routledge, Abington, Oxon.Google Scholar
  32. Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review, 22(3), 215–243.CrossRefGoogle Scholar
  33. Marshall, G., & Cox, M. J. (2008). Research methods: Their design, applicability and reliability. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 983–1002). Boston: Springer.CrossRefGoogle Scholar
  34. Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Computers & Education, 70, 29–40.CrossRefGoogle Scholar
  35. Miller, D. J., & Robertson, D. P. (2011). Educational benefits of using game consoles in a primary classroom: A randomised controlled trial. British Journal of Educational Technology, 42(5), 850–864.CrossRefGoogle Scholar
  36. Nardi, B. A. (1996). Studying context: A comparison of activity theory, situated action models, and distributed cognition. In Context and consciousness: Activity theory and human-computer interaction (pp. 69–102). London: MIT Press.Google Scholar
  37. NESTA. (2016). Using research evidence for success: A practical guide. Retrieved on from https://www.nesta.org.uk/sites/default/files/using_research_evidence_for_success_-_a_practice_guide.pdf.
  38. Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography: Synthesizing qualitative studies. Newbury Park: Sage.CrossRefGoogle Scholar
  39. O’Leary, Z. (2004). The essential guide to doing research. London: Sage.Google Scholar
  40. Pampaka, M., Williams, J., & Homer, M. (2016). Is the educational ‘what works’ agenda working? Critical methodological developments. International Journal of Research and Method in Education, 39, 231–236.CrossRefGoogle Scholar
  41. Perry, B. (2015). Gamifying French language learning: A case study examining a quest-based, augmented reality mobile learning-tool. Procedia-Social and Behavioral Sciences, 174, 2308–2315.CrossRefGoogle Scholar
  42. Petty, G. (2009). Evidence-based teaching: A practical approach. Cheltenham: Nelson Thornes.Google Scholar
  43. Pilkington, R. (2008). Measuring the impact of IT on students’ learning. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education. Berlin/Heidelberg/New York: Springer.Google Scholar
  44. Reeves, T. C. (2008). Evaluation of the design and development of IT tools in education. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education. Berlin/Heidelberg/New York: Springer.Google Scholar
  45. Slavin, R. E. (2017). Evidence-based reform in education. Journal of Education for Students Placed at Risk (JESPAR), 22(3), 178–184.CrossRefGoogle Scholar
  46. Stern, E. (2015). Impact evaluation: A guide for commissioners and manager. London: Big Lottery Fund and the Department for International Development.Google Scholar
  47. Tondeur, J., van Braak, J., Guoyuan, S., Voogt, J., Fisser, P., & Ottenbreitt-Leftwich, A. S. (2012). Preparing student teachers to integrate ICT in classroom practice: A synthesis of qualitative evidence. Computers & Education, 59(1), 134–144.  https://doi.org/10.1016/j.compedu.2011.10.009.CrossRefGoogle Scholar
  48. Underwood, J. (2017). Exploring AI language assistants with primary EFL students. In CALL in a climate of change: Adapting to turbulent global conditions–EUROCALL 2017 (p. 317). Southampton: University of Southampton.Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.UCL Knowledge Lab, Institute of EducationUniversity College LondonLondonUK

Section editors and affiliations

  • Margaret Cox
    • 1
  • Joke Voogt
    • 2
  1. 1.King's College LondonLondonUK
  2. 2.Department of Child Development and EducationUniversity of Amsterdam/ Windesheim University of Applied SciencesAmsterdamThe Netherlands

Personalised recommendations