Skip to main content
SpringerLink
Log in
Book cover

Second Handbook of Information Technology in Primary and Secondary Education pp 1–19Cite as

Measuring the Impact of Emerging Technologies in Education: A Pragmatic Approach

  • Living reference work entry
  • First Online:

Part of the book series: Springer International Handbooks of Education ((SIHE))

Abstract

The evaluation of emerging technologies is important for their impacts to be effectively integrated into learning and teaching settings to bring the best benefit to learners and teachers. Educators, learners, parents, and policymakers alike, therefore, need reliable methodologies for evaluating the effectiveness of such emerging technologies. However, the impact evaluations of technology in education are challenging. This challenge is more significant for emerging technologies, as change is the essence of emerging educational technologies. Therefore, the value of traditional impact evaluations in education requires being reconsidered within this context. Here, we present a pragmatic approach to measuring the impact of emerging technologies in education which focuses on the suitability of the proposed evaluation methods and the types of evidence rather than on the hierarchy of these methods and evidence types. The approach has two main steps. First one is the creation of a clear theory of change to identify outcome measure(s) and assumptions that are behind the expected impact of the emerging technology intervention. Second is the identification of the type of evidence and methods to generate it that are the most appropriate for the current innovation stage of the emerging technology.

This is a preview of subscription content, log in via an institution.

References

  • Ainley, J., Enger, L., & Searle, D. (2008). Students in a digital age: Implications of ICT for teaching and learning. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 63–80). New York: Springer.

    Chapter  Google Scholar 

  • Bakker, M., Heuvel-Panhuizen, M., & Robitzsch, A. (2015). Longitudinal data on the effectiveness of mathematics mini-games in primary education. British Journal of Educational Technology, 46(5), 999–1004.

    Article  Google Scholar 

  • Beraza, I., Pina, A., & Demo, B. (2010, November). Soft & hard ideas to improve interaction with robots for kids & teachers. In Workshop proceedings of conference on simulation, modelling, and programming for autonomous robots, (pp. 549–557). Darmstadt (Germany).

    Google Scholar 

  • Biesta, G. (2007). Why “what works” won’t work: Evidence-based practice and the democratic deficit in educational research. Educational Theory, 57(1), 1–22.

    Article  Google Scholar 

  • Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141–178.

    Article  Google Scholar 

  • Buck, S., & McGee, J. (2015). Why government needs more randomized controlled trials: Refuting the myths. Houston: Laura and John Arnold Foundation. www.arnoldfoundation.org.

  • Cobb, P., Confrey, J., DiSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.

    Article  Google Scholar 

  • Coburn, C. E., Russell, J. L., Kaufman, J. H., & Stein, M. K. (2012). Supporting sustainability: Teachers’ advice networks and ambitious instructional reform. American Journal of Education, 119(1), 137–182.

    Article  Google Scholar 

  • Cole, K. C., Van Tilburg, D., Burch-Vernon, A., & Riccio, D. C. (1996). The importance of context in the US preexposure effect in CTA: Novel versus latently inhibited contextual stimuli. Learning and Motivation, 27(4), 362–374.

    Article  Google Scholar 

  • Condie, R., & Munro, B. (2007). The impact of ICT in schools: Landscape review. UCL Institute of Education. London: United Kingdom

    Google Scholar 

  • Cox, M. J. (2005). Educational conflict: The problems in institutionalizing new technologies in education. In G. Kouzelis, M. Pournari, M. Stoeppler, & V. Tselfes (Eds.), Knowledge in the new technology (pp. 139–165). Frankfurt: Peter Lang.

    Google Scholar 

  • Cox, M. J. (2008). Researching IT in education. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 965–981). New York: Springer.

    Google Scholar 

  • Cox, M. J., & Abbott, C. (Eds.). (2004). ICT and attainment – A review of the research literature. Full report. Coventry/London: Becta/DfES. http://www.becta.org.uk/research/index.cfm.

  • Cox, M. J., & Marshall, G. (2007). Effects of ICT: Do we know what we should know? Education and Information Technologies, 12, 59–70.

    Article  Google Scholar 

  • Cox, M. J., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2003). ICT and pedagogy: A review of the research literature (ICT in schools research and evaluation series, Vol. 18) London: UCL Institute of Education.

    Google Scholar 

  • Cox, M., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2004). An investigation of the research evidence relating to ICT pedagogy. A report to DfES. Retrieved from http://dera.ioe.ac.uk/1601/1/becta_2003_attainmentpedagogy_queensprinter.pdf

  • Cress, U., Moskaliuk, J., & Jeong, H. (Eds.). (2016). Mass collaboration and education (Vol. 16). Cham: Springer.

    Google Scholar 

  • Crompton, H., Burke, D., & Gregory, K. H. (2017). The use of mobile learning in PK-12 education: A systematic review. Computers & Education, 110, 51–63.

    Article  Google Scholar 

  • Cukurova, M., Luckin, R., & Baines, E. (2017). The significance of context for the emergence and implementation of research evidence: The case of collaborative problem-solving. Oxford Review of Education, 1–16. https://www.tandfonline.com/doi/abs/10.1080/03054985.2017.1389713.

  • Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solving from students’ physical interactions. Computers & Education, 116, 93–109.

    Article  Google Scholar 

  • De Bruin, C. L. (2015). Conceptualizing effectiveness in disability research. International Journal of Research & Method in Education. https://doi.org/10.1080/1743727X.2015.1033391.

  • Department for International Development. (2017). Public Bulleting Board 7749: Education technology to improve learning for all. Retrieved from https://supplierportal.dfid.gov.uk/selfservice/pages/public/viewPublicNotice.cmd?bm90aWNlSWQ9Njc3MjA%3D.

  • Dunleavy, M., & Dede, C. (2014). Augmented reality teaching and learning. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 735–745). New York: Springer.

    Google Scholar 

  • Fullan, M. (1983). Evaluating program implementation: What can be learned from Follow Through. Curriculum Inquiry, 13(2), 215–227.

    Article  Google Scholar 

  • Fullan, M. (2001). Leading in a culture of change. San Francisco: Jossey-Bass.

    Google Scholar 

  • Greany, T., & Maxwell, B. (2017). Evidence-informed innovation in schools: Aligning collaborative research and development with high quality professional learning for teachers. International Journal of Innovation in Education, 4(2–3), 147–170.

    Article  Google Scholar 

  • Gulson, K. N., & Symes, C. (2007). Spatial theories of education: Policy and geography matters. New York: Routledge.

    Google Scholar 

  • Hattie, J. (2008). Visible learning: A synthesis of over 800 meta analyses related to achievement. New York: Routledge.

    Google Scholar 

  • Hoeken, H. (2001). Anecdotal, statistical, and causal evidence: Their perceived and actual persuasiveness. Argumentation, 15(4), 425–437.

    Article  Google Scholar 

  • Katterfeldt, E., Cukurova, M., Spikol, D., & Cuartielles, D. (2018). Physical computing with plug-and-play toolkits: Key recommendations for collaborative learning implementations. International Journal of Child-Computer Interaction, 1–23. https://www.sciencedirect.com/science/article/pii/S2212868917300351.

  • Kelly, A. E., Baek, J. Y., Lesh, R. A., & Bannan-Ritland, B. (2008). Enabling innovations in education and systematizing their impact. In A. E. Kelly, R. A. Lesh, & J. Y. Baek (Eds.), Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching (pp. 3–18). Routledge, Abington, Oxon.

    Google Scholar 

  • Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review, 22(3), 215–243.

    Article  Google Scholar 

  • Marshall, G., & Cox, M. J. (2008). Research methods: Their design, applicability and reliability. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 983–1002). Boston: Springer.

    Chapter  Google Scholar 

  • Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Computers & Education, 70, 29–40.

    Article  Google Scholar 

  • Miller, D. J., & Robertson, D. P. (2011). Educational benefits of using game consoles in a primary classroom: A randomised controlled trial. British Journal of Educational Technology, 42(5), 850–864.

    Article  Google Scholar 

  • Nardi, B. A. (1996). Studying context: A comparison of activity theory, situated action models, and distributed cognition. In Context and consciousness: Activity theory and human-computer interaction (pp. 69–102). London: MIT Press.

    Google Scholar 

  • NESTA. (2016). Using research evidence for success: A practical guide. Retrieved on from https://www.nesta.org.uk/sites/default/files/using_research_evidence_for_success_-_a_practice_guide.pdf.

  • Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography: Synthesizing qualitative studies. Newbury Park: Sage.

    Book  Google Scholar 

  • O’Leary, Z. (2004). The essential guide to doing research. London: Sage.

    Google Scholar 

  • Pampaka, M., Williams, J., & Homer, M. (2016). Is the educational ‘what works’ agenda working? Critical methodological developments. International Journal of Research and Method in Education, 39, 231–236.

    Article  Google Scholar 

  • Perry, B. (2015). Gamifying French language learning: A case study examining a quest-based, augmented reality mobile learning-tool. Procedia-Social and Behavioral Sciences, 174, 2308–2315.

    Article  Google Scholar 

  • Petty, G. (2009). Evidence-based teaching: A practical approach. Cheltenham: Nelson Thornes.

    Google Scholar 

  • Pilkington, R. (2008). Measuring the impact of IT on students’ learning. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education. Berlin/Heidelberg/New York: Springer.

    Google Scholar 

  • Reeves, T. C. (2008). Evaluation of the design and development of IT tools in education. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education. Berlin/Heidelberg/New York: Springer.

    Google Scholar 

  • Slavin, R. E. (2017). Evidence-based reform in education. Journal of Education for Students Placed at Risk (JESPAR), 22(3), 178–184.

    Article  Google Scholar 

  • Stern, E. (2015). Impact evaluation: A guide for commissioners and manager. London: Big Lottery Fund and the Department for International Development.

    Google Scholar 

  • Tondeur, J., van Braak, J., Guoyuan, S., Voogt, J., Fisser, P., & Ottenbreitt-Leftwich, A. S. (2012). Preparing student teachers to integrate ICT in classroom practice: A synthesis of qualitative evidence. Computers & Education, 59(1), 134–144. https://doi.org/10.1016/j.compedu.2011.10.009.

    Article  Google Scholar 

  • Underwood, J. (2017). Exploring AI language assistants with primary EFL students. In CALL in a climate of change: Adapting to turbulent global conditions–EUROCALL 2017 (p. 317). Southampton: University of Southampton.

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. UCL Knowledge Lab, Institute of Education, University College London, London, UK

    Mutlu Cukurova & Rosemary Luckin

Authors
  1. Mutlu Cukurova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rosemary Luckin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mutlu Cukurova .

Editor information

Editors and Affiliations

  1. University of Amsterdam , Amsterdam, The Netherlands

    Joke Voogt

  2. University of North Texas , Denton, Texas, USA

    Gerald Knezek

  3. University of North Texas , Denton, Texas, USA

    Rhonda Christensen

  4. University of Otago , Dunedin, New Zealand

    Kwok-Wing Lai

Section Editor information

  1. King's College London, London, UK

    Margaret Cox

  2. Department of Child Development and Education, University of Amsterdam/ Windesheim University of Applied Sciences, Amsterdam, The Netherlands

    Joke Voogt

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Cukurova, M., Luckin, R. (2018). Measuring the Impact of Emerging Technologies in Education: A Pragmatic Approach. In: Voogt, J., Knezek, G., Christensen, R., Lai, KW. (eds) Second Handbook of Information Technology in Primary and Secondary Education . Springer International Handbooks of Education. Springer, Cham. https://doi.org/10.1007/978-3-319-53803-7_81-1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-53803-7_81-1

  • Received:

  • Accepted:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-53803-7

  • Online ISBN: 978-3-319-53803-7

  • eBook Packages: Springer Reference EducationReference Module Humanities and Social SciencesReference Module Education

Publish with us

Policies and ethics

Search

Navigation