Illustrating the need for a ‘Theory of Change’ in implementation processes

Abstract

The purpose of this study is to present the need for stakeholders to share an agreed-on Theory of Change (ToC) during the various phases of an implementation process. To this end, we draw on the Swedish Boost for Mathematics (BM), a large-scale national educational initiative, and its intrinsic ‘algebra’ instructional module. More precisely, we analyze a cohort of official evaluation reports on the BM from the perspective of a critical case study. The BM’s embedded algebra module and its associated material offer an instrumental example. This analysis relies on a selection of theoretical constructs related to Implementation Research (IR), such as (educational) innovation, value, stakeholders, degree of merit, policy dimension, theory-driven evaluation, and ToC. These theoretical constructs derive from both general IR and mathematics education research. We identify four phases of implementation, based on the analyses of the BM, namely, policy and organization, development, implementation, and evaluation. The analyses illustrate the need for alignment among these phases, associated with the endorsed, enacted, recognized and applied ToC of stakeholders. As a critical example, BM illustrates the many pitfalls of large-scale, scaled-up educational innovations—one of these being the potential failure of evaluations—owing to lack of an agreed-on ToC.

This is a preview of subscription content, access via your institution.

Fig. 1

Notes

  1. 1.

    After the official evaluations of the BM, the algebra module was updated in August 2018 to include a text about programming.

  2. 2.

    Skolverket is the Swedish government’s central administrative authority for the public school system. A school’s responsibilities encompass training principals, further education, professional development, and providing information about, and disseminating, knowledge about education (Skolverket 2017).

  3. 3.

    The NCM is the Swedish national resource center for mathematics. The NCM’s main responsibility is to support the development of Swedish mathematics education. The NCM is an independent institution at the University of Gothenburg, http://ncm.gu.se.

  4. 4.

    The translation of Matematiklyftet into Boost for Mathematics is Skolverket’s own.

  5. 5.

    When referring to the Lärportal we refer exclusively to the BM internet portal: https://larportalen.skolverket.se/#/.

  6. 6.

    Personal communication from Peter Nyström, director of the NCM, May 16, 2018.

  7. 7.

    https://larportalen.skolverket.se/#/modul/1-matematik/Grundskola/431_algebra%20%C3%A5k7-9 (accessed July 14, 2020).

  8. 8.

    Learning studies is a reinterpretation of lesson studies, where a chosen theory is used to design a lesson.

  9. 9.

    Excluding the video from 2017, which was added after the evaluation of the BM.

  10. 10.

    Kieran (1992) found that a change in students’ understanding of the equals sign from operational to indicating equivalence requires arithmetical experience of expressions that include at least two operations, on both the right- and left-hand sides of the equals sign, e.g., 7 ∙ 2 + 3 − 2 = 5 ∙ 2 − 1 + 6.

References

  1. Bason, C. (2010). Leading public sector innovation: Co-creating for a better society. Bristol: The Policy Press.

    Google Scholar 

  2. Bentley, P.-O. (2008). TIMSS 2007 Swedish pupils’ mathematical knowledge (An analysis, Report 323). https://www.skolverket.se/publikationsserier/ovrigt-material/2009/timss-2007-swedish-pupils-mathematical-knowledgeSkolverket.

  3. Cai, J., Morris, A., Hohensee, C., Hwang, S., Robison, V., & Hiebert, J. (2017). Making classroom implementation an integral part of research. Journal for Research in Mathematics Education, 48(4), 342–347. https://doi.org/10.5951/jresematheduc.48.4.0342.

    Article  Google Scholar 

  4. Century, J., & Cassata, A. (2016). Implementation research: Finding common ground on what, how, why, where, and who. Review of Research in Education, 40(1), 169–215. https://doi.org/10.3102/0091732X16665332.

    Article  Google Scholar 

  5. Chen, H. T. (1990). Theory-driven evaluations. Thousand Oaks: Sage.

    Google Scholar 

  6. Chen, H. T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective (2nd ed.). Thousand Oaks: SAGE Publications.

    Google Scholar 

  7. Clement, J. (1982). Algebra word problem solutions: Thought processes underlying a common misconception. Journal for Research in Mathematics Education, 13(1), 16–30.

    Article  Google Scholar 

  8. Cohen, E., & Kanim, S. E. (2005). Factors influencing the algebra ‘reversal error.’ American Journal of Physics, 73(11), 1072–1078. https://doi.org/10.1119/1.2063048.

    Article  Google Scholar 

  9. Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226. https://doi.org/10.1177/1098214010389321.

    Article  Google Scholar 

  10. Dysthe, O. (1996). The multivoiced classroom: Interactions of writing and classroom discourse. Written Communication, 13(3), 385–425. https://doi.org/10.1177/0741088396013003004.

    Article  Google Scholar 

  11. Ekdahl, A.-L. & Olteanu, C. (2014). Algebra årskurs 7–9. [Algebra grades 7–9]. Lärportal, July 14, 2020.

  12. Flyvbjerg, B. (2006). Five misunderstandings about case-study research. Qualitative Inquiry, 12(2), 219–245. https://doi.org/10.1177/1077800405284363.

    Article  Google Scholar 

  13. Gregersen, R. M., Lauridsen, S. D. & Jankvist, U. T. (2019). Operationalizing implementation theory in mathematics education research—Identifying enablers and barriers in the Swedish ‘Boost for Mathematics’. In U. T. Jankvist, M. Van der Heuvel-Panhuizen & M. Veldhuis (Eds.), Proceedings of the Eleventh Congress of the European Society for Research in Mathematics Education (pp. 4373–4380). Freudenthal Group & Freudenthal Institute, Utrecht University and ERME.

  14. Gunnarsson, R. (2014). Tolka formler. [Interpreting formulas]. Lärportal, July 14, 2020.

  15. Jankvist, U. T., Aguilar, M. S., Dreyøe, J. & Misfeldt, M. (2019). Adapting implementation research frameworks for mathematics education. In U. T. Jankvist, M. Van den Heuvel-Panhuizen & M. Veldhuis (Eds.), Proceedings of the Eleventh Congress of the European Society for Research in Mathematics Education (pp. 4405–4412). Freudenthal Group & Freudenthal Institute, Utrecht University and ERME.

  16. Jankvist, U. T., & Niss, M. (2020). Fostering an intimate interplay between research and practice: Danish ‘maths counsellors’ for upper secondary school. Nordic Studies in Mathematics Education, 25(2), 99–118.

    Google Scholar 

  17. Kaput, J., & Clement, J. (1979). Letter to the editor of JCMB. Journal of Children’s Mathematical Behavior, 2(2), 208.

    Google Scholar 

  18. Kieran, C. (1992). The learning and teaching of school Algebra. In D. A. Grouws (Ed.), Handbook of research on mathematics teaching and learning (pp. 390–419). New York: Macmillan.

    Google Scholar 

  19. Krainer, K. (2014). Teachers as stakeholders in mathematics education research. The Mathematics Enthusiast, 11(1), 49–60.

    Google Scholar 

  20. Lauridsen, S. D., & Gregersen, R. M. (2017). Implementering af matematikdidaktiske forskningsresultater i praksi. (Unpublished master’s thesis). Emdrup: Aarhus University.

    Google Scholar 

  21. Levenson, E., Tirosh, D., & Tsamir, P. (2009). Students’ perceived sociomathematical norms: The missing paradigm. The Journal of Mathematical Behavior, 28(2–3), 171–187.

    Article  Google Scholar 

  22. Levine, D. U., & Cooper, E. J. (1991). The change process and its implications in teaching thinking. In L. Idol & B. Jones (Eds.), Educational values and cognitive instruction (pp. 387–408). New York: Routledge.

    Google Scholar 

  23. Marton, F. (1981). Phenomenography—Describing conceptions of the world around us. Instructional Science, 10(2), 177–200. https://doi.org/10.1007/bf00132516.

    Article  Google Scholar 

  24. Marton, F., & Tsui, A. B. M. (Eds.). (2004). Classroom discourse and the space of learning. Mahwah: Lawrence Erlbaum.

    Google Scholar 

  25. Merriam, S. B. (1998). Qualitative research and case study applications in education (Rev. and expanded edition.). San Francisco: Jossey-Bass.

    Google Scholar 

  26. Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10(1), 53. https://doi.org/10.1186/s13012-015-0242-0.

    Article  Google Scholar 

  27. Olteanu, C. (2014). Interaktion i algebraklassrummet [Interaction in the algebra classroom]. Lärportal, July 14, 2020.

  28. Österholm, M., Bergqvist, T., Liljekvist, Y., & Bommel, J. V. (2016). Utvärdering av Matematiklyftets resultat: Slutrapport. [Evaluation of the results of the Boost for Mathematics: Final report]. Stockholm: Utbildningsdepartementet.

    Google Scholar 

  29. Philipp, R. A. (2007). Mathematics teachers’ beliefs and affect. In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 257–315). Charlotte: Information Age.

    Google Scholar 

  30. Rambøll. (2013). Utvärdering matematiklyftets utprövningsomgång [Evaluation of the Boost for Mathematics: small-scale trial]. Stockholm: Utbildningsdepartementet.

    Google Scholar 

  31. Rambøll (2016). Slututvärdering utvärderingen av Matematiklyftet 2013–2016 [Final evaluation. Evaluation of the Boost for Mathematics 2013–16]. Stockholm: Utbildningsdepartementet.

  32. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press.

    Google Scholar 

  33. Rogers, P. J. (2000). Program theory evaluation: Not whether programs work but how they work. In D. L. Stufflebeam, G. F. Madaus, & T. Kellaghan (Eds.), Evaluation models: Viewpoints on educational and human services evaluation, 49 (pp. 209–232). Boston: Kluwer.

    Google Scholar 

  34. Rogers, P., Petrosino, A., Huebner, T., & Hacsi, T. (2000). Program theory evaluation: Practice, promise, and problems. New Directions for Evaluation, 2000(87), 5–13. https://doi.org/10.1002/ev.1177.

    Article  Google Scholar 

  35. Rosnick, P. (1981). Some misconceptions concerning the concept of variable. Mathematics Teacher, 74(6), 418–420.

    Article  Google Scholar 

  36. Schumpeter, J. A. (1934). The fundamental phenomenon of economic development. In The theory of economic development (pp. 57–94). New York: Oxford University Press. https://www.hup.harvard.edu/catalog.php?isbn=9780674879904&content=toc.

  37. Skolverket. (2011). Utredning och förslag på en didaktisk fortbildning för alla matematiklärare, Delredovisning av regeringsuppdrag [Investigation and proposals for a didactic in-service training for all mathematics teachers, Interim report on government assignments]. U2011/2229/G. Stockholm: Utbildningsdepartementet.

  38. Skolverket. (2012a). Delredovisning av uppdrag om att svara för utbildning [Interim report on the assignment of being responsible for education]. Dnr 2011:643. Stockholm: Utbildningsdepartementet.

    Google Scholar 

  39. Skolverket. (2012b). Matematiklyftet (Beslut) [Boost for Mathematics (decision)] Dnr 2011:643. Stockholm: Utbildningsdepartementet.

    Google Scholar 

  40. Skolverket. (2013a). Delredovisning av uppdrag om att svara för utbildning [Interim report on the assignment of being responsible for education]. Dnr 2011:643. Stockholm: Utbildningsdepartementet.

    Google Scholar 

  41. Skolverket. (2013b). Matematiklyftet: kollegialt lärande för matematiklärare [The boost for mathematics: Collegiate learning for mathematics teachers]. Stockholm: Utbildningsdepartementet.

    Google Scholar 

  42. Skolverket. (2016). Slutredovisning av uppdrag om att svara för utbildning [Final report on the assignment of being responsible for education]. Dnr 2011:643. Stockholm: Utbildningsdepartementet.

    Google Scholar 

  43. Skolverket. (2017). Skolverkets årsredovisning 2016 [Skolverket’s yearly report 2016]. Stockholm: Utbildningsdepartementet.

    Google Scholar 

  44. Skolverket. (2018). Curriculum for the compulsory school, preschool class and school-age educare. Stockholm: Utbildningsdepartementet.

    Google Scholar 

  45. Stake, R. (1995). The art of case study research. Thousand Oaks: Sage.

    Google Scholar 

  46. Trouche, L., Drijvers, P., Gueudet, G., & Sacristán, A. I. (2013). Technology-driven developments and policy implications for mathematics education. In M. A. Clements, A. J. Bishop, C. Keitel, J. Kilpatrick, & F. K. S. Leung (Eds.), Third international handbook of mathematics education (pp. 753–789). New York: Springer.

    Google Scholar 

  47. Utbildningsdepartementet, P. (2012). Regeringsbeslut: Uppdrag att svara för utbildning [Governmental decision: Responsibility assignment of qualification]. (I:44). Stockholm: Statens Skolverk.

    Google Scholar 

  48. Weiss, C. H. (1995). Nothing as practical as a good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. P. Connell, A. C. Kubisch, L. B. Schorr, & C. H. Weiss (Eds.), New approaches to evaluating community initiatives: Concepts, methods, and contexts (pp. 65–92). Washington: Aspen Institute.

    Google Scholar 

  49. Yackel, E., & Cobb, P. (1996). Sociomathematical norms, argumentation, and autonomy in mathematics. Journal for Research in Mathematics Education, 27(4), 458–477. https://doi.org/10.2307/749877.

    Article  Google Scholar 

Download references

Acknowledgements

This paper was partly written in the frame of project 2020-04090 under Swedish Research Council.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Uffe Thomas Jankvist.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jankvist, U.T., Gregersen, R.M. & Lauridsen, S.D. Illustrating the need for a ‘Theory of Change’ in implementation processes. ZDM Mathematics Education (2021). https://doi.org/10.1007/s11858-021-01238-1

Download citation

Keywords

  • Implementation research
  • Theory of change
  • Stakeholders
  • Value
  • Degree of merit