Rethinking ICT4D Impact Assessments: Reflections from the Siyakhula Living Lab in South Africa

  • Hafeni MthokoEmail author
  • Caroline Khene
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 933)


The approach to outcome and impact assessments of ICTD has often relied solely on identifying project effects in relation to project baseline data; however, such an approach limits the potential learning that could be occurring throughout a project’s lifecycle. Impact assessments should be conducted in a comprehensive manner, taking into account the evaluation data that has been captured from the initiation of the project through to its implementation, and beyond. This study sought to reflect on the implementation of an impact assessment framework that is based on a comprehensive approach to evaluation. The framework was implemented in the Siyakhula Living Lab to assess for its outcomes and impacts on the community. A pragmatic approach was applied through a reflective process, to assess the utility of the framework within this context. Semi-structured interviews with project stakeholders were conducted to further gain insight into the comprehensive approach to conducting impact assessments. It was found that a comprehensive approach to assessing impacts provided a meaningful way to understand the effects of the ICTD initiative, and provided an overview of project areas that required improvement. However, it was found that the proposed assessment framework required a customisation component in order to modify it to better suit the project context. The way in which future impact assessments are conducted can draw on the lessons gained from following a more comprehensive approach to evaluation, and thus improve learning over time.


Impact assessment Evaluation Learning Living lab Sustainability 


  1. 1.
    Heeks, R.: Information and Communication Technology for Development (ICT4D). Routledge, London (2018)Google Scholar
  2. 2.
    Walsham, G.: ICT4D research: reflections on history and future agenda. Inf. Technol. Dev. 23(1), 1–24 (2017)CrossRefGoogle Scholar
  3. 3.
    Gomez, R., Pather, S.: ICT evaluation: are we asking the right questions? Electron. J. Inf. Syst. Dev. Ctries. 50, 1–14 (2012)Google Scholar
  4. 4.
    Heeks, R., Molla, A.: Impact Assessment of ICT-for-Development Projects: A Compendium of Approaches (2009)Google Scholar
  5. 5.
    Pade-Khene, C., Lannon, J.: Learning to be sustainable in ICT for development: a citizen engagement initiative in South Africa. In: Choudrie, J., Islam, M.S., Wahid, F., Bass, J.M., Priyatma, J.E. (eds.) ICT4D 2017. IAICT, vol. 504, pp. 475–486. Springer, Cham (2017). Scholar
  6. 6.
    Pade-Khene, C., Sewry, D.: Towards a Comprehensive Evaluation Framework for ICT for Development Evaluation – An Analysis of Evaluation Frameworks. Presented at the (2011)Google Scholar
  7. 7.
    Mthoko, H., Khene, C.: Building theory in ICT4D evaluation: a comprehensive approach to assessing outcome and impact. Inf. Technol. Dev. (2017). Scholar
  8. 8.
    Pade-Khene, C., Lannon, J.: Learning to be Sustainable in ICT for Development: A Citizen Engagement Initiative in South Africa. Presented at the (2017)CrossRefGoogle Scholar
  9. 9.
    Toyama, K.: Geek Heresy: Rescuing Social Change from the Cult of Technology. Public Affairs, New York (2015)Google Scholar
  10. 10.
    Dodson, L., Sterling, S.R., Bennett, J.K.: Considering failure: eight years of ITID research. In: Proceedings of the Fifth International Conference on Information and Communication Technologies and Development, pp. 56–64. Georgia (2012)Google Scholar
  11. 11.
    Pade-Khene, C., Sewry, D.: Proposed stages of a rural ICT comprehensive evaluation framework in ICT for rural development projects. In: Proceedings of the South African Institute of Computer Scientists and Information Technologists Conference on Knowledge, Innovation and Leadership in a Diverse, Multidisciplinary Environment, SAICSIT 2011, p. 326 (2011).
  12. 12.
    Rossi, P.H., Lipsey, M.W., Freeman, H.E.: Evaluation: A Systematic Approach. Sage Publications, London (2004)Google Scholar
  13. 13.
    Weick, K.E.: What theory is not, theorizing is. Adm. Sci. Q. 40, 385–390 (1995)CrossRefGoogle Scholar
  14. 14.
    Gigler, B.S.: Including the excluded: can ICTs empower poor communities? Towards an alternative evaluation framework based on the capability approach. Presented at the (2004)Google Scholar
  15. 15.
    Mansell, R.: Power and interests in information and communication and development: exogenous and endogenous discourses in contention. J. Int. Dev. 26, 109–127 (2011)CrossRefGoogle Scholar
  16. 16.
    Brydon, L.: Ethical practices in doing development research. In: Doing Development Research, pp. 25–33. Sage Publications, London (2006)Google Scholar
  17. 17.
    Grunfeld, H.: Framework for evaluating contributions of ICT to capabilities, empowerment and sustainability in disadvantaged communities. In: CPRSouth2 (Communication Policy Research) Conference, “Empowering Rural Communities Through ICT Policy and Research, Madras (2007)Google Scholar
  18. 18.
    Krauss, K.: Ethical research practice for community entry: using ICT4D in a deep rural context. In: 3rd International IDIA Development Informatics Conference, Kruger National Park (2009)Google Scholar
  19. 19.
    Brunet, P., Tiemtore, O., Vettraino-Soulard, M.C.: Ethics and the Internet in West Africa: Toward an Ethical Model of Integration. IDRC, Ottawa (2004)Google Scholar
  20. 20.
    McNamara, K.S.: Information and Communication Technologies, Poverty and Development: Learning from Experience. InfoDev, Geneva (2003)Google Scholar
  21. 21.
    Mthoko, H., Pade-Khene, C.: Towards a theoretical framework on ethical practice in ICT4D programmes. Inf. Dev. J. 1–18 (2012). Scholar
  22. 22.
    Hallikainen, P., Chen, L.: A holistic framework on information systems evaluation with a case analysis. Electron. J. Inf. Syst. Eval. 9, 57–64 (2005)Google Scholar
  23. 23.
    Hamel, J.: ICT4D and the human development and capabilities approach: the potentials of information and communication technology. United Nations Development Programme, New York (2010)Google Scholar
  24. 24.
    Batchelor, S., Norrish, P.: Framework for Assessment of ICT Pilot Projects: Beyond Monitoring and Evaluation to Applied Research. InfoDev, The World Bank, Washington D.C (2006)Google Scholar
  25. 25.
    Rabaya, K.S.: Economic and social empowerment of women through information and communication technology: a case study of palestine. J. Commun. Inform. 5Google Scholar
  26. 26.
    Wieman, A., Gast, L., Hagen, I., Van Der Krogt, S.: Monitoring and Evaluation at IICD: An Instrument to Assess Development Impact and Effectiveness. International Institute for Communication and Development (IICD), The Hague (2001)Google Scholar
  27. 27.
    The World Bank: ICT for Greater Development Impact-Information and Communication Technology. The World Bank, Washington D.C (2012)Google Scholar
  28. 28.
    Alsop, R., Bertelsen, M.F., Holland, J.: Empowerment in Practice: From Analysis to Implementation. The World Bank, Washington D.C (2006)Google Scholar
  29. 29.
    Parkinson, S., Ramirez, R.: Using a sustainable livelihoods approach to assessing the impact of ICTs in development. Commun. Inform. 2, 116–127 (2006)Google Scholar
  30. 30.
    Duncombe, R.: Using the livelihoods framework to analyse ICT applications for poverty reduction through microenterprise. Inf. Technol. Int. Dev. 3, 81–100 (2006)CrossRefGoogle Scholar
  31. 31.
    Marais, M., Meyer, I.: Design for sustainability: countering the drivers of unsustainability in development projects. J. Commun. Inform. 11, 1–13 (2015)Google Scholar
  32. 32.
    Souter, D.: ICTS, the Internet and Sustainability: A Discussion Paper. International Institute for Sustainable Development, Winnipeg (2012)Google Scholar
  33. 33.
    Hilty, L.M., Hercheui, M.D.: ICT and sustainable development. In: Berleur, J., Hercheui, M.D., Hilty, L.M. (eds.) CIP/HCC -2010. IAICT, vol. 328, pp. 227–235. Springer, Heidelberg (2010). Scholar
  34. 34.
    Ali, M., Bailur, S.: The Challenge of “Sustainability” in ICT4D - Is Bricolage the Answer? Presented at the (2007)Google Scholar
  35. 35.
    Unwin, T.: ICT4D: Information and Communication Technology for Development. Cambridge University Press, Cambridge (2009)Google Scholar
  36. 36.
    Goldkuhl, G.: Pragmatism vs interpretivism in qualitative information systems research. Eur. J. Inf. Syst. 21, 135–146 (2012)CrossRefGoogle Scholar
  37. 37.
    Feilzer, M.Y.: Doing mixed methods research pragmatically: implications for the rediscovery of pragmatism as a research paradigm. J. Mix. Methods Res. 4, 6–16 (2010)CrossRefGoogle Scholar
  38. 38.
    Pade-Khene, C., Palmer, R., Kavhai, M.: A baseline study of a dwesa rural community for the siyakhula information and communication technology for development project: understanding the reality on the ground. Inf. Dev. 26, 265–288 (2010). Scholar
  39. 39.
    Yin, R.K.: Case Study Research: Design and Methods. SAGE Publications, Thousand Oaks (2009)Google Scholar
  40. 40.
    O’Neill, D.: Assessing community informatics: a review of methodological approaches for evaluating community networks and community technology centers. Internet Res. Electron. Netw. Appl. Policy 12, 76–102 (2002)CrossRefGoogle Scholar
  41. 41.
    Rogers, P., et al.: IE4D – Impact Evaluation for Development: Principles for Action (2011)Google Scholar
  42. 42.
    Patton, M.Q.: Utilization-Focused Evaluation Checklist (U-FE). SAGE Publications, London (2012)Google Scholar
  43. 43.
    Maxwell, S.: The Meaning and Measurement of Poverty. Overseas Development Institute (ODI), London (1999)Google Scholar
  44. 44.
    Markey, S., Connelly, S., Roseland, M.: Back of the envelope: pragmatic planning for sustainable rural community development. Plan. Pract. Res. 25, 1–23 (2010)CrossRefGoogle Scholar
  45. 45.
    Henry, G.T., Mark, M.M.: Beyond use: understanding evaluation’s influence on attitudes and actions. Am. J. Eval. 24, 293–314 (2003)Google Scholar
  46. 46.
    Myers, M.: Monitoring and Evaluating Information and Communication for Development (ICD) Programmes: Guidelines. Department for International Development (DFID), London (2005)Google Scholar
  47. 47.
    Pfeffer, J., Sutton, R.I.: The Knowing-Doing Gap. Harvard Business School Press, Boston (2000)Google Scholar
  48. 48.
    Tabassum, G., Yeo, A.W.: Measurement of tangible and intangible impacts of telecentres on rural communities. Presented at the (2015)Google Scholar
  49. 49.
    Atkinson, R.: The life story interview. In: Handbook of Interview Research: Context and Method, pp. 120–140. Thousand Oaks, CA (2001)Google Scholar
  50. 50.
    Hosman, L., Fife, E.: Improving the prospects for sustainable ICT projects in the developing world. Int. J. Media Cult. Polit. 4, 51–69 (2008)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of Cape TownCape TownSouth Africa
  2. 2.Rhodes UniversityGrahamstownSouth Africa

Personalised recommendations