Advertisement

Evaluation Research in Public Health

  • Angela J. DawsonEmail author
Reference work entry

Abstract

Evaluation research is concerned with assessing the merit of health projects and programs and produces information for decision-making to improve public health. Evaluation results are critical to continuous quality improvement efforts, building organizational capacity to respond to health needs and ensuring the accountable and efficient use of resources. This chapter will introduce evaluation research to assess the outcomes of health programs and policy. The key characteristics and principles of evaluation will be examined, and the range of approaches can be taken in this applied area of research. Examples of process, outcome, and impact evaluation in health contexts will enable readers to:
  1. 1.

    Discuss approaches to evaluation using logic models and theories of change

     
  2. 2.

    Examine program/project evaluation designs to assess methodological rigor and appropriateness

     
  3. 3.

    Apply knowledge of global/national/state strategies and public health evidence to guide the development of evaluation indicators

     
  4. 4.

    Examine the culturally appropriate and ethically sound approaches in evaluation

     

Keywords

Program evaluation Theory-based evaluation Theory of change Logical frameworks Results-based management Evaluation indicators Gender-sensitive evaluation 

References

  1. ACI. Understanding program evaluation an ACI framework, agency for clinical innovation. Chatswood, NSW Department of Health; 2013.Google Scholar
  2. Amaya M, Petosa R. An evaluation of a worksite exercise intervention using the social cognitive theory: a pilot study. Health Educ J. 2012;71(2):133–43.CrossRefGoogle Scholar
  3. Australian Communities Foundation. Theory of change. Fitzroy: Australian Communities Foundation; 2015. Viewed 17 Jan 2018, http://www.communityfoundation.org.au/about-acf/theory-of-change/.
  4. Azar FE, Solhi M, Nejhaddadgar N, Amani F. The effect of intervention using the PRECEDE-PROCEED model based on quality of life in diabetic patients. Electron Physician. 2017;9(8):5024–30.CrossRefGoogle Scholar
  5. Bauer G. Developing community health indicators to support comprehensive community building initiatives: A case study of a participatory action research project. (Dr.P.H.), University of California, Berkeley, California; 1999.Google Scholar
  6. Belkora J, Volz S, Loth M, Teng A, Zarin-Pass M, Moore D, Esserman L. Coaching patients in the use of decision and communication aids: RE-AIM evaluation of a patient support program. BMC Health Serv Res. 2015;15:209.  https://doi.org/10.1186/s12913-015-0872-6.CrossRefGoogle Scholar
  7. Breuer E, De Silva M, Shidaye R, Petersen I, Nakku J, Jordans M, Fekadu A, Lund C. Planning and evaluating mental health services in low-and middle-income countries using theory of change. Br J Psychiatry. 2015:s1–8.  https://doi.org/10.1192/bjp.bp.114.153841.CrossRefGoogle Scholar
  8. Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016;11:63.  https://doi.org/10.1186/s13012-016-0422-6.CrossRefGoogle Scholar
  9. Byng R, Norman I, Redfern S, Jones R. Exposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation. BMC Health Serv Res. 2008;8:271.  https://doi.org/10.1186/472-6963-8-274.CrossRefGoogle Scholar
  10. CDC. Framework for program evaluation in public health. CDC Evaluation Working Group, Centers for Disease Control and Prevention, No. RR-11. 1999. https://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm.
  11. Chen HT. Theory-driven evaluations. Thousand Oaks: Sage; 1990.Google Scholar
  12. Connell JP, Kubisch AC. Applying a theory of change approach to the evaluation of comprehensive community initiatives: progress, prospects, and problems. New approaches to evaluating community initiatives, 1998;2(15–44):1–16.Google Scholar
  13. Coryn CL, Noakes LA, Westine CD, Schröter DC. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval. 2011;32(2):199–226.CrossRefGoogle Scholar
  14. De Silva M, Lee L, Ryan G. Using theory of change in the development, implementation and evaluation of complex health interventions A practical guide. London: The Centre for Global Mental Health & the Mental Health Innovation Network; 2014a.Google Scholar
  15. De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C, Patel V. Theory of change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials. 2014b;15:267.  https://doi.org/10.1186/745-6215-15-267.CrossRefGoogle Scholar
  16. Foy R, Ovretveit J, Shekelle PG, Pronovost PJ, Taylor SL, Dy S, Hempel S, McDonald KM, Rubenstein LV, Wachter RM. The role of theory in research to develop and evaluate the implementation of patient safety practices. Qual Saf Health Care. 2011;20(5):453–9.Google Scholar
  17. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.CrossRefGoogle Scholar
  18. Green L, Kreuter M. Health program planning: an educational and ecological approach, vol. 4.New York: McGraw-Hill Higher Education; 2015.Google Scholar
  19. Habicht J-P, Victora C, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol. 1999;28(1):10–8.CrossRefGoogle Scholar
  20. Hailemariam M, Fekadu A, Selamu M, Alem A, Medhin G, Giorgis TW, DeSilva M, Breuer E. Developing a mental health care plan in a low resource setting: the theory of change approach. BMC Health Serv Res. 2015;15:429.  https://doi.org/10.1186/s12913-015-1097-4.CrossRefGoogle Scholar
  21. Hou RJ, Wong SY-S, Yip BH-K, Hung AT, Lo HH-M, Chan PH, Lo CS, Kwok TC-Y, Tang WK, Mak WW. The effects of mindfulness-based stress reduction program on the mental health of family caregivers: a randomized controlled trial. Psychother Psychosom. 2014;83(1):45–53.CrossRefGoogle Scholar
  22. Hubbard SM, Hayashi SW. Use of diffusion of innovations theory to drive a federal agency's program evaluation. Eval Program Plann. 2003;26(1):49–56.CrossRefGoogle Scholar
  23. IFRC. Project/programme monitoring and evaluation (M&E) guide. International Federation of Red Cross and Red Crescent Societies, Geneva; 2011.Google Scholar
  24. Jenkinson KA, Naughton G, Benson AC. The GLAMA (Girls! Lead! Achieve! Mentor! Activate!) physical activity and peer leadership intervention pilot project: a process evaluation using the RE-AIM framework. BMC Public Health. 2012;12(1):55.CrossRefGoogle Scholar
  25. Kegler MC, Twiss JM, Look V. Assessing community change at multiple levels: the genesis of an evaluation framework for the California Healthy Cities Project. Health Educ Behav. 2000;27(6):760–79.CrossRefGoogle Scholar
  26. Lakatos BE, Schaffer AC, Gitlin D, Mitchell M, Delisle L, Etheredge ML, Shellman A, Baytos M. A population-based care improvement initiative for patients at risk for delirium, alcohol withdrawal, and suicide harm. Jt Comm J Qual Patient Saf. 2015;41(7):291–AP3.CrossRefGoogle Scholar
  27. Lea S, Callaghan L, Eick S, Heslin M, Morgan J, Bolt M, Healey A, Barrett B, Rose D, Patel A. The management of individuals with enduring moderate to severe mental health needs: a participatory evaluation of client journeys and the interface of mental health services with the criminal justice system in Cornwall. National Institute of Health Research, Southampton, No. 3.15. 2015. https://www.ncbi.nlm.nih.gov/books/NBK285789/.
  28. Logan S, Boutotte J, Wilce M, Etkind S. Using the CDC framework for program evaluation in public health to assess tuberculosis contact investigation programs. Int J Tuberc Lung Dis. 2003;7(12):S375–S83.Google Scholar
  29. Marsh D, Aakesson A, Anah K. Community case management essentials: treating common childhood illnesses in the community. In: A guide for program managers. Washington, DC: CORE Group Save the Children BASICS MCHIP; 2012.Google Scholar
  30. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wight D. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.  https://doi.org/10.1136/bmj.h258.CrossRefGoogle Scholar
  31. Patton MQ. Utilization-focused evaluation. 4th ed. Thousand Oaks: Sage; 2008.Google Scholar
  32. Patton MQ, McKegg K, Wehipeihana N. Developmental evaluation exemplars: principles in practice. New York: Guilford Press; 2016.Google Scholar
  33. Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.Google Scholar
  34. Post DK, Daniel M, Misan G, Haren MT. A workplace health promotion application of the Precede-Proceed model in a regional and remote mining company in Whyalla, South Australia. Int J Workplace Health Manag. 2015;8(3):154–74.CrossRefGoogle Scholar
  35. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health Ment Health Serv Res. 2011;38(2):65–76.CrossRefGoogle Scholar
  36. Ramsay CR, Thomas RE, Croal BL, Grimshaw JM, Eccles MP. Using the theory of planned behaviour as a process evaluation tool in randomised trials of knowledge translation strategies: a case study from UK primary care. Implement Sci. 2010;5:71.  https://doi.org/10.1186/748-5908-5-71.CrossRefGoogle Scholar
  37. Reisman J, Gienapp A, Langley K, Stachowiak S. Theory of change a practical tool for action, results and learning. Seattle: Organizational Research Services, Annie E. Casey Foundation; 2004. http://www.aecf.org/resources/theory-of-change/.
  38. Rogers P. Theory of change, methodological briefs: impact evaluation, vol. 2. Florence: UNICEF Office of Research; 2014.Google Scholar
  39. Schierhout G, Hains J, Si D, Kennedy C, Cox R, Kwedza R, O’Donoghue L, Fittock M, Brands J, Lonergan K. Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change. Implement Sci. 2013;8:119.  https://doi.org/10.1186/748-5908-8-119.CrossRefGoogle Scholar
  40. Setiawan A, Dignam D, Waters C, Dawson A. Improving access to child health care in Indonesia through community case management. Matern Child Health J. 2016;20(11):2254–60.CrossRefGoogle Scholar
  41. Sinha RK, Haghparast-Bidgoli H, Tripathy PK, Nair N, Gope R, Rath S, Prost A. Economic evaluation of participatory learning and action with women’s groups facilitated by Accredited Social Health Activists to improve birth outcomes in rural eastern India. Cost Effectiveness and Resource Allocation, 2017;15(2),  https://doi.org/10.1186/s12962-12017-10064-12969.
  42. Sridharan S, Nakaima A. Ten steps to making evaluation matter. Eval Program Plann. 2011;34(2):135–46.CrossRefGoogle Scholar
  43. Sweet SN, Ginis KAM, Estabrooks PA, Latimer-Cheung AE. Operationalizing the RE-AIM framework to evaluate the impact of multi-sector partnerships. Implement Sci. 2014;9(1):74.CrossRefGoogle Scholar
  44. Taplin D, Clark C, Collins E, Colby D. Theory of change a series of papers to support development of theories of change based on practice in the field. New York: Center for Human Environments; 2013.Google Scholar
  45. UNDP. Handbook on planning, monitoring and evaluating for development results. New York: United Nations Development Programme; 2009. http://www.undp.org/eo/handbook.Google Scholar
  46. University of Kansas. Developing a framework or model of change. Kansas: Center for Community Health and Development, University of Kansas; 2017. Viewed 17 Jan 2018, http://ctb.ku.edu/en/4-developing-framework-or-model-change.
  47. USAID. Evaluation learning from experience: USAID evaluation policy. Washington, DC: U.S. Agency for International Development; 2011.Google Scholar
  48. Van Belle SB, Marchal B, Dubourg D, Kegels G. How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa. BMC Public Health. 2010;10:741.  https://doi.org/10.1186/471-2458-10-741.CrossRefGoogle Scholar
  49. Vogel I. Review of the use of ‘theory of change’ in international development review report. UK Department of International Development, London; 2012.Google Scholar
  50. Weiss CH. Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives for children and families. In: Connell JP, Kubisch AC, Schorr LB, Weiss CH, editors. New approaches to evaluating community initiatives: concepts, methods and context. Washington DC: The Aspen Institute; 1995. p. 65–92.Google Scholar
  51. Weitzman BC, Silver D, Dillman K-N. Integrating a comparison group design into a theory of change evaluation: the case of the urban health initiative. Am J Eval. 2002;23(4):371–85.CrossRefGoogle Scholar
  52. WHO. WHO evaluation practice handbook. Geneva: World Health Organization; 2013.Google Scholar
  53. Young SL, Tuthill E. Ethnography as a tool for formative research and evaluation in public health nutrition: illustrations from the world of infant and young child feeding. In: Chrzan J, Brett J, editors. Research methods for anthropological studies of food and nutrition: food research. New York: Berghahn Books; 2017.Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Australian Centre for Public and Population Health ResearchUniversity of Technology SydneySydneyAustralia

Personalised recommendations