Advertisement

Program Evaluation

  • Peijia ZhaEmail author
Chapter

Abstract

The purpose of this chapter is to explain the purpose of program evaluation and its significance in determining effectiveness of initiatives and approaches toward vulnerable populations. Major concepts and principles guiding the development of program evaluation design, selection of data collection methods and data analysis, standards, process of implementation, outcomes and effects to be evaluated, and utilization of evaluation outcomes are presented. The framework for program evaluation in public health used by the Centers for Disease Control and Prevention can be a valuable tool in designing and implementing a program evaluation. Program evaluation can increase the capacity for monitoring outcomes, cost-effectiveness, and sustainability of programs particularly in high-risk vulnerable populations.

Keywords

Program evaluation Vulnerable population Program effectiveness 

References

  1. 1.
    Flaskerud JH, Lesser J, Dixon E, Anderson N, Conde F, Kim S, et al. Health disparities among vulnerable populations: evolution of knowledge over five decades in nursing research publications. Nurs Res. 2002;51(2):74–85.CrossRefGoogle Scholar
  2. 2.
    Agency for Healthcare Research and Quality (AHRQ). 2013 National healthcare disparities report; 2013. Available from https://www.ahrq.gov/research/findings/nhqrdr/nhdr13/index.html.
  3. 3.
    Silow-Carroll S, Alteras T. Community-based oral health programs: a need and plan for evaluation. Economic and Social Research Institute; 2005.Google Scholar
  4. 4.
    Advisory Commission on Consumer Protection and Quality in the Health Care Industry. Advisory Commission’s final report Chapter Eight: focusing on vulnerable populations; 1998. Available from http://archive.ahrq.gov/hcqual/final/chap08.html.
  5. 5.
    Substance Abuse and Mental Health Services Administration. Results from the 2009 National survey on drug use and health: volume I. Summary of national findings. Rockville, MD: Office of Applied Studies; 2010. NSDUH Series H-38A, HHS Publication No. SMA 10-4586Findings.Google Scholar
  6. 6.
    Rossi HP, Lipsey WM, Freeman EH. Evaluation: a systematic approach. 7th ed. Thousand Oaks, CA: Sage Publications, Inc; 2004.Google Scholar
  7. 7.
    Patton MQ. Utilization-focused evaluation: the new century text. 4th edn. Thousand Oaks, CA: Sage Publications, Inc.; 2008.Google Scholar
  8. 8.
    Habicht PJ, Victora GC, Vauqhan PJ. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol. 1999;28(1):10–8.CrossRefGoogle Scholar
  9. 9.
    World Health Organization. Health programme evaluation. Geneva: World Health Organization; 1985.Google Scholar
  10. 10.
    Levin-Rozalis M. Evaluation and research, differences and similarities. Can J Program Eval. 2003;18(2):1–31.Google Scholar
  11. 11.
    Beney T. Distinguishing Evaluation from Research; 2011. Available from http://www.uniteforsight.org/evaluation-course/module10.
  12. 12.
    Center for Disease Control and Prevention. Introduction to program evaluation for public health program: a self-study guide; 2011. Available from https://www.cdc.gov/eval/guide/index.htm.
  13. 13.
    Bliss MJ, Emshoff JG. Workbook for designing a process evaluation. Atlanta, GA: Georgia State University; 2002.Google Scholar
  14. 14.
    Hawe P, Degeling D, Hall J, Brierley A. Evaluating health promotion: a health worker’s guide. Sydney, NSW: MacLennan & Petty; 1990.Google Scholar
  15. 15.
    Linnell, D. Process evaluation vs. outcome evaluation; 2014. Available from http://tsne.org/process-evaluation-vs-outcome-evaluation.
  16. 16.
    Public Health Ontario. Evaluating health promotion programs [PDF document]; 2012. Available from http://www.publichealthontario.ca/en/eRepository/Evaluating_health_promotion_programs_2012.pdf.
  17. 17.
    Hassandra M, Zourbanos N, Kofou G, Gourgoulianis K, Theodorakis Y. Process and outcome evaluation of the “No more smoking! It’s time for physical activity” program. J Sport Health Sci. 2013;2(4):242–8.CrossRefGoogle Scholar
  18. 18.
    Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR. 1999;48(RR-11):1–40.Google Scholar
  19. 19.
    Yarbrough DB, Shulha LM, Hopson RK, Caruthers FA. The program evaluation standards: a guide for evaluators and evaluation users. 3rd ed. Thousand Oaks, CA: Sage Publications, Inc; 2011.Google Scholar
  20. 20.
    Connell JP, Kubisch AC. Applying a theory of change approach to the evaluation of comprehensive community initiatives: progress, prospects, and problems. In: Fulbright-Anderson K, Kubisch AC, Connell JP, editors. New approaches to evaluating community initiatives: theory, measurement, and analysis. Washington, DC: The Aspen Institute; 1998. p. 15–44.Google Scholar
  21. 21.
    Chen HT. Theory-driven evaluation: conceptual framework, application and advancement. Evaluation von Programmen und Projekten für eine demokratische Kultur. Wiesbaden: Springer; 2012. p. 17–40.CrossRefGoogle Scholar
  22. 22.
    U.S. General Accounting Office. Managing for results: measuring program results that are under limited federal control. Washington, DC: U.S. General Accounting Office; 1999.Google Scholar
  23. 23.
    Haddix CA, Teutsch SM, Corso PS. Prevention effectiveness: a guide to decision analysis and economic evaluation. New York, NY: Oxford University Press; 2003.Google Scholar
  24. 24.
    Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, editors. Cost-effectiveness in health and medicine. Oxford: Oxford University Press; 2016.Google Scholar
  25. 25.
    Eoyang GH, Berkas T. Evaluation in a complex adaptive system. In: Lissack MR, Gunz HP, editors. Managing complexity in organizations. Westport, CT: Quorum Books; 1999. p. 313–5.Google Scholar
  26. 26.
    Fitzpartick JL, Sanders JR, Worthen BR. Program evaluation: alternative approaches and practical guidelines. 4th ed. Upper Saddle River, NJ: Pearson Education, Inc; 2011.Google Scholar
  27. 27.
    Wholey JS. Evaluation and effective public management. Boston, MA: Scott Foresman & Co.; 1983.Google Scholar
  28. 28.
    McLaughlin JA, Jordan GB. Logic models: a tool for telling your programs performance story. Eval Program Plann. 1999;22(1):65–72.CrossRefGoogle Scholar
  29. 29.
    Rush B, Ogbourne A. Program logic models: expanding their role and structure for program planning and evaluation. Can J Program Eval. 1991;6(2):95–106.Google Scholar
  30. 30.
    Kellogg, W. K. Foundation. Using logic models to bring together planning, evaluation, and action logic model development guide. Battle Creek, MI: W.K. Kellogg Foundation; 2004.Google Scholar
  31. 31.
    Fetterman DM, Kaftarian SJ, Wandersman AH. Empowerment evaluation: knowledge and tools for self-assessment and accountability. 2nd ed. Thousand Oaks, CA: Sage Publications, Inc.; 2014.Google Scholar
  32. 32.
    Weiss CH. Evaluation: methods for studying programs and policies. 2nd ed. Upper Saddle River, NJ: Prentice Hall; 1998.Google Scholar
  33. 33.
    Bickman L, Rog DJ. Applied research design: a practical approach. In: Bickman L, Rob DJ, editors. The SAGE handbook of applied social research methods. Thousand Oaks, CA: Sage Publications, Inc.; 2008. p. 3–42.Google Scholar
  34. 34.
    McQueen DV. Perspectives on health promotion: theory, evidence, practice and the emergence of complexity. Health Promot Int. 2000;15(2):95–7.CrossRefGoogle Scholar
  35. 35.
    Cook TD, Campbell DT. Quasi-experimentation: design and analysis issues for field settings. Boston, MA: Houghton Mifflin; 1979.Google Scholar
  36. 36.
    Taylor-Powell E, Steele S, Douglah M. Planning a program evaluation. Madison, WI: University of Wisconsin Cooperative Extension; 1996.Google Scholar
  37. 37.
    Fitzpatrick JL, Morris M. Current and emerging ethical challenges in evaluation: new directions for program evaluation, Number 82, vol. 46. San Francisco, CA: Jossey-Bass; 1999.Google Scholar
  38. 38.
    Newman DL, Brown RD. Applied ethics for program evaluation. Thousand Oaks, CA: Sage Publications, Inc; 1996.Google Scholar
  39. 39.
    Perrin EB, Koshel JJ. Assessment of performance measures for public health, substance abuse, and mental health. Washington, DC: National Academy Press; 1997.Google Scholar
  40. 40.
    Durch JS, Bailey LA, Stoto MA. Improving health in the community: a role for performance monitoring. Washington, DC: National Academies Press; 1997.Google Scholar
  41. 41.
    Centers for Disease Control and Prevention. Introduction to program evaluation for public health programs: evaluating appropriate antibiotic use programs. Atlanta, GA: Centers for Disease Control and Prevention; 2006.Google Scholar
  42. 42.
    Holloway I, Wheeler S. Qualitative research in nursing and healthcare. 3rd ed. Oxford: John Wiley & Sons; 2013.Google Scholar
  43. 43.
    Patton MQ. Qualitative evaluation and research methods. 4th ed. Newbury Park, CA: Sage Publications, Inc.; 2015.Google Scholar
  44. 44.
    Lipsy MW. What can you build with thousands of bricks? Musings on the cumulation of knowledge in program evaluation. New Direct Eval. 1997;1997(76):7–23.CrossRefGoogle Scholar
  45. 45.
    Lipsy MW, Hurley SM. Design sensitivity: statistical power for applied experimental research. In: Bickman L, Rob DJ, editors. The SAGE handbook of applied social research methods. Thousand Oaks, CA: Sage Publications, Inc; 2008. p. 44–76.Google Scholar
  46. 46.
    Sieber JE. Planning ethically responsible research. In: Bickman L, Rob DJ, editors. The SAGE handbook of applied social research methods. Thousand Oaks, CA: Sage Publications, Inc; 2008. p. 106–42.Google Scholar
  47. 47.
    Scriven M. Minimalist theory of evaluation: the least theory that practice requires. Am J Eval. 1998;19(1):57–70.CrossRefGoogle Scholar
  48. 48.
    Weick KE. Sensemaking in organizations. Thousand Oaks, CA: Sage Publications, Inc.; 1995.Google Scholar
  49. 49.
    Rogers PJ, Hough G. Improving the effectiveness of evaluations: making the link to organizational theory. Eval Program Plann. 1995;18(4):21–32.CrossRefGoogle Scholar
  50. 50.
    Barabba V. The decision loom: a design for interactive decision-making in organizations. Devon: Triarchy Press; 2011.Google Scholar
  51. 51.
    Mueller NB, Burke RC, Luke DA, Harris JK. Getting the word out: multiple methods for disseminating evaluation findings. J Public Health Manag Pract. 2008;14(2):170–6.CrossRefGoogle Scholar
  52. 52.
    Operations in evaluating community interventions; n.d. Available from http://ctb.ku.edu/en/table-of-contents/evaluate/evaluate-community-interventions. Accessed on 5 Jan 2015.
  53. 53.
    MacDonald G, Starr G, Schooley M, Yee SL, Klimowski K, Turner K. Introduction to program evaluation for comprehensive tobacco control programs. Atlanta, GA: Centers for Disease Control and Prevention; 2001.CrossRefGoogle Scholar
  54. 54.
    Stoto AM, Cosler EL. Evaluation of public health interventions. In: LIoy NF, Morrow CB, Mays GP, editors. Public health administration: principles for population-based management. 2nd ed. Sudbury, MA: Jones and Bartlett publishers; 2007. p. 495–526.Google Scholar
  55. 55.
    Creswell J, Plano Clark V. Designing and conducting mixed methods research. Thousand Oaks, CA: Sage; 2011.Google Scholar
  56. 56.
    U.S. Agency for International Development. Mixed-method evaluation; 2013. Available from https://usaidlearninglab.org/library/technical-note-mixed-methods-evaluations.
  57. 57.
    Trochim WM. The research methods knowledge base, 2nd edn. Internet WWW page. Available from http://www.socialresearchmethods.net/kb/. Accessed on 20 Oct 2006.
  58. 58.
    American Academy of Pediatrics. Evaluating your community-based program: putting your evaluation plan to work [PDF document]; 2008. Available from http://www2.aap.org/commpeds/htpcp/EvalGuide2.pdf.
  59. 59.
    Coombe MC. Participatory approaches to evaluating community organizing and coalition builiding. In: Minkler M, editor. Community organizing and community building for health and welfare. 3rd ed. New Brunswick, NJ: Rutgers University Press; 2012. p. 346–70.Google Scholar
  60. 60.
    Taylor-Powell E, Hermann C. Program development and evaluation, collecting evaluation data: surveys, May 2000. Available from http://learningstore.uwex.edu/assets/pdfs/G3658-10.pdf.
  61. 61.
    Loi CXA, Alfonso ML, Chan I, Anderson K, Tyson DDM, Gonzales J, et al. Application of mixed-methods design in community-engaged research: lessons learned from an evidence-based intervention for Latinos with chronic illness and minor depression. Eval Program Plann. 2017;63:29–38.CrossRefGoogle Scholar
  62. 62.
    Jagosh J, Bush PL, Salsber J, Macaulay AC, Greenhagh T, Wong G, et al. A realist evaluation of community-based participatory research; partnership synergy, trust building, and related ripple effect. BMC Public Health. 2015;15(725):1–11.Google Scholar
  63. 63.
    Olshansky E, Zender R. The use of community-based participatory research to understand and work with vulnerable populations. In: Chesnay M, Anderson BA, editors. Caring for the vulnerable: perspectives in nursing theory, practice, and research. 4th ed. Burlington, MA: Jones and Bartlett Learning; 2016. p. 243–52.Google Scholar
  64. 64.
    Community Health Scholars Program. Stories of impact. Ann Arbor, MI: W.K. Kellogg Foundation; 2002.Google Scholar
  65. 65.
    Olshansky E, Sacco D, Braxter B, Dodge P, Hughes E, Ondeck M, et al. Participatory action research to understand and reduce health disparities. Nurs Outlook. 2005;53(3):121–6.CrossRefGoogle Scholar
  66. 66.
    Sandoval JA, Lucero J, Oetzel J, Avila M, Belone L, Mau M, et al. Process and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health Educ Res. 2012;27(4):680–90.CrossRefGoogle Scholar
  67. 67.
    Wallerstein N, Oetzel JG, Duran B, Tafoya G, Belone L, Rae R. What predicts outcomes in CBPR? In: Minkler M, Wallerstein N, editors. Communication based participatory research. 2nd ed. San Francisco, CA: John Wiley & Co; 2008.Google Scholar
  68. 68.
    Tapp H, White L, Steuerwald M, Dulin M. Use of community-based participatory research in primary care to improve healthcare outcomes and disparities in care. J Comp Effect Res. 2013;2(4):405–19.CrossRefGoogle Scholar
  69. 69.
    Baumann A, Rodríguez MD, Parra-Cardona JR. Community-based applied research with Latino immigrant families: informing practice and research according to ethical and social justice principles. Fam Process. 2011;50(2):132–48.CrossRefGoogle Scholar
  70. 70.
    Diaz AEK, Johnson CRS, Arcury TA. Variation in the interpretation of scientific integrity in community-based participatory health research. Soc Sci Med. 2013;97:134–42.CrossRefGoogle Scholar
  71. 71.
    Chang CYT. Evaluation and adaptations of a community-based participatory research partnership in San Francisco’s Chinatown, Doctoral dissertation, University of California, Berkeley; 2010.Google Scholar
  72. 72.
    Rabinowitz P. Community-based participatory research; n.d. Available from http://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/intervention-research/main.
  73. 73.
    Lipton DS. How to maximize utilization of evaluation research by policymakers. Ann Am Acad Pol Soc Sci. 1992;521(1):175–88.CrossRefGoogle Scholar
  74. 74.
    U.S. Agency for International Development. Evaluation learning from experience: USAID evaluation policy. Washington, DC: USAID; 2011.Google Scholar
  75. 75.
    Hepler AN, Guida F, Messina M, Kanu M. Program evaluation with vulnerable populations in. In: Estrine SA, Robert T, Hettenbach AH, Maria M, editors. Service delivery for vulnerable population: new directions in behavioral health. New York, NY: Springer Publisher; 2011. p. 355–71.Google Scholar
  76. 76.
    Holm-Hansen C. Ethical issues-tips for conducting program evaluation issue 12, Fact sheet. Saint Paul, MN: Wilder Research; 2007.Google Scholar
  77. 77.
    El-Shahawy O, Shires DA, Elsoton LJ. Assessment of the efficiency of tobacco cessation counseling in primary care. Eval Health Prof. 2016;39(3):326–35.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Division of Nursing Science, School of NursingRutgers UniversityNewarkUSA

Personalised recommendations