Advertisement

The Funding, Administrative, and Policy Influences on the Evaluation of Primary Prevention Programs in Australia

  • J. SchwarzmanEmail author
  • A. Bauman
  • B. J. Gabbe
  • C. Rissel
  • T. Shilton
  • B. J. Smith
Article
  • 3 Downloads

Abstract

Evaluation of primary prevention and health promotion programs contributes necessary information to the evidence base for prevention programs. There is increasing demand for high-quality evaluation of program impact and effectiveness for use in public health decision making. Despite the demand for evidence and known benefits, evaluation of prevention programs can be challenging and organizations face barriers to conducting rigorous evaluation. Evaluation capacity building efforts are gaining attention in the prevention field; however, there is limited knowledge about how components of the health promotion and primary prevention system (e.g., funding, administrative arrangements, and the policy environment) may facilitate or hinder this work. We sought to identify the important influences on evaluation practice within the Australian primary prevention and health promotion system. We conducted in-depth semi-structured interviews with experienced practitioners and managers (n = 40) from government and non-government organizations, and used thematic analysis to identify the main factors that impact on prevention program evaluation. Firstly, accountability and reporting requirements impacted on evaluation, especially if expectations were poorly aligned between the funding body and prevention organization. Secondly, the funding and political context was found to directly and indirectly affect the resources available and evaluation approach. Finally, it was found that participants made use of various strategies to modify the prevention system for more favorable conditions for evaluation. We highlight the opportunities to address barriers to evaluation in the prevention system, and argue that through targeted investment, there is potential for widespread gain through improved evaluation capacity.

Keywords

Evaluation capacity Primary prevention Health promotion Evidence-based public health 

Notes

Acknowledgements

We would like to thank Emily McCluskey for her valuable contribution to participant recruitment and data analysis in this study.

Funding

This work was supported by the Australian Research Council (ARC) [DP150103575]. Prof. Gabbe was supported by ARC Future Fellowship [FT170100048] during the preparation of this manuscript.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Supplementary material

11121_2019_997_MOESM1_ESM.docx (15 kb)
ESM 1 (DOCX 15 kb)

References

  1. Baugh Littlejohns, L., Baum, F., Lawless, A., & Freeman, T. (2018). Disappearing health system building blocks in the health promotion policy context in South Australia (2003–2013). Critical Public Health, 1–13.  https://doi.org/10.1080/09581596.2017.1418501.
  2. Bourgeois, I., & Cousins, J. B. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34, 299–319.  https://doi.org/10.1177/1098214013477235.CrossRefGoogle Scholar
  3. Bowen, S., & Zwi, A. B. (2005). Pathways to “evidence-informed” policy and practice: a framework for action. PLoS Medicine, 2, e166.  https://doi.org/10.1371/journal.pmed.0020166.CrossRefGoogle Scholar
  4. Brehaut, J. C., & Eva, K. W. (2012). Building theories of knowledge translation interventions: Use the entire menu of constructs. Implementation Science, 7, 114.  https://doi.org/10.1186/1748-5908-7-114.CrossRefGoogle Scholar
  5. Brug, J., Tak, N. I., & Te Velde, S. J. (2011). Evaluation of nationwide health promotion campaigns in the Netherlands: an exploration of practices, wishes and opportunities. Health Promotion International, 26, 244–254.  https://doi.org/10.1093/heapro/daq058.CrossRefGoogle Scholar
  6. Carman, J. G. (2011). Understanding evaluation in nonprofit organizations. Public Performance & Management Review, 34, 350–377.  https://doi.org/10.2753/PMR1530-9576340302.CrossRefGoogle Scholar
  7. Carman, J. G., & Fredericks, K. A. (2009). Evaluation capacity and nonprofit organizations: Is the glass half-empty or half-full? American Journal of Evaluation, 31, 84–104.  https://doi.org/10.1177/1098214009352361.CrossRefGoogle Scholar
  8. Dahler-Larsen. (2009). Evaluation and public management. In E. Ferlie, L. E. Lynn, & C. Pollitt (Eds.), The Oxford handbook of public management. Oxford: Oxford University Press.Google Scholar
  9. Datta, J., & Petticrew, M. (2013). Challenges to evaluating complex interventions: a content analysis of published papers. BMC Public Health, 13, 568.  https://doi.org/10.1186/1471-2458-13-568.CrossRefGoogle Scholar
  10. Edwards, B., Stickney, B., Milat, A., Campbell, D., & Thackway, S. (2016). Building research and evaluation capacity in population health: the NSW health approach. Health Promotion Journal of Australia, 27, 264–267.  https://doi.org/10.1071/HE16045.CrossRefGoogle Scholar
  11. Francis, L. J., & Smith, B. J. (2015). Toward best practice in evaluation: a study of Australian health promotion agencies. Health Promotion Practice, 16, 715–723.  https://doi.org/10.1177/1524839915572574.CrossRefGoogle Scholar
  12. Huckel Schneider, C., Milat, A. J., & Moore, G. (2016). Barriers and facilitators to evaluation of health policies and programs: Policymaker and researcher perspectives. Evaluation and Program Planning, 58, 208–215.  https://doi.org/10.1016/j.evalprogplan.2016.06.011.CrossRefGoogle Scholar
  13. Jackson, H., & Shiell, A. (2017). Preventive health: How much does Australia spend and is it enough? Canberra: Foundation for Alcohol Research and Education.Google Scholar
  14. Jolley, G. M., Lawless, A. P., Baum, F. E., Hurley, C. J., & Fry, D. (2007). Building an evidence base for community health: a review of the quality of program evaluations. Australian Health Review, 31, 603–610.CrossRefGoogle Scholar
  15. Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338.  https://doi.org/10.1177/1098214011434608.CrossRefGoogle Scholar
  16. Liket, K. C., Rey-Garcia, M., & Maas, K. E. H. (2014). Why aren’t evaluations working and what to do about it. American Journal of Evaluation, 35, 171–188.  https://doi.org/10.1177/1098214013517736.CrossRefGoogle Scholar
  17. Lobo, R., McManus, A., Brown, G., Hildebrand, J., & Maycock, B. (2010). Evaluating peer-based youth programs: Barriers and enablers. Evaluation Journal of Australasia, 10, 36–43.CrossRefGoogle Scholar
  18. Lobo, R., Petrich, M., & Burns, S. K. (2014). Supporting health promotion practitioners to undertake evaluation for program development. BMC Public Health, 14.  https://doi.org/10.1186/1471-2458-14-1315.
  19. Lobo, R., Crawford, G., Hallett, J., Laing, S., Mak, D. B., Jancey, J., …, Doherty, M. (2016). A research and evaluation capacity building model in Western Australia. Health Promotion International  https://doi.org/10.1093/heapro/daw088.
  20. Milat, A., & Li, B. (2017). Narrative review of frameworks for translating research evidence into policy and practice. Public Health Research & Practice, 27, e2711704.  https://doi.org/10.17061/phrp2711704.CrossRefGoogle Scholar
  21. Milat, A. J., King, L., Bauman, A. E., & Redman, S. (2013). The concept of scalability: Increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promotion International, 28, 285–298.CrossRefGoogle Scholar
  22. Napp, D., Gibbs, D., Jolly, D., Westover, B., & Uhl, G. (2002). Evaluation barriers and facilitators among community-based HIV prevention programs. AIDS Education and Prevention, 14, 38–48.CrossRefGoogle Scholar
  23. Nebot, M. (2006). Health promotion evaluation and the principle of prevention. Journal of Epidemiology and Community Health, 60, 5–6.Google Scholar
  24. Nielsen, S. B., Lemire, S., & Skov, M. (2011). Measuring evaluation capacity—Results and implications of a Danish study. American Journal of Evaluation, 32, 324–344.  https://doi.org/10.1177/1098214010396075.CrossRefGoogle Scholar
  25. Norton, S., Milat, A., Edwards, B., & Giffin, M. (2016). Narrative review of strategies by organizations for building evaluation capacity. Evaluation and Program Planning, 58, 1–19.  https://doi.org/10.1016/j.evalprogplan.2016.04.004.CrossRefGoogle Scholar
  26. Nutbeam, D. (1998). Promoting the health of Australians-how strong is our infrastructure support? Australian and New Zealand Journal of Public Health, 22, 301–302.CrossRefGoogle Scholar
  27. Oxman, A. D., Bjørndal, A., Becerra-Posada, F., Gibson, M., Block, M. A. G., Haines, A., … Wibulpolprasert, S. (2010). A framework for mandatory impact evaluation to ensure well informed public policy decisions. The Lancet, 375, 427–431.  https://doi.org/10.1016/S0140-6736(09)61251-4.
  28. Petticrew, M. (2013). Public health evaluation: Epistemological challenges to evidence production and use. Evidence & Policy: A Journal of Research, Debate and Practice, 9, 87–95.  https://doi.org/10.1332/174426413X663742.CrossRefGoogle Scholar
  29. Pettman, T. L., Armstrong, R., Doyle, J., Burford, B., Anderson, L. M., Hillgrove, T., … Waters, E. (2012). Strengthening evaluation to capture the breadth of public health practice: Ideal vs. real. Journal of Public Health (Oxford, England), 34, 151–155.  https://doi.org/10.1093/pubmed/fds014.
  30. Rychetnik, L., Bauman, A., Laws, R., King, L., Rissel, C., Nutbeam, D., … Caterson, I. (2012). Translating research for evidence-based public health: Key concepts and future directions. Journal of Epidemiology and Community Health, 66, 1187–1192.  https://doi.org/10.1136/jech-2011-200038.
  31. Schwarzman, J., Bauman, A., Gabbe, B., Rissel, C., Shilton, T., & Smith, B. J. (2018). Organizational determinants of evaluation practice in Australian prevention agencies. Health Education Research, Published online ahead of print, 1–13.  https://doi.org/10.1093/her/cyy015.
  32. Smith, B. J., Rissel, C., Shilton, T., & Bauman, A. (2016). Advancing evaluation practice in health promotion. Health Promotion Journal of Australia, 27, 184–186.  https://doi.org/10.1071/HEv27n3_ED2.CrossRefGoogle Scholar
  33. South, J., & Tilford, S. (2000). Perceptions of research and evaluation in health promotion practice and influences on activity. Health Education Research, 15, 729–741.  https://doi.org/10.1093/her/15.6.729.CrossRefGoogle Scholar
  34. Spoth, R., Rohrbach, L., Greenberg, M., Leaf, P., Brown, C., Fagan, A., … Hawkins, J. (2013). Addressing Core challenges for the next generation of type 2 translation research and systems: the translation science to population impact (TSci impact) framework. Prevention Science, 14, 319–351.  https://doi.org/10.1007/s11121-012-0362-6.
  35. van Koperen, T. M., Renders, C. M., Spierings, E. J. M., Hendriks, A.-M., Westerman, M. J., Seidell, J. C., & Schuit, A. J. (2016). Recommendations and improvements for the evaluation of integrated community-wide interventions approaches. Journal of Obesity, 2016, 2385698–2385698.  https://doi.org/10.1155/2016/2385698.Google Scholar
  36. Wutzke, S., Morrice, E., Benton, M., & Wilson, A. (2016). What will it take to improve prevention of chronic diseases in Australia? A case study of two national approaches. Australian Health Review, 41, 176–181.  https://doi.org/10.1071/AH16002.CrossRefGoogle Scholar
  37. Wutzke, S., Morrice, E., Benton, M., Milat, A., Russell, L., & Wilson, A. (2017). Australia's National Partnership Agreement on preventive health: Critical reflections from states and territories. Health Promotion Journal of Australia.  https://doi.org/10.1002/hpja.9.
  38. Zardo, P., Collie, A., & Livingstone, C. (2014). External factors affecting decision-making and use of evidence in an Australian public health policy environment. Social Science and Medicine, 108, 120–127.  https://doi.org/10.1016/j.socscimed.2014.02.046.CrossRefGoogle Scholar

Copyright information

© Society for Prevention Research 2019

Authors and Affiliations

  1. 1.School of Public Health and Preventive MedicineMonash UniversityMelbourneAustralia
  2. 2.Prevention Research Collaboration, School of Public HealthThe University of SydneySydneyAustralia
  3. 3.Farr InstituteSwansea University Medical School, Swansea University, Singleton ParkSwanseaUK
  4. 4.National Heart Foundation Western AustraliaSubiacoAustralia

Personalised recommendations