Skip to main content
SpringerLink
Log in

Barriers to the evaluation of evidence-based public health policy

  • Viewpoint
  • Published:
Journal of Public Health Policy Aims and scope Submit manuscript

Abstract

Public health policy has the potential to produce great benefits for individuals and communities. There is growing demand that such efforts be rigorously evaluated to ensure that the expected benefits are, in fact, realised. Commonly, public health policy is evaluated by consumer acceptability, reach, or changes in knowledge and attitudes. Non-robust research designs are often used. But these approaches to evaluation do not answer three critical questions: Has a change in the desired outcome occurred? Was it a consequence of the policy and not some extraneous factor? Was the size of the change considered significant and cost-effective? We, a team of government and academic scholars working in research and evaluation, have examined some of the more common impediments to robust evaluation: political impediments, a lack of investment in evaluation capacity within bureaucracy, and the failure of academic researchers to understand the need for the evaluation of public health policy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Rychetnik L, et al. A glossary for evidence based public health. J Epidemiol Community Health. 2004;58(7):538–45.

    Article  Google Scholar 

  2. Centers for Disease Control and Prevention. Ten great public health achievements–United States, 1900-1999. MMWR Morb Mortal Wkly Rep. 1999;48(12):241.

    Google Scholar 

  3. Masters S, et al. Return on investment of public health interventions: a systematic review. J Epidemiol Commun Health. 2017;71:827–34.

    Article  Google Scholar 

  4. Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009;30:175–201.

    Article  Google Scholar 

  5. Oxman AD, et al. A framework for mandatory impact evaluation to ensure well informed public policy decisions. Lancet. 2010;375(9712):427–31.

    Article  Google Scholar 

  6. House of Commons Health Committee. Health Inequalities: Third Report of Session 2008-09 (Vol 1). London: HMSO; 2009.

    Google Scholar 

  7. Komro KA, et al. Research design issues for evaluating complex multicomponent interventions in neighborhoods and communities. Transl Behav Med. 2016;6(1):153–9.

    Article  Google Scholar 

  8. Wolfenden L, et al. What is generated and what is used: a description of public health research output and citation. Eur J Public Health. 2016;26(3):523–5.

    Article  Google Scholar 

  9. Hoomans T, Severens JL. Economic evaluation of implementation strategies in health care. Implement Sci. 2014;9(1):168.

    Article  Google Scholar 

  10. Milat AJ, et al. The concept of scalability: increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promot Int. 2013;28(3):285–98.

    Article  Google Scholar 

  11. Craig P, et al. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.

    Article  Google Scholar 

  12. Datta J, Petticrew M. Challenges to evaluating complex interventions: a content analysis of published papers. BMC Public Health. 2013;13(1):1.

    Article  Google Scholar 

  13. Schauer F. Transparency in three dimensions. Univ Ill Law Rev. 2011;2011(4):1339–57.

    Google Scholar 

  14. Aldridge R, et al. Lancet UK policy matters: better evidence for better health. Lancet. 2011;377(9778):4.

    Article  Google Scholar 

  15. Jansen MW, et al. Public health: disconnections between policy, practice and research. Health Res Policy Syst. 2010;8(1):37.

    Article  Google Scholar 

  16. Robson J, et al. The NHS Health Check in England: an evaluation of the first 4 years. BMJ Open. 2016;6(1):e008840.

    Article  Google Scholar 

  17. Sanson-Fisher RW, et al. Evaluation of systems-oriented public health interventions: alternative research designs. Annu Rev Public Health. 2014;35:9–27.

    Article  Google Scholar 

  18. EPOC. What study designs should be included in an EPOC review? EPOC resources for review authors. Cochrane Effective Practice and Organisation of Care. 2017; Available from: Available at http://epoc.cochrane.org/resources/epoc-resources-review-authors.

  19. Hopkins DP, et al. Smokefree policies to reduce tobacco use: a systematic review. Am J Prev Med. 2010;38(2):S275–89.

    Article  Google Scholar 

  20. Olstad D, et al. Can policy ameliorate socioeconomic inequities in obesity and obesity-related behaviours? A systematic review of the impact of universal policies on adults and children. Obes Rev. 2016;17(12):1198–217.

    Article  Google Scholar 

  21. Shanks CB, Banna J, Serrano EL. Food waste in the national school lunch program 1978–2015: a systematic review. J Acad Nutr Diet. 2017;117(11):1792–807.

    Article  Google Scholar 

  22. Lipsey MW, Cullen FT. The effectiveness of correctional rehabilitation: a review of systematic reviews. Annu Rev Law Soc Sci. 2007;3:297–320.

    Article  Google Scholar 

  23. Drake EK, Aos S, Miller MG. Evidence-based public policy options to reduce crime and criminal justice costs: implications in Washington State. Vict Offenders. 2009;4(2):170–96.

    Article  Google Scholar 

  24. Hawe P, Degeling D, Hall J. Evaluating health promotion: a health workers guide. Sydney: MacLennan & Petty; 1990.

    Google Scholar 

  25. Mercer SL, et al. Study designs for effectiveness and translation research: identifying trade-offs. Am J Prev Med. 2007;33(2):139–54.

    Article  Google Scholar 

  26. Burstein P, Linton A. The impact of political parties, interest groups, and social movement organizations on public policy: some recent evidence and theoretical concerns. Soc Forces. 2002;81(2):380–408.

    Article  Google Scholar 

  27. Brownson RC, et al. Researchers and policymakers: travelers in parallel universes. Am J Prev Med. 2006;30(2):164–72.

    Article  Google Scholar 

  28. Orton L, et al. The use of research evidence in public health decision making processes: systematic review. PLoS ONE. 2011;6(7):e21704.

    Article  Google Scholar 

  29. Bauman A, Nutbeam D. Planning and evaluating population interventions to reduce noncommunicable disease risk–reconciling complexity and scientific rigour. Public Health Res Pract. 2014;25(1):e2511402.

    Article  Google Scholar 

  30. Knai C, et al. Reported barriers to evaluation in chronic care: experiences in six European countries. Health Policy. 2013;110(2):220–8.

    Article  Google Scholar 

  31. Petticrew M, et al. In search of social equipoise. BMJ. 2013;346:40–1.

    Google Scholar 

  32. Milton S, Petticrew M, Green J. Why do local authorities undertake controlled evaluations of health impact? A qualitative case study of interventions in housing. Public Health. 2014;128(12):1112–7.

    Article  Google Scholar 

  33. Schneider CH, Milat AJ, Moore G. Barriers and facilitators to evaluation of health policies and programs: policymaker and researcher perspectives. Eval Program Plan. 2016;58:208–15.

    Article  Google Scholar 

  34. Jacob S, Speer S, Furubo J-E. The institutionalization of evaluation matters: updating the International Atlas of Evaluation 10 years later. Evaluation. 2015;21(1):6–31.

    Article  Google Scholar 

  35. Rosenstein, Parliamentarians Forum for Development Evaluation. National Evaluation Policies Global Mapping Report-2015. 2nd edition. http://www.pfde.net/index.php/news/60-parliamentarians-forum-published-mapping-national-evaluation-policies-2nd-edition. 2015.

  36. Bourgeois I, Cousins JB. Understanding dimensions of organizational evaulation capacity. Am J Eval. 2013;34(3):21.

    Article  Google Scholar 

  37. Nutbeam D. What’s in a word? Finding the value in evaluation. The Mandarin, 2017. http://www.themandarin.com.au/76565-whats-word-finding-value-evaluation/. Accessed 9 March 2017.

  38. Heaton J, Day J, Britten N. Inside the “black box” of a knowledge translation program in applied health research. Qual Health Res. 2015;25(11):1477–91.

    Article  Google Scholar 

  39. Wolfenden L, et al. Embedding researchers in health service organizations improves research translation and health service performance: the Australian Hunter New England Population Health example. J Clin Epidemiol. 2017;85:3–11.

    Article  Google Scholar 

  40. Kerr EA, Riba M, Udow-Phillips M. Helping health service researchers and policy makers speak the same language. Health Serv Res. 2015;50(1):1–11.

    Article  Google Scholar 

  41. Kuo T, Gase LN, Inkelas M. Dissemination, implementation, and improvement science research in population health: opportunities for public health and CTSAs. Clin Transl Sci. 2015;8(6):807–13.

    Article  Google Scholar 

  42. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):53.

    Article  Google Scholar 

  43. Eccles MP, et al. An implementation research agenda. Implement Sci. 2009;4(1):18.

    Article  Google Scholar 

  44. Remme JH, et al. Defining research to improve health systems. PLoS Med. 2010;7(11):e1001000.

    Article  Google Scholar 

  45. NSW Government, NSW Government Program Evaluation Guidelines. Department of Premier and Cabinet, Sydney: Available online at: http://arp.nsw.gov.au/c2016-01-program-evaluation, 2006.

  46. Norton S, et al. Narrative review of strategies by organizations for building evaluation capacity. Eval Progr Plan. 2016;58:1–19.

    Article  Google Scholar 

  47. Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health. 2009;99(9):1576–83.

    Article  Google Scholar 

  48. Tabak RG, et al. Dissemination and implementation science training needs: insights from practitioners and researchers. Am J Prev Med. 2017;52(3):S322–9.

    Article  Google Scholar 

  49. Jensen JD, Smed S. The Danish tax on saturated fat–short run effects on consumption, substitution patterns and consumer prices of fats. Food Policy. 2013;42:18–31.

    Article  Google Scholar 

  50. Snowdon C. The proof of the pudding: Denmark’s fat tax fiasco, IEA Current Controversies Paper No. 42, Institute of Economic Affairs. 2013.

  51. Bødker M, et al. The rise and fall of the world’s first fat tax. Health Policy. 2015;119(6):737–42.

    Article  Google Scholar 

  52. Bødker M, et al. The Danish fat tax—effects on consumption patterns and risk of ischaemic heart disease. Prev Med. 2015;77:200–3.

    Article  Google Scholar 

  53. Smed S, et al. The effects of the Danish saturated fat tax on food and nutrient intake and modelled health outcomes: an econometric and comparative risk assessment evaluation. Eur J Clin Nutr. 2016;70(6):681.

    Article  Google Scholar 

  54. Thow AM, et al. The effect of fiscal policy on diet, obesity and chronic disease: a systematic review. Bull World Health Organ. 2010;88:609–14.

    Article  Google Scholar 

  55. Brownson RC, et al. Building capacity for dissemination and implementation research: one university’s experience. Implement Sci. 2017;12(1):104.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported funding from The Australian Prevention Partnership Centre and infrastructure funding from the Hunter Medical Research Institute. Megan Freund is supported by a National Health and Medical Research Institute Translating Research Into Practice fellowship. Dr Lisa Mackenzie is supported by a Postdoctoral Fellowship grant [PF-16-011] from the Australian National Breast Cancer Foundation.

Author information

Authors and Affiliations

  1. Health Behaviour Research Collaborative, School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Callaghan, NSW, 2308, Australia

    Megan Freund, Alison Zucca, Robert Sanson-Fisher, Lisa Mackenzie & Heidi Turon

  2. Priority Research Centre for Health Behaviour, Faculty of Health and Medicine, The University of Newcastle, Callaghan, NSW, 2308, Australia

    Megan Freund, Alison Zucca, Robert Sanson-Fisher, Lisa Mackenzie & Heidi Turon

  3. Hunter Medical Research Institute, New Lambton, NSW, Australia

    Megan Freund, Alison Zucca, Robert Sanson-Fisher, Lisa Mackenzie & Heidi Turon

  4. Sydney Medical School, University of Sydney, City Road, Camperdown, NSW, 2006, Australia

    Andrew Milat

Authors
  1. Megan Freund
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alison Zucca
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Robert Sanson-Fisher
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andrew Milat
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lisa Mackenzie
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Heidi Turon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Megan Freund.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Freund, M., Zucca, A., Sanson-Fisher, R. et al. Barriers to the evaluation of evidence-based public health policy. J Public Health Pol 40, 114–125 (2019). https://doi.org/10.1057/s41271-018-0145-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/s41271-018-0145-9

Keywords

Search

Navigation