Skip to main content
SpringerLink
Log in

Measurement Properties of Occupational Health and Safety Management Audits: A Systematic Literature Search and Traditional Literature Synthesis

  • Systematic Review
  • Published:
Canadian Journal of Public Health Aims and scope Submit manuscript

Abstract

Objective

The measurement properties of Occupational Health and Safety (OHS) management audits might be important in some applications, especially when audit scores are treated as performance measures. The review, therefore, sought to identify and summarize the research evidence on the measurement properties (e.g., reliability, validity) of methods of OHS management audit.

Methods

Bibliographic databases in business, medicine and OHS were systematically searched. Evidence from relevant publications was synthesized using traditional narrative review methods.

Synthesis

The literature on the measurement properties of OHS management audit methods is sparse. Seventeen relevant audit methods were identified. Content validity was demonstrated for only five audit methods. Inter-rater reliability was formally tested for only three audit methods and construct validity for only one. There were no studies of test-retest reliability or responsiveness. The investigations of inter-rater reliability (i.e., consistency among auditors) showed that it is often unacceptably low.

Conclusion

There is a research gap concerning the measurement properties of OHS management audit methods. The available research raises questions about the properties of audit methods in current use.

Résumé

Objectif

Les propriétés de mesure des audits de gestion en santé et sécurité du travail (SST) pourraient être importantes pour certaines applications, surtout lorsque les scores d’audit sont considérées comme des mesures de rendement. Nous avons donc voulu répertorier et résumer les résultats de recherche sur les propriétés de mesure (p. ex., fiabilité, validité) des méthodes utilisées pour les audits de gestion en SST.

Méthode

Les bases de données bibliographiques des domaines des affaires, de la médecine et de la SST ont été systématiquement interrogées. Les données probantes de publications pertinentes ont été résumées à l’aide de méthodes classiques de recension narrative des écrits.

Synthèse

Les travaux publiés sur les propriétés de mesure des méthodes utilisées pour les audits de gestion en SST sont rares. Dix-sept méthodes d’audit pertinentes ont été recensées. Nous n’avons pu démontrer la validité de contenu que pour cinq de ces méthodes. La fiabilité inter-évaluateurs n’a été véritablement testée que pour trois méthodes d’audit, et la validité de construit, pour une seule méthode. Il n’y avait aucune étude de fiabilité de test-retest, ni de sensibilité. Les études de fiabilité inter-évaluateurs (cohérence d’un évaluateur à l’autre) ont montré que cette fiabilité est souvent trop faible pour être acceptable.

Conclusion

Il y a des lacunes dans la recherche sur les propriétés de mesure des méthodes de gestion en SST. Les travaux publiés soulèvent des questions quant aux propriétés des méthodes d’audit utilisées actuellement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Canadian Standards Association. CSA Z1000-06. Occupational Health and Safety Management. Mississauga, ON: CSA, 2006.

    Google Scholar 

  2. International Labour Organization. Meeting of Experts on ILO Guidelines on Occupational Safety and Health Management Systems: Final Report. Geneva: International Labour Organization, 2001.

    Google Scholar 

  3. OHSAS 18001:2007. Occupational Health and Safety Management Systems–Requirements. London: OHSAS Project Group, 2007.

    Google Scholar 

  4. Karapetrovic S, Willborn W. Generic audit of management systems: Fundamentals. Managerial Auditing J 2000;15(6):279–94.

    Article  Google Scholar 

  5. Waring A. Safety Management Systems. London: Chapman & Hall, 1996.

    Google Scholar 

  6. Nash JL. Who is auditing your safety auditors? Occup Haz 2005;67(7):31–34.

    Google Scholar 

  7. Grant J, Brown D. The inspector cometh. Cdn HR Reporter 2005;18(2):13, 17.

    Google Scholar 

  8. Robson LS, Macdonald S, Van Eerd D, Gray G, Bigelow P. Prevention System OHS Management Audit Methods: Description, Content Validation and an Assessment of the Feasibility of Measurement Research. Final Report to WSIB RAC on Project #06112. Toronto, ON: Institute for Work & Health, 2008.

    Google Scholar 

  9. Workplace Safety & Insurance Board of Ontario. The Safety Groups Program. Available at: https://doi.org/www.wsib.on.ca/wsib/wsibsite.nsf/public/SafetyGroups Program (Accessed February 13, 2009).

    Google Scholar 

  10. Government of Alberta. What is a Certificate of Recognition? Available at: https://doi.org/employment.alberta.ca/cps/rde/xchg/hre/hs.xsl/334.html (Accessed February 13, 2009).

    Google Scholar 

  11. WorkSafe BC (the Workers’ Compensation Board of BC). Partners Program (COR). Available at: https://doi.org/www.worksafebc.com/insurance/partners_ program/default.asp (Accessed February 13, 2009).

    Google Scholar 

  12. Workplace Health, Safety & Compensation Commission of Newfoundland and Labrador. PRIME. Frequently asked questions. Available at: https://doi.org/www.whscc.nl.ca/prime/PR_FrequentlyAskedQuestions.whscc (Accessed February 13, 2009).

    Google Scholar 

  13. EUSA. Zeroquest. Introduction. Available at: www.63.135.106.115/Home.aspx?PageID=13&mID=_ctl0_MainMenu2__ctl1-menuItem000 (Accessed February 13, 2009).

    Google Scholar 

  14. NEPCon. OHSAS 18001 certification. Available at: https://doi.org/www.nepcon.net/index.php?ID=381 (Accessed February 13, 2009).

    Google Scholar 

  15. Workplace Safety & Insurance Board of Ontario. Workwell. Available at: https://doi.org/www.wsib.on.ca/wsib/wsibsite.nsf/public/Workwell (Accessed February 13, 2009).

    Google Scholar 

  16. Robson LS, Clarke JA, Cullen K, Bielecky A, Severin C, Bigelow PL, et al. The effectiveness of occupational health and safety management system interventions: A systematic review. Safety Sci 2007;45(2):329–53.

    Article  Google Scholar 

  17. Bigelow PL, Greenstein SL, Keefe TJ, Gilkey DP. Development of an on-site, behavior-based safety audit for the residential construction industry. Work 1998;11:11–20.

    Article  CAS  PubMed  Google Scholar 

  18. Laitinen H, Marjamaki M, Paivarinta K. The validity of the TR safety observation method on building construction. Accid Anal Prev 1999;31:463–72.

    Article  CAS  PubMed  Google Scholar 

  19. Straker L, Burgess-Limerick R, Pollock C, Egeskov R. A randomized and controlled trial of a participative ergonomics intervention to reduce injuries associated with manual tasks: Physical risk and legislative compliance. Ergonomics 2004;47(2):166–88.

    Article  CAS  PubMed  Google Scholar 

  20. Guldenmund F, Hale A, Goosens L, Betten J, Duijm NJ. The development of an audit technique to assess the quality of safety barrier management. J Hazard Mater 2006;130:234–41.

    Article  CAS  PubMed  Google Scholar 

  21. Hale AR, Heming BHJ, Smit K, Rodenburg FGTh, Van Leeuwen ND. Evaluating safety in the management of maintenance activities in the chemical process industry. Safety Sci 1998;28(1):21–44.

    Article  Google Scholar 

  22. Hee DD, Pickrell BD, Bea RG, Roberts KH, Williamson RB. Safety management assessment system (SMAS): A process for identifying and evaluating human and organization factors in marine system operations with field test results. Reliab Eng Syst Safe 1999;65:125–40.

    Article  Google Scholar 

  23. Hurst NW, Hankin R, Bellamy LJ, Wright MJJ. Auditing–a European perspective. J Loss Prevent Proc 1994;7(2):197–200.

    Article  Google Scholar 

  24. Papazoglou IA, Bellamy LJ, Hale AR, Aneziris ON, Ale BJM, Post JG, et al. I-risk: Development of an integrated technical and management risk methodology for chemical installations. J Loss Prevent Proc 2003;16(6):575–91.

    Article  Google Scholar 

  25. Pitblado R, Williams JC, Slater DH. Quantitative assessment of process safety programs. Plant/Operations Prog 1990;9(3):169–75.

    Article  CAS  Google Scholar 

  26. Bigelow PL, Robson LS. Occupational Health and Safety Management Audit Instruments: A Literature Review. Toronto, ON: Institute for Work & Health, 2005.

    Google Scholar 

  27. Diekemper RF, Spartz DA. A quantitative and qualitative measurement of industrial safety activities. ASSE J 1970;Dec:12–19.

    Google Scholar 

  28. Kuusisto A. Safety management systems: Audit tools and reliability of auditing [dissertation]. Tampere, Finland: Tampere University of Technology, 2000.

    Google Scholar 

  29. Uusitalo T, Mattila M. Evaluation of industrial safety practices in five industries. In: Mital A (Ed.), Advances in Industrial Ergonomics and Safety, vol.1. London: Taylor & Francis, 1989.

    Google Scholar 

  30. Eisner HS, Leger JP. The international safety rating system in South African mining. J Occup Accid 1988;10:141–60.

    Article  Google Scholar 

  31. Eisner HS. Safety rating systems in South African mines. J Health Safety 1993;9:25–30.

    Google Scholar 

  32. Guastello SJ. Some further evaluations of the International Safety Rating System. Safety Sci 1991;14:253–59.

    Article  Google Scholar 

  33. Collison JE, Booth RT. An evaluation of two proprietary health and safety auditing systems. J Health Safety 1993;9:31–38.

    Google Scholar 

  34. Gaunt LD. The Effect of the International Safety Rating System (I.S.R.S.) on Organizational Performance. Georgia: Center for Risk Management and Insurance Research, Georgia State University, 1989.

    Google Scholar 

  35. Chaplin R. An evaluation of the use of the International Safety Rating System (ISRS) as intervention to improve the organisation of safety. In: Hale A, Baram, M (Eds.), Safety Management: The Challenge of Change. Oxford: Pergamon, 1998.

    Google Scholar 

  36. Glendon AI, Boyle AJ, Hewitt DM. Computerized health and safety audit systems. In: Matilla M, Karwowski W (Eds.), Computer Applications in Ergonomics, Occupational Safety and Health. Amsterdam: Elsevier, 1992.

    Google Scholar 

  37. Dyjack DT. Development and evaluation of an ISO 9000-harmonized occupational health and safety management system [dissertation]. Ann Arbor, MI: University of Michigan, 1996.

    Google Scholar 

  38. Dyjack DT, Levine SP, Holtshouser JL, Schork MA. Comparison of AIHA ISO 9001-based occupational health and safety management system guidance document with a manufacturer’s occupational health and safety assessment instrument. Am Ind Hyg Assoc J 1998;59(6):419–29.

    Article  CAS  PubMed  Google Scholar 

  39. Alteren B, Hovden J. The Safety Element Method–a user developed tool for improvement of safety management. Safety Sci Monitor 1997;1(3):1–23. Available at: https://doi.org/www.monash.edu.au/muarc/ipso/issues.html (Accessed February 24, 2009).

    Google Scholar 

  40. Alteren B. Implementation and evaluation of the Safety Element Method at four mining sites. Safety Sci 1999;31:231–64.

    Article  Google Scholar 

  41. Dyjack DT, Redinger CF, Ridge RS. Health and safety management system audit reliability pilot project. Am Ind Hyg Assoc J 2003;64(6):785–91.

    Article  CAS  Google Scholar 

  42. Redinger CF. Occupational health and safety management system conformity assessment: Development and evaluation of a universal assessment instrument [dissertation]. Ann Arbor, MI: University of Michigan, 1998.

    Google Scholar 

  43. Redinger CF, Levine SP. Development and evaluation of the Michigan occupational health and safety management system assessment instrument: A universal OHSMS performance measurement tool. Am Ind Hyg Assoc J 1998;59:572–81.

    Article  Google Scholar 

  44. Redinger CF, Levine SP, Blotzer MJ, Majewski MP. Evaluation of an occupational health and safety management system performance measurement tool–II: Scoring methods and field study sites. Am Ind Hyg Assoc J 2002;63(1):34–40.

    Article  CAS  Google Scholar 

  45. Redinger CF, Levine SP, Blotzer MJ, Majewski MP. Evaluation of an occupational health and safety management system performance measurement tool–III: Measurement of initiation elements. Am Ind Hyg Assoc J 2002;63(1):41–46.

    Article  CAS  Google Scholar 

  46. Kam YK, Wong YY. Introduction to SMA-CON: Safety audit for construction. In: Zeng Q, Xie X, Wang L, Qian X (Eds.), Progress in Safety Science & Technology, v 1, Proceedings of the International Symposium on Safety Science and Technology. Beijing: Science Press, 1998.

    Google Scholar 

  47. Schweigert MK, House RA, Holness DL. Occupational health and safety management systems in the Canadian pulp and paper industry: Methods of auditing. J Occup Environ Med 1999;41(10):857–62.

    Article  CAS  PubMed  Google Scholar 

  48. Fuller C. Benchmarking health and safety performance through company safety competitions. Benchmarking: an Intl J 1999;6(4):325–37.

    Article  Google Scholar 

  49. Booth RT, Lee TR. The role of human factors and safety culture in safety management. Proc Instn Mech Engrs 1995;209(B5):393–400.

    Article  Google Scholar 

  50. Pearse W. Club Zero: Implementing OHS management systems in small to medium fabricated metal product companies. In: Pearse W, Gallagher C, Bluff L (Eds.), Occupational Health & Safety Management Systems. Proceedings of the First National Conference. Melbourne: Crown Content, 2001.

    Google Scholar 

  51. Pearse W. Club Zero: Implementing OHSMS in small to medium fabricated metal product companies. J Occup Health Safety - Aust NZ 2002;18(4):347–56.

    Google Scholar 

  52. Bunn WB, Pikelny DB, Slavin TJ, Parlkar S. Health, safety, and productivity in a manufacturing environment. J Occup Environ Med 2001;43(1):47–55.

    Article  PubMed  Google Scholar 

  53. LaMontagne AD, Barbeau E, Youngstrom RA, Lewiton M, Stoddard AM, McLellan D, et al. Assessing and intervening on OSH programmes: Effectiveness evaluation of the Wellworks-2 interventions in 15 manufacturing worksites. J Occup Environ Med 2004;61(8):651–60.

    Article  CAS  Google Scholar 

  54. Teo EAL, Ling FYY. Developing a model to measure the effectiveness of safety management systems of construction sites. Build Environ 2006;41:1584–92.

    Article  Google Scholar 

  55. Nielsen KJ, Rasmussen K, Glasscock D, Spangenberg S. Changes in safety climate and accidents at two identical manufacturing plants. Safety Sci 2008;46(3):440–49.

    Article  Google Scholar 

  56. Terwee CB, Bot SD, de Boer MR, Van Der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007;60(1):34–42.

    Article  Google Scholar 

  57. Scientific Advisory Committee of the Medical Outcomes Trust. Assessing health status and quality-of-life instruments. Qual Life Res 2002;11:193–205.

    Article  Google Scholar 

  58. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74.

    Article  CAS  Google Scholar 

  59. Gouttebarge V, Haije Wind H, Kuijer PPFM, Frings-Dresen MHW. Reliability and validity of functional capacity evaluation methods: A systematic review with reference to Blankenship system, Ergos work simulator, Ergo-Kit and Isernhagen work system. Int Arch Occup Environ Health 2004;77:527–37.

    Article  PubMed  Google Scholar 

  60. Matheson A, Mughal W, Thomas-Olson L, Spiwak R, Wasdell M. Inter-rater reliability assessment of an OH&S management systems audit tool. Presented at National Occupational Injury Research Symposium, Pittsburgh, PA, October 21–23, 2008.

    Google Scholar 

  61. Gillette D, Campbell P, Busby B. The evolution of a radiation safety audit program for a research institution. Health Phys 2004;86(2):S80–S84.

    Article  CAS  PubMed  Google Scholar 

  62. Birkmire JC, Lay JR, McMahon MC. Keys to effective third-party process safety audits. J Hazard Mater 2007;142:574–81.

    Article  CAS  PubMed  Google Scholar 

  63. Budworth N, Cox S. Trusting the tools. Safety Health Pract 2005;23(7):46–48.

    Google Scholar 

  64. Cooper D. Safety management system auditing. In: Cooper D (Ed.), Improving Safety Culture–A Practical Guide. Chichester, UK: John Wiley & Sons Ltd., 1998.

    Google Scholar 

  65. Henriksson L. Looking for gold. OHS Canada 1998;12:48–51.

    Google Scholar 

  66. International Organization for Standardization (ISO). ISO 19011: Guidelines for quality and/or environmental management systems auditing. Geneva (Switzerland): ISO, 2002.

    Google Scholar 

  67. Karapetrovic S, Willborn W. Quality assurance and effectiveness of audit systems. Int J Qual Reliability Manage 2000;17(6):679–703.

    Article  Google Scholar 

  68. Grant J, Bricker R, Shiptsova R. Audit quality and professional self-regulation: A social dilemma perspective and laboratory investigation. Auditing: J Pract Theor 1996;15(1):142–56.

    Google Scholar 

  69. Messier WF, Jr, Kachelmeier SJ, Jensen KL. An experimental assessment of recent professional developments in nonstatistical audit sampling guidance. Auditing: J Pract Theor 2001;20(1):81–96.

    Article  Google Scholar 

  70. Meyer MJ, Rigsby JT, Boone J. The impact of auditor-client relationships on the reversal of first-time audit qualifications. Managerial Accounting J 2007;22(1):53–79.

    Article  Google Scholar 

  71. Mohd-Sanusi Z, Mohd-Iskandar T. Audit judgment performance: Assessing the effect of performance incentives, effort and task complexity. Managing Auditing J 2007;22(1):34–52.

    Article  Google Scholar 

  72. Nelson M, Tan H-T. Judgment and decision making research in auditing: A task, person, and interpersonal interaction perspective. Auditing: J Pract Theor 2005;24(Suppl):41–71.

    Article  Google Scholar 

  73. O’Leary C. The consistency of individual auditors in performing evaluations. Managing Auditing J 2004;19(5):597–605.

    Article  Google Scholar 

  74. Pflugrath G, Martinov-Bennie N, Chen L. The impact of codes of ethics and experience on auditor judgments. Managerial Auditing J 2007;22(6):566–89.

    Article  Google Scholar 

  75. Preuss L. On ethical theory in auditing. Managerial Auditing J 1998;13(9):500–8.

    Article  Google Scholar 

  76. Richard C. Why an auditor can’t be competent and independent: A French case study. Eur Account Rev 2006;15(2):153–79.

    Article  Google Scholar 

  77. Sweeney B, Pierce B. Management control in audit firms: A qualitative examination. Account Audit Accountability J 2004;17(5):779–812.

    Article  Google Scholar 

  78. Beaton DE, Hogg-Johnson S, Bombardier C. Evaluating changes in health status: Reliability and responsiveness of five generic health status measures in workers with musculoskeletal disorders. J Clin Epidemiol 1997;50(1):79–93.

    Article  CAS  PubMed  Google Scholar 

  79. Bot SD, Terwee CB, Van Der Windt DA, Bouter LM, Dekker J, de Vet HC. Clini-metric evaluation of shoulder disability questionnaires: A systematic review of the literature. Ann Rheum Dis 2004;4:335–41.

    Article  Google Scholar 

  80. Williams RM, Schmuck G, Allwood S, Sanchez M, Shea R, Wark G. Psychometric evaluation of health-related work outcome measures for muscu-loskeletal disorders: A systematic review. J Occup Rehab 2007;17(3):504–21.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Work & Health, 481 University Ave., Suite 800, Toronto, ON, M5G 2E9, Canada

    Lynda S. Robson PhD & Philip L. Bigelow PhD

  2. University of Waterloo, Waterloo, ON, Canada

    Philip L. Bigelow PhD

Authors
  1. Lynda S. Robson PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Philip L. Bigelow PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lynda S. Robson PhD.

Additional information

Acknowledgement: The Institute for Work & Health receives core funding from the Workplace Safety & Insurance Board of Ontario (WSIB). The review was also supported in part by the WSIB Prevention Reviews initiative. The authors appreciate the key support of the library staff, particularly Emma Irvin and Rachel Couban, and the administrative assistance of Diana Pugliese.

Conflict of Interest: None to declare.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Robson, L.S., Bigelow, P.L. Measurement Properties of Occupational Health and Safety Management Audits: A Systematic Literature Search and Traditional Literature Synthesis. Can J Public Health 101 (Suppl 1), S34–S40 (2010). https://doi.org/10.1007/BF03403844

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF03403844

Key words

Mots clés

Search

Navigation