Advertisement

Developing and validating a tool for measuring the educational environment in clinical anesthesia

  • Navdeep S. Sidhu
  • Eleri Clissold
Reports of Original Investigations

Abstract

Purpose

We aimed to develop a contemporary measure for anesthesia teaching and learning in the operating theatre that was applicable to a variety of training jurisdictions, the Measure for the Anaesthesia Theatre Educational Environment (MATE).

Methods

A systematic review of the literature and modified Delphi approach was used to identify items for content validity. Reliability and exploratory factor analyses were conducted after a pilot survey of trainees to show construct validity, with removal of redundant items. Item domains were identified through a global assessment of factor structure accuracy and relation to real-world constructs.

Results

Literature review generated an initial 73-item list. A modified Delphi approach with 24 experts identified 44 relevant items. The pilot survey generated 390 responses. Reliability analysis, exploratory factor analysis, and global assessment refined the measure to 33 items. Four domains were identified according to factor structure: teaching preparation and practice, assessment and feedback, procedures and responsibility, and overall atmosphere. The educational environment was rated by trainees at 74.6 ± 15.6% with excellent internal consistency (Cronbach’s α = 0.975).

Conclusion

The MATE survey tool generated valid and reliable scores when measuring the educational environment in the operating theatre. Further research is required to investigate possible differences between the training countries and age of junior doctors and the associated underlying factors. Other researchers are invited to administer the survey and share results within a central database.

Élaboration et validation d’un outil de mesure de l’environnement éducatif en anesthésie clinique

Résumé

Objectif

Nous avons cherché à élaborer une mesure contemporaine pour l’enseignement et l’apprentissage de l’anesthésie en salles d’opération qui pourrait être appliquée dans différents cadres de formation : la mesure pour l’environnement éducatif en salle d’anesthésie ou MATE (Measure for the Anaesthesia Theatre Educational Environment).

Méthodes

Une revue systématique des publications et une approche de Delphes modifiée ont servi à identifier les éléments de validation du contenu. Des analyses de fiabilité et de facteurs exploratoires ont été menées après une enquête pilote auprès de stagiaires pour montrer la fiabilité du montage avec la suppression d’éléments redondants. Les domaines d’items ont été identifiés via une évaluation globale de l’exactitude des structures de facteurs et leurs relations avec des montages en situation réelle.

Résultats

La recherche bibliographique a permis de créer une liste initiale de 73 éléments. Une approche de Delphes modifiée avec 24 experts a identifié 44 éléments pertinents. L’enquête pilote a généré 390 réponses. L’analyse de fiabilité, l’analyse des facteurs exploratoires et l’évaluation globale ont permis d’affiner la mesure à 33 éléments. Quatre domaines ont été identifiés en fonction de la structure des facteurs : préparation et pratique de l’enseignement, évaluation et rétroaction, procédures et responsabilité, et environnement global. Les stagiaires ont attribué à l’environnement éducatif une cote de 74,6 ± 15,6 % avec une excellente homogénéité interne (coefficient α de Cronbach = 0,975).

Conclusion

L’outil d’enquête MATE a généré des scores valides et fiables pour la mesure de l’environnement éducatif en salle d’opération. Des recherches supplémentaires sont nécessaires pour étudier les différences possibles entre les pays de formations, l’âge des jeunes médecins et les facteurs sous-jacents associés. D’autres chercheurs sont invités à administrer l’enquête et à en partager les résultats dans une base de données centrale.

Notes

Acknowledgements

We thank the following individuals for their input: Tom Burrows, Damien Castanelli, Nina Civil, Marlin De Silva, Kirsty Forrest, Alistair Kan, Laura Kwan, David Law, Emelyn Lee, Helen Lindsay, Neil Macdonald, Nola Ng, Lindy Roberts, Ross Scott-Weekly, Natalie Smith, Ben Snow, Melanie Speer, Timothy Starkie, Ghassan Talab, Michael Tan, Kersi Taraporewalla, Jennifer Weller, Eva Wilson, and Caroline Zhou. We also thank the Australian and New Zealand College of Anaesthetists (ANZCA) Clinical Trials Network for facilitating survey distribution to New Zealand and Australian trainees and all educational supervisors and residents internationally who engaged in the study.

Declaration of interests

No external funding and no competing interests declared

Editorial responsibility

This submission was handled by Dr. Gregory L. Bryson, Deputy Editor-in-Chief, Canadian Journal of Anesthesia.

Author contributions

Navdeep Sidhu contributed substantially to all aspects of this manuscript, including the conception and design, acquisition, analysis and interpretation of data, and drafting the article. Eleri Clissold contributed substantially to the conception and design of the manuscript, analysis of data, and drafting the article.

Supplementary material

12630_2018_1185_MOESM1_ESM.pdf (142 kb)
Supplementary material 1 (PDF 141 kb)

References

  1. 1.
    American Medical Association. Report of the Council on Medical Education 7-A-09. Transforming the medical education learning environment. Available from URL: https://www.ama-assn.org/sites/default/files/media-browser/public/about-ama/councils/Council Reports/council-on-medical-education/a09-cme-transforming-medical-education-learning-environment.pdf (accessed May 2018).
  2. 2.
    Soemantri D, Herrera C, Riquelme A. Measuring the educational environment in health professions studies: a systematic review. Med Teach 2010; 32: 947-52.CrossRefGoogle Scholar
  3. 3.
    Holt MC, Roff S. Development and validation of the Anaesthetic Theatre Educational Environment Measure (ATEEM). Med Teach 2004; 26: 553-8.CrossRefGoogle Scholar
  4. 4.
    Smith NA, Castanelli DJ. Measuring the clinical learning environment in anaesthesia. Anaesth Intensive Care 2015; 43: 199-203.PubMedGoogle Scholar
  5. 5.
    Castanelli DJ, Smith NA. Measuring the anaesthesia clinical learning environment at the department level is feasible and reliable. Br J Anaesth 2017; 118: 733-9.CrossRefGoogle Scholar
  6. 6.
    Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs 2000; 32: 1008-15.PubMedGoogle Scholar
  7. 7.
    Clayton MJ. Delphi: a technique to harness expert opinion for critical decision-making tasks in education. Educ Psychol 1997; 17: 373-86.CrossRefGoogle Scholar
  8. 8.
    Beavers AS, Lounsbury JW, Richards JK, Huck SW, Skolits GJ, Esquivel SL. Practical considerations for using exploratory factor analysis in educational research. Practical Assessment, Research & Evaluation 2013; 18: 1-13. Available from URL: http://pareonline.net/pdf/v18n6.pdf (accessed May 2018).
  9. 9.
    Costello AB, Osborne JW. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation 2005; 10: 1-9. Available from URL: http://pareonline.net/getvn.asp?v=10&n=7 (accessed May 2018).
  10. 10.
    Fabrigar LR, Wegener DT, MacCallum RC, Strahan EJ. Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods 1999; 4: 272-99.CrossRefGoogle Scholar
  11. 11.
    Cleave-Hogg D, Benedict C. Characteristics of good anaesthesia teachers. Can J Anaesth 1997; 44: 587-91.CrossRefGoogle Scholar
  12. 12.
    Ortwein H, Blaum WE, Spies CD. Anesthesiology residents’ perspective about good teaching - a qualitative needs assessment. Ger Med Sci 2014; 12: Doc05.Google Scholar
  13. 13.
    Haydar B, Charnin J, Voepel-Lewis T, Baker K. Resident characterization of better-than- and worse-than-average clinical teaching. Anesthesiology 2014; 120: 120-8.CrossRefGoogle Scholar
  14. 14.
    Lombarts KM, Bucx MJ, Arah OA. Development of a system for the evaluation of the teaching qualities of anesthesiology faculty. Anesthesiology 2009; 111: 709-16.CrossRefGoogle Scholar
  15. 15.
    de Oliveira Filho GR, Dal Mago AJ, Garcia JM, Goldschmidt R. An instrument designed for faculty supervision evaluation by anesthesia residents and its psychometric properties. Anesth Analg 2008; 107: 1316-22.CrossRefGoogle Scholar
  16. 16.
    Moors G, Kieruj ND, Vermunt JK. The effect of labeling and numbering of response scales on the likelihood of response bias. Sociol Methodol 2014; 44: 369-99.CrossRefGoogle Scholar
  17. 17.
    Alwin DF, Krosnick JA. The reliability of survey attitude measurement: the influence of question and respondent attributes. Sociol Methods Res 1991; 20: 139-81.CrossRefGoogle Scholar
  18. 18.
    Saris WE, Gallhofer IN. Design, Evaluation, and Analysis of Questionnaires for Survey Research. Hoboken, NJ: John Wiley & Sons; 2007.CrossRefGoogle Scholar
  19. 19.
    Kanashiro J, McAleer S, Roff S. Assessing the educational environment in the operating room - a measure of resident perception at one Canadian institution. Surgery 2006; 139: 150-8.CrossRefGoogle Scholar
  20. 20.
    Mahoney A, Crowe PJ, Harris P. Exploring Australasian surgical trainees’ satisfaction with operating theatre learning using the ‘surgical theatre educational environment measure’. ANZ J Surg 2010; 80: 884-9.CrossRefGoogle Scholar
  21. 21.
    Yin T, Child S. The Auckland Surgical Theatre Educational Environment Measure: does attending surgery benefit house officers? N Z Med J 2015; 128: 94-8.PubMedGoogle Scholar
  22. 22.
    Binsaleh S, Babaeer A, Rabah D, Madbouly K. Evaluation of urology residents’ perception of surgical theater educational environment. J Surg Educ 2014; 72: 73-9.CrossRefGoogle Scholar
  23. 23.
    Palmgren PJ, Sundberg T, Laksov KB. Reassessing the educational environment among undergraduate students in a chiropractic training institution: a study over time. J Chiropr Educ 2015; 29: 110-26.CrossRefGoogle Scholar
  24. 24.
    Wong PN, John DN, Deslandes RE, Hughes ML. Same syllabus, different country - using DREEM to compare the educational environments at two Pharmacy schools. Pharm Educ 2015; 15: 87-92.Google Scholar
  25. 25.
    Leung Y, Salfinger S, Mercer A. The positive impact of structured teaching in the operating room. Aust NZ J Obstet Gynaecol 2015; 55: 601-5.CrossRefGoogle Scholar
  26. 26.
    Finn Y, Avalos G, Dunne F. Positive changes in the medical educational environment following introduction of a new systems-based curriculum: DREEM or reality? Curricular change and the environment. Ir J Med Sci 2014; 183: 253-8.CrossRefGoogle Scholar
  27. 27.
    Shankar PR, Bharti R, Ramireddy R, Balasubramanium R, Nuguri V. Students’ perception of the learning environment at Xavier University School of Medicine, Aruba: a follow-up study. J Educ Eval Health Prof 2014; 11: 9.CrossRefGoogle Scholar
  28. 28.
    Qin Y, Wang Y, Floden RE. The effect of problem-based learning on improvement of the medical educational environment: a systematic review and meta-analysis. Med Princ Pract 2016; 25: 525-32.CrossRefGoogle Scholar
  29. 29.
    Chan CY, Sum MY, Lim WS, Chew NW, Samarasekera DD, Sim K. Adoption and correlates of Postgraduate Hospital Educational Environment Measure (PHEEM) in the evaluation of learning environments - a systematic review. Med Teach 2016; 38: 1248-55.CrossRefGoogle Scholar
  30. 30.
    Shimizu T, Tsugawa Y, Tanoue Y, et al. The hospital educational environment and performance of residents in the General Medicine In Training Examination: a multicenter study in Japan. Int J Gen Med 2013; 6: 637-40.CrossRefGoogle Scholar
  31. 31.
    Al-Ansari AA, El Tantawi MM. Predicting academic performance of dental students using perception of educational environment. J Dent Educ 2015; 79: 337-44.PubMedGoogle Scholar
  32. 32.
    Sarwar S, Tarique S. Perception of educational environment: does it impact academic performance of medical students? J Pak Med Assoc 2016; 66: 1210-4.PubMedGoogle Scholar
  33. 33.
    Payne LK, Glaspie T. Associations between baccalaureate nursing students’ perceptions of educational environment and HESI™ scores and GPA. Nurse Educ Today 2014; 34: e64-8.CrossRefGoogle Scholar
  34. 34.
    Tempski P, Santos IS, Mayer FB, et al. Relationship among medical student resilience, educational environment and quality of life. PLoS One 2015; 10: e0131535.CrossRefGoogle Scholar

Copyright information

© Canadian Anesthesiologists' Society 2018

Authors and Affiliations

  1. 1.Department of Anaesthesia and Perioperative MedicineNorth Shore HospitalAucklandNew Zealand
  2. 2.Institute for Innovation and ImprovementWaitemata District Health BoardAucklandNew Zealand

Personalised recommendations