Internal and Emergency Medicine

, Volume 12, Issue 5, pp 651–656 | Cite as

An Italian version of the Ottawa Crisis Resource Management Global Rating Scale: a reliable and valid tool for assessment of simulation performance

  • Jeffrey Micheal FrancEmail author
  • Manuela Verde
  • Alba Ripoll Gallardo
  • Luca Carenzo
  • Pier Luigi Ingrassia


Objective measurement of simulation performance requires a validated and reliable tool. However, no published Italian language assessment tool is available. Translation of a published English language tool, the Ottawa Crisis Resource Management Global Rating Scale (GRS), may lead to a validated and reliable tool. After developing an Italian language translation of the English language tool, the study measured the reliability of the new tool by comparison with the English language tool used independently in the same simulation scenarios. In addition, the validity of the Italian language tool was measured by comparison to a skills score also applied independently. The correlation coefficient between the Italian language overall GRS and the English language overall GRS was 0.82 (adjusted 95 % confidence interval: 0.62–0.92). The correlation coefficient between the Italian language overall GRS and the skill score was 0.85 (adjusted 95 % confidence interval 0.68–0.94). This study demonstrated that the Italian language GRS has acceptable reliability when compared with the English language tool, suggesting that it can be used reliably to evaluate the performance during simulated emergencies. The study also suggests that the tool has acceptable validity for assessing the simulation performance. The study suggests that the Italian language GRS translation has reasonable reliability when compared with the English language GRS and reasonable validity when compared with the assessment of the skills scores. Data suggest that the instrument is adequately reliable for informal and formative type of examinations, but may require further confirmation before use for high-stake examinations such as licensing.


High-fidelity simulation Reliability Validity Assessment tool Emergency medicine 


Compliance with ethical standards

Conflicts of interest

The authors declare that they have no conflict of interest.

Statement of human and animal rights

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institution an/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Supplementary material

11739_2016_1486_MOESM1_ESM.pdf (114 kb)
Italian Version of the Ottawa Global Rating Scale (PDF 114 kb)
11739_2016_1486_MOESM2_ESM.pdf (977 kb)
Sample Skill Scoring Sheet (PDF 976 kb)


  1. 1.
    Ericsson KA, Krampe RT, Tesch-Römer C (1993) The role of deliberate practice in the acquisition of expert performance. Psychol Rev 100:363–406CrossRefGoogle Scholar
  2. 2.
    Gibbs G (1988) Learning by doing: a guide to teaching and learning methods. FEUGoogle Scholar
  3. 3.
    Grant J (1992) Training senior house officers by service-based learning. Joint centre for education in medicine, LondonGoogle Scholar
  4. 4.
    Kolb DA (2014) Experiential learning: experience as the source of learning and development. Pearson EducationGoogle Scholar
  5. 5.
    Levine AI, Schwartz AD, Bryson EO, Demaria S (2012) Role of simulation in US physician licensure and certification. Mt Sinai J Med 79:140–153CrossRefPubMedGoogle Scholar
  6. 6.
    Buyske J (2010) The role of simulation in certification. Surg Clin North Am 90:619–621CrossRefPubMedGoogle Scholar
  7. 7.
    Gordon JA, Tancredi DN, Binder WD, Wilkerson WM, Shaffer DW (2003) Assessment of a clinical performance evaluation tool for use in a simulator-based testing environment: a pilot study. Acad Med 78:S45–S47CrossRefPubMedGoogle Scholar
  8. 8.
    Epstein RM (2007) Medical education—assessment in medical education. N Engl J Med 356:387–396CrossRefPubMedGoogle Scholar
  9. 9.
    Downing SM (2004) Reliability: on the reproducibility of assessment data. Med Educ 38:1006–1012CrossRefPubMedGoogle Scholar
  10. 10.
    Downing SM (2003) Validity: on the meaningful interpretation of assessment data. Med Educ 37:830–837CrossRefPubMedGoogle Scholar
  11. 11.
    Kim J, Neilipovitz D, Cardinal P, Chiu M, Clinch J (2006) A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: the University of Ottawa Critical Care Medicine, high-fidelity simulation, and crisis resource management I study. Crit Care Med 34:2167–2174CrossRefPubMedGoogle Scholar
  12. 12.
    Okuda Y, Godwin SA, Jacobson L, Wang E, Weingart S (2014) SimWars. J Emerg Med 47:586–593CrossRefPubMedGoogle Scholar
  13. 13.
    Devore J (2011) Probability and statistics for engineering and the sciences. Thomson Brooks/Cole, USAGoogle Scholar
  14. 14.
    Litwin MS (1995) How to measure survey reliability and validity. SAGE Publications, USACrossRefGoogle Scholar
  15. 15.
    Hsu J (1996) Multiple comparisons: theory and methods. CRC Press, USACrossRefGoogle Scholar
  16. 16.
    Peng RD (2011) Reproducible research in computational science. Science 334:1226–1227CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Franc JM (2016) Reveal project. Accessed 18 Mar 2016
  18. 18.
    Montgomery DC (2009) Design and analysis of experiments. Wiley, USAGoogle Scholar
  19. 19.
    Flin R, Yule S, Paterson-Brown S, Maran N, Rowley D, Youngson G (2007) Teaching surgeons about non-technical skills. Surgeon 5:86–89CrossRefPubMedGoogle Scholar
  20. 20.
    Powers KA, Rehrig ST, Irias N, Albano HA, Malinow A, Jones SB, Moorman DW, Pawlowski JB, Jones DB (2008) Simulated laparoscopic operating room crisis: an approach to enhance the surgical team performance. Surg Endosc 22:885–900CrossRefPubMedGoogle Scholar
  21. 21.
    McCulloch P, Mishra A, Handa A, Dale T, Hirst G, Catchpole K (2009) The effects of aviation-style non-technical skills training on technical performance and outcome in the operating theatre. Qual Saf Health Car 18:109–115CrossRefGoogle Scholar
  22. 22.
    Riem N, Boet S, Bould MD, Tavares W, Naik VN (2012) Do technical skills correlate with non-technical skills in crisis resource management: a simulation study. Br J Anaesth 109:723–728CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Brunckhorst O, Shahid S, Aydin A, Khan S, Mcllhenny C, Brewin J, Sahai A, Bello F, Kneebone R, Shamim Khan M, Dasgupta P, Ahmed K (2015) The relationship between technical and nontechnical skills within a simulation-based ureteroscopy training environment. J Surg Educ 72(5):1039–1044CrossRefPubMedGoogle Scholar

Copyright information

© SIMI 2016

Authors and Affiliations

  • Jeffrey Micheal Franc
    • 1
    • 2
    Email author
  • Manuela Verde
    • 2
  • Alba Ripoll Gallardo
    • 2
  • Luca Carenzo
    • 2
  • Pier Luigi Ingrassia
    • 2
  1. 1.University of AlbertaEdmontonCanada
  2. 2.Universita’ del Piemonte OrientaleNovaraItaly

Personalised recommendations