Validation of Metrics Coupled to Simulation Training

  • Anthony G. Gallagher
  • Gerald C. O’Sullivan
  • Gerald C. O’Sullivan
Part of the Improving Medical Outcome - Zero Tolerance book series (IMOZT)


In the previous chapters we identified very specific human factor reasons why medical procedures such as image guided surgery are difficult to learn and practice. They make unique perceptual, cognitive and psychomotor demands on the trainee and practitioner. The same research that has allowed us to identify these difficulties also provides us with potential solutions. One of the most important ways individuals learn is through “knowledge of results” of what they do and how their actions (or inaction) impacts where intended. The natural world provides human beings with ecologically valid information on their knowledge of results. However, this feedback information can be delivered at times when it is not contiguous to the behavior of the individual. In other words, an individual can do something which has negative consequences in the real world but they may not see the consequences of their behavior until much later. They fail to establish a relationship between their behavior and the negative results or outcomes and consequently, they almost certainly fail to modify their behavior.


Virtual Reality Predictive Validity Reliability Coefficient Simulation Training Procedural Skill 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Ahlberg G, Enochsson L, Gallagher AG, et al. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg. 2007;193(6):797-804.PubMedCrossRefGoogle Scholar
  2. APA. (1999). NCME (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education). Standards for educational and psychological tests.Google Scholar
  3. Bann S, Kwok KF, Lo CY, Darzi A, Wong J. Objective assessment of technical skills of surgical trainees in Hong Kong. Br J Surg. 2003;90(10):1294-1299.PubMedCrossRefGoogle Scholar
  4. Cates CU, Patel AD, Nicholson WJ. Use of virtual reality simulation for mission rehearsal for carotid stenting. J Am Med Assoc. 2007;297(3):265.CrossRefGoogle Scholar
  5. Centres R. Cholecystectomy practice transformed. Lancet. 1991;338(8770):789-790.Google Scholar
  6. Champion HR, Gallagher AG. Surgical simulation - a ‘good idea whose time has come’. Br J Surg. 2003;90(7):767-768.PubMedCrossRefGoogle Scholar
  7. Cooper C. Individual Differences. London: Arnold; 2002.Google Scholar
  8. Gallagher AG, Cates CU. Approval of virtual reality training for carotid stenting: what this means for procedural-based medicine. J Am Med Assoc. 2004a;292(24):3024-3026.CrossRefGoogle Scholar
  9. Gallagher AG, Cates CU. Virtual reality training for the operating room and cardiac catheterisation laboratory. Lancet. 2004b;364(9444):1538-1540.PubMedCrossRefGoogle Scholar
  10. Gallagher AG, Satava RM. Virtual reality as a metric for the assessment of laparoscopic psychomotor skills. Learning curves and reliability measures. Surg Endosc. 2002;16(12):1746-1752.PubMedCrossRefGoogle Scholar
  11. Gallagher AG, Smith CD. From the operating room of the present to the operating room of the future. Human-factors lessons learned from the minimally invasive surgery revolution. Semin Laparosc Surg. 2003;10(3):127-139.PubMedGoogle Scholar
  12. Gallagher AG, McClure N, McGuigan J, Crothers I, Browning J. Virtual reality training in laparoscopic surgery: a preliminary assessment of minimally invasive surgical trainer virtual reality (MIST VR). Endoscopy. 1999;31(4):310-313.PubMedCrossRefGoogle Scholar
  13. Gallagher AG, Richie K, McClure N, McGuigan J. Objective psychomotor skills assessment of experienced, junior, and novice laparoscopists with virtual reality. World J Surg. 2001;25(11):1478-1483.PubMedCrossRefGoogle Scholar
  14. Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg Endosc. 2003a;17(10):1525-1529.PubMedCrossRefGoogle Scholar
  15. Gallagher AG, Cowie R, Crothers I, Jordan-Black JA, Satava RM. PicSOr: an objective test of perceptual skill that predicts laparoscopic technical skill in three initial studies of laparoscopic performance. Surg Endosc. 2003b;17(9):1468-1471.PubMedCrossRefGoogle Scholar
  16. Gallagher AG, Lederman AB, McGlade K, Satava RM, Smith CD. Discriminative validity of the minimally invasive surgical trainer in virtual reality (MIST-VR) using criteria levels based on expert performance. Surg Endosc. 2004;18(4):660-665.PubMedCrossRefGoogle Scholar
  17. Gallagher AG, Ritter EM, Champion H, et al. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 2005;241(2):364-372.PubMedCrossRefGoogle Scholar
  18. Geis WP, Coletta AV, Verdeja JC, Plasencia G, Ojogho O, Jacobs M. Sequential psychomotor skills development in laparoscopic colon surgery. Arch Surg. 1994;129(2):206.PubMedCrossRefGoogle Scholar
  19. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg. 2004;91(2):146-150.PubMedCrossRefGoogle Scholar
  20. Hance J, Aggarwal R, Stanbridge R, et al. Objective assessment of technical skills in cardiac surgery. Eur J Cardiothorac Surg. 2005;28(1):157-162.PubMedCrossRefGoogle Scholar
  21. Horton R. Surgical research or comic opera: questions, but few answers. Lancet. 1996;347(9007):984.PubMedCrossRefGoogle Scholar
  22. Kennedy AM, Carroll S, Traynor O, Gallagher AG. Assessing surgical skill using bench station models. Plast Reconstr Surg. 2008;121(5):1869-1870.PubMedCrossRefGoogle Scholar
  23. Khan M, Bann S, Darzi A, Butler P. Assessing surgical skill using bench station models. Plast Reconstr Surg. 2007;120(3):793-800.PubMedCrossRefGoogle Scholar
  24. Larsen CR, Grantcharov T, Aggarwal R, et al. Objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator. Surg Endosc. 2006;20(9):1460-1466.PubMedCrossRefGoogle Scholar
  25. Larsen CR, Grantcharov T, Schouenborg L, Ottosen C, Sørensen JL, Ottesen B. Objective assessment of surgical competence in gynaecological laparoscopy: development and validation of a procedure specific rating scale. BJOG. 2008;115(7):908-916.PubMedCrossRefGoogle Scholar
  26. Larsen CR, Soerensen JL, Grantcharov TP, et al. Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. Br Med J. 2009;338:b1802.CrossRefGoogle Scholar
  27. Martin P, Martin PR, Bateson P. Measuring Behaviour: An Introductory Guide. Cambridge: Cambridge University Press; 1993.CrossRefGoogle Scholar
  28. Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273-278.PubMedCrossRefGoogle Scholar
  29. Melvin W, Johnson J, Ellison E. Laparoscopic skills enhancement. Am J Surg. 1996;172(4):377-379.PubMedCrossRefGoogle Scholar
  30. Moore MJ, Bennett CL. The learning curve for laparoscopic cholecystectomy. The southern surgeons club. Am J Surg. 1995;170(1):55.PubMedCrossRefGoogle Scholar
  31. Ojimba TA, Cameron AEP. Drawbacks of endoscopic thoracic sympathectomy. Br J Surg. 2004;91(3):264-269.PubMedCrossRefGoogle Scholar
  32. Reber AS, Reber ES. The Penguin Dictionary of Psychology. London: Penguin Books; 1985.Google Scholar
  33. Ritter EM, McClusky DA 3rd, Gallagher AG, Enochsson L, Smith CD. Perceptual, visuospatial, and psychomotor abilities correlate with duration of training required on a virtual-reality flexible endoscopy simulator. Am J Surg. 2006;192(3):379-384.PubMedCrossRefGoogle Scholar
  34. Rosser JC Jr, Rosser LE, Savalgi RS. Objective evaluation of a laparoscopic surgical skill program for residents and senior surgeons. Arch Surg. 1998;133(6):657.PubMedCrossRefGoogle Scholar
  35. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236(4):458-463. discussion 463-454.PubMedCrossRefGoogle Scholar
  36. Smith WS, Sung G, Saver J, et al. Mechanical thrombectomy for acute ischemic stroke: final results of the Multi MERCI trial. Stroke. 2008;39(4):1205.PubMedCrossRefGoogle Scholar
  37. Sutherland LM, Middleton PF, Anthony A, et al. Surgical simulation: a systematic review. Ann Surg. 2006;243(3):291.PubMedCrossRefGoogle Scholar
  38. Sutton C, McCloy R, Middlebrook A, Chater P, Wilson M, Stone R. MIST VR. A laparoscopic surgery procedures trainer and evaluator. Stud Health Technol Inform. 1997;39:598.PubMedGoogle Scholar
  39. Taffinder N, Sutton C, Fishwick R, McManus I, Darzi A. Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: results from randomised controlled studies using the MIST VR laparoscopic simulator. Stud Health Technol Inform. 1998;50:124-130.PubMedGoogle Scholar
  40. Van Sickle K, Iii D, Gallagher AG, Smith CD. Construct validation of the ProMIS simulator using a novel laparoscopic suturing task. Surg Endosc. 2005;19(9):1227-1231.PubMedCrossRefGoogle Scholar
  41. Van Sickle K, Baghai M, Huang IP, Goldenberg A, Smith CD, Ritter EM. Construct validity of an objective assessment method for laparoscopic intracorporeal suturing and knot tying. Am J Surg. 2008;196(1):74-80.PubMedCrossRefGoogle Scholar
  42. Wherry DC, Rob CG, Marohn MR, Rich NM. An external audit of laparoscopic cholecystectomy performed in medical treatment facilities of the department of defense. Ann Surg. 1994;220(5):626.PubMedCrossRefGoogle Scholar
  43. Winder J, Zheng H, Hughes S, Kelly B, Wilson C, Gallagher A. Increasing face validity of a vascular interventional training system. Stud Health Technol Inform. 2004;98:410-415.PubMedGoogle Scholar
  44. Yadav JS, Wholey MH, Kuntz RE, et al. Protected carotid-artery stenting versus endarterectomy in high-risk patients. N Engl J Med. 2004;351(15):1493.PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Anthony G. Gallagher
    • 1
  • Gerald C. O’Sullivan
    • 2
  • Gerald C. O’Sullivan
    • 3
  1. 1.School of MedicineUniversity College CorkCorkIreland
  2. 2.Mercy University HospitalCorkIreland
  3. 3.Cork Cancer Research CentreUniversity College CorkCorkIreland

Personalised recommendations