Surgical Endoscopy

, Volume 32, Issue 10, pp 4200–4208 | Cite as

Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure

  • Lisette Hvid HovgaardEmail author
  • Steven Arild Wuyts Andersen
  • Lars Konge
  • Torur Dalsgaard
  • Christian Rifbjerg Larsen



The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study's objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum.


Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module ‘Guided Vaginal Cuff Closure’ six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach’s alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups’ method.


The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach’s alpha). The experienced surgeons’ mean composite score for all six repetitions were significantly better than the novice surgeons’ (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives).


Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.


Robotic surgery Virtual reality simulation Gynaecology Assessment Proficiency-based training 



The research group sincerely thank all doctors from the Centre for Robotic Surgery at Copenhagen University Hospital, Herlev-Gentofte and from Robotic Surgery Section at Department of Gynaecology, Copenhagen University Hospital, Rigshospitalet-Glostrup for participation.

Compliance with ethical standards


Lisette H. Hovgaard and Drs. Steven A. W. Andersen, Lars Konge, Torur Dalsgaard and Christian R. Larsen have no conflicts of interest or financial ties to disclose.

Supplementary material

464_2018_6165_MOESM1_ESM.docx (109 kb)
Supplementary material 1 (DOCX 109 KB)
464_2018_6165_MOESM2_ESM.docx (93 kb)
Supplementary material 2 (DOCX 93 KB)


  1. 1.
    Schreuder HW, Verheijen RH (2009) Robotic surgery. BJOG 116(2):198–213. CrossRefPubMedGoogle Scholar
  2. 2.
    Weinberg L, Rao S, Escobar P (2011) Robotic surgery in gynecology: an updated systematic review. Obstet Gynecol Int 2011:852061. CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Yates DR, Vaessen C, Roupret M (2011) From Leonardo to da Vinci: the history of robot-assisted surgery in urology. BJU Int 108(11):1708–1713. (discussion 1714)CrossRefPubMedGoogle Scholar
  4. 4.
    Maan ZN, Gibbins N, Al-Jabri T, D’souza AR (2012) The use of robotics in otolaryngology-head and neck surgery: a systematic review. Am J Otolaryngol 33(1):137–146. CrossRefPubMedGoogle Scholar
  5. 5.
    Veronesi G (2013) Robotic surgery for the treatment of early-stage lung cancer. Curr Opin Oncol 25(2):107–114. CrossRefPubMedGoogle Scholar
  6. 6.
    Baek SK, Carmichael JC, Pigazzi A (2013) Robotic surgery: colon and rectum. Cancer J 19(2):140–146. CrossRefPubMedGoogle Scholar
  7. 7.
    Liu Z, Li X, Tian S, Zhu T, Yao Y, Tao Y (2017) Superiority of robotic surgery for cervical cancer in comparison with traditional approaches: a systematic review and meta-analysis. Int J Surg 40:145–154. CrossRefPubMedGoogle Scholar
  8. 8.
    Xie W, Cao D, Yang J, Shen K, Zhao L (2016) Robot-assisted surgery versus conventional laparoscopic surgery for endometrial cancer: a systematic review and meta-analysis. J Cancer Res Clin Oncol 142(10):2173–2183. CrossRefPubMedGoogle Scholar
  9. 9.
    Foote JR, Valea FA (2016) Robotic surgical training: where are we? Gynecol Oncol 143(1):179–183. CrossRefPubMedGoogle Scholar
  10. 10.
    Bric J, Connolly M, Kastenmeier A, Goldblatt M, Gould J (2014) Proficiency training on a virtual reality robotic surgical skills curriculum. Surg Endosc 28(12):3343–3348. CrossRefPubMedGoogle Scholar
  11. 11.
    Sandadi S, Gadzinski JA, Lee S, Chi DS, Sonoda Y, Jewell EL, Brown CL, Gardner GJ, Barakat RR, Leitao MM (2014) Fellowship learning curve associated with completing a robotic assisted total laparoscopic hysterectomy. Gynecol Oncol 132(1):102–106. CrossRefPubMedGoogle Scholar
  12. 12.
    Seamon LG, Fowler JM, Richardson DL, Carlson MJ, Valmadre S, Phillips GS, Cohn DE (2009) A detailed analysis of the learning curve: robotic hysterectomy and pelvic-aortic lymphadenectomy for endometrial cancer. Gynecol Oncol 114(2):162–167. CrossRefPubMedGoogle Scholar
  13. 13.
    Balasundaram I, Aggarwal R, Darzi A (2008) Short-phase training on a virtual reality simulator improves technical performance in tele-robotic surgery. Int J Med Robot 4(2):139–145. CrossRefPubMedGoogle Scholar
  14. 14.
    Schreuder HW, Wolswijk R, Zweemer RP, Schijven MP, Verheijen RH (2012) Training and learning robotic surgery, time for a more structured approach: a systematic review. BJOG 119(2):137–149. CrossRefPubMedGoogle Scholar
  15. 15.
    Andersen SA, Konge L, Cayé-Thomasen P, Sørensen MS (2015) Learning curves of virtual mastoidectomy in distributed and massed practice. JAMA Otolaryngol 141(10):913–918. CrossRefGoogle Scholar
  16. 16.
    Lendvay TS, Brand TC, White L, Kowalewski T, Jonnadula S, Mercer LD, Khorsand D, Andros J, Hannaford B, Satava R (2013) Virtual reality robotic surgery warm-up improves task performance in a dry laboratory environment: a prospective randomized controlled study. J Am Coll Surg 216(6):1181–1192. CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Larsen CR, Soerensen JL, Grantcharov TP, Dalsgaard T, Schouenborg L, Ottosen C, Schroeder TV, Ottesen BS (2009) Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. BMJ 338:B1802. CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Larsen CR, Oestergaard J, Ottosen C, Soerensen JL, (2012) The efficacy of virtual reality simulation training in laparoscopy: a systematic review of randomized trials. Acta Obstet Gynecol Scand 91(9):1015–1028. CrossRefPubMedGoogle Scholar
  19. 19.
    Culligan P, Gurshumov E, Lewis C, Priestley J, Komar J, Salamon C (2014) Predictive validity of a training protocol using a robotic surgery simulator. Female Pelvic Med Reconstr Surg 20(1):48–51. CrossRefPubMedGoogle Scholar
  20. 20.
    Kiely DJ, Gotlieb WH, Lau S, Zeng X, Samouelian V, Ramanakumar AV, Zakrzewski H, Brin S, Fraser SA, Korsieporn P, Drudi L (2015) Virtual reality robotic surgery simulation curriculum to teach robotic suturing: a randomized controlled trial. J Robot Surg 9(3):179–186. CrossRefPubMedGoogle Scholar
  21. 21.
    Hung AJ, Jayaratna IS, Teruya K, Desai MM, Gill IS, Goh AC (2013) Comparative assessment of three standardized robotic surgery training methods. BJU Int 112(6):864–871. CrossRefPubMedGoogle Scholar
  22. 22.
    Kenney PA, Wszolek MF, Gould JJ, Libertino JA, Moinzadeh A (2009) Face, content, and construct validity of dV-trainer, a novel virtual reality simulator for robotic surgery. Urology 73(6):1288–1292. CrossRefPubMedGoogle Scholar
  23. 23.
    Raza SJ, Froghi S, Chowriappa A, Ahmed K, Field E, Stegemann AP, Rehman S, Sharif M, Shi Y, Wilding GE, Kesavadas T (2014) Construct validation of the key components of fundamental skills of robotic surgery (FSRS) curriculum—a multi-institution prospective study. J Surg Educ 71(3):316–324. CrossRefPubMedGoogle Scholar
  24. 24.
    Whittaker G, Aydin A, Raison N, Kum F, Challacombe B, Khan MS, Dasgupta P, Ahmed K (2016) Validation of the robotix mentor robotic surgery simulator. J Endourol 30(3):338–346. CrossRefPubMedGoogle Scholar
  25. 25.
  26. 26.
    Fisher RA, Dasgupta P, Mottrie A, Volpe A, Khan MS, Challacombe B, Ahmed K (2015) An over-view of robot assisted surgery curricula and the status of their validation. Int J Surg 13:115–123. CrossRefPubMedGoogle Scholar
  27. 27.
    Bric JD, Lumbard DC, Frelich MJ, Gould JC (2016) current state of virtual reality simulation in robotic surgery training: a review. Surg Endosc 30(6):2169–2178. CrossRefPubMedGoogle Scholar
  28. 28.
    Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R (2013) Mastery learning for health professionals using technology-enhanced simulation: a systematic review and meta-analysis. Acad Med 88(8):1178–1186. CrossRefPubMedGoogle Scholar
  29. 29.
    Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R (2014) What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract 19(2):233–250. CrossRefPubMedGoogle Scholar
  30. 30.
    Ghaderi I, Manji F, Park YS, Juul D, Ott M, Harris I, Farrell T (2015) Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg 261(2):251–262. CrossRefPubMedGoogle Scholar
  31. 31.
    Larsen CR, Grantcharov T, Aggarwal R, Tully A, Sorensen JL, Dalsgaard T, Ottesen B (2006) objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator. Surg Endosc 20(9):1460–1466. CrossRefPubMedGoogle Scholar
  32. 32.
    Tulandi T, Einarsson J (2014) The use of barbed suture for laparoscopic hysterectomy and myomectomy: a systematic review and meta-analysis. J Minim Invasive Gynecol 21(2):210–216. CrossRefPubMedGoogle Scholar
  33. 33.
    American Educational Research Association APA, National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing (2014) Standards for educational and psychological testing. American Educational Research Association, Washington, DCGoogle Scholar
  34. 34.
    Bloch R, Norman G (2012) Generalizability theory for the perplexed: a practical introduction and guide: AMEE Guide No. 68. Med Teach 34(11):960–992. CrossRefPubMedGoogle Scholar
  35. 35.
    Downing SM, Yudkowsky R (2009) Assessment in Health Professions Education. Routledge, New YorkGoogle Scholar
  36. 36.
    Dyre L, Norgaard LN, Tabor A, Madsen ME, Sorensen JL, Ringsted C, Tolsgaard M (2016) Collecting validity evidence for the assessment of mastery learning in simulation-based ultrasound training. Ultraschall Med 37(4):386–392. CrossRefPubMedGoogle Scholar
  37. 37.
    Madsen ME, Konge L, Norgaard LN, Tabor A, Ringsted C, Klemmensen AK, Ottesen B, Tolsgaard M (2014) Assessment of performance measures and learning curves for use of a virtual-reality ultrasound simulator in transvaginal ultrasound examination. Ultrasound Obstet Gynecol 44(6):693–699. CrossRefPubMedGoogle Scholar
  38. 38.
    Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, Smith CD, Satava RM (2005) Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg 241(2):364–372CrossRefGoogle Scholar
  39. 39.
    Connolly M, Seligman J, Kastenmeier A, Goldblatt M, Gould J (2014) Validation of a virtual reality-based robotic surgical skills curriculum. Surg Endosc 28(5):1691–1694. CrossRefPubMedGoogle Scholar
  40. 40.
    Lee JY, Mucksavage P, Kerbl DC, Huynh VB, Etafy M, McDougall EM (2012) Validation study of a virtual reality robotic simulator—role as an assessment tool? J Urol 187(3):998–1002. CrossRefPubMedPubMedCentralGoogle Scholar
  41. 41.
    Sethi AS, Peine WJ, Mohammadi Y, Sundaram CP (2009) Validation of a novel virtual reality robotic simulator. J Endourol 23(3):503–508. CrossRefPubMedGoogle Scholar
  42. 42.
    Yule S, Flin R, Paterson-Brown S, Maran N (2006) Non-technical skills for surgeons in the operating room: a review of the literature. Surgery 139(2):140–149. CrossRefPubMedGoogle Scholar
  43. 43.
    Lyons C, Goldfarb D, Jones SL, Badhiwala N, Miles B, Link R, Dunkin B (2013) Which skills really matter? Proving face, content, and construct validity for a commercial robotic simulator. Surg Endosc 27(6):2020–2030. CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Lisette Hvid Hovgaard
    • 1
    • 2
    Email author
  • Steven Arild Wuyts Andersen
    • 3
    • 4
  • Lars Konge
    • 2
    • 3
  • Torur Dalsgaard
    • 5
  • Christian Rifbjerg Larsen
    • 1
    • 2
  1. 1.Centre for Robotic Surgery, Department of Gynaecology G-115Copenhagen University HospitalHerlev-GentofteDenmark
  2. 2.Faculty of Health and Medical SciencesUniversity of CopenhagenCopenhagenDenmark
  3. 3.The Simulation Centre at Rigshospitalet, Copenhagen Academy for Medical Education and Simulation (CAMES)Centre for HR, the Capital Region of DenmarkCopenhagenDenmark
  4. 4.Department of Otorhinolaryngology-Head & Neck SurgeryCopenhagen University HospitalRigshospitalet-GlostrupDenmark
  5. 5.Endometriosis Team and Robotic Surgery Section, Department of GynaecologyCopenhagen University HospitalRigshospitalet-GlostrupDenmark

Personalised recommendations