Skip to main content
SpringerLink
Log in

Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study's objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum.

Methods

Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module ‘Guided Vaginal Cuff Closure’ six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach’s alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups’ method.

Results

The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach’s alpha). The experienced surgeons’ mean composite score for all six repetitions were significantly better than the novice surgeons’ (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives).

Conclusion

Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Schreuder HW, Verheijen RH (2009) Robotic surgery. BJOG 116(2):198–213. https://doi.org/10.1111/J.1471-0528.2008.02038.X

    Article  CAS  PubMed  Google Scholar 

  2. Weinberg L, Rao S, Escobar P (2011) Robotic surgery in gynecology: an updated systematic review. Obstet Gynecol Int 2011:852061. https://doi.org/10.1155/2011/852061

    Article  PubMed  PubMed Central  Google Scholar 

  3. Yates DR, Vaessen C, Roupret M (2011) From Leonardo to da Vinci: the history of robot-assisted surgery in urology. BJU Int 108(11):1708–1713. https://doi.org/10.1111/J.1464-410x.2011.10576.X (discussion 1714)

    Article  PubMed  Google Scholar 

  4. Maan ZN, Gibbins N, Al-Jabri T, D’souza AR (2012) The use of robotics in otolaryngology-head and neck surgery: a systematic review. Am J Otolaryngol 33(1):137–146. https://doi.org/10.1016/J.Amjoto.2011.04.003

    Article  PubMed  Google Scholar 

  5. Veronesi G (2013) Robotic surgery for the treatment of early-stage lung cancer. Curr Opin Oncol 25(2):107–114. https://doi.org/10.1097/Cco.0b013e32835daf4f

    Article  PubMed  Google Scholar 

  6. Baek SK, Carmichael JC, Pigazzi A (2013) Robotic surgery: colon and rectum. Cancer J 19(2):140–146. https://doi.org/10.1097/Ppo.0b013e31828ba0fd

    Article  PubMed  Google Scholar 

  7. Liu Z, Li X, Tian S, Zhu T, Yao Y, Tao Y (2017) Superiority of robotic surgery for cervical cancer in comparison with traditional approaches: a systematic review and meta-analysis. Int J Surg 40:145–154. https://doi.org/10.1016/J.Ijsu.2017.02.062

    Article  PubMed  Google Scholar 

  8. Xie W, Cao D, Yang J, Shen K, Zhao L (2016) Robot-assisted surgery versus conventional laparoscopic surgery for endometrial cancer: a systematic review and meta-analysis. J Cancer Res Clin Oncol 142(10):2173–2183. https://doi.org/10.1007/S00432-016-2180-X

    Article  PubMed  Google Scholar 

  9. Foote JR, Valea FA (2016) Robotic surgical training: where are we? Gynecol Oncol 143(1):179–183. https://doi.org/10.1016/J.Ygyno.2016.05.031

    Article  PubMed  Google Scholar 

  10. Bric J, Connolly M, Kastenmeier A, Goldblatt M, Gould J (2014) Proficiency training on a virtual reality robotic surgical skills curriculum. Surg Endosc 28(12):3343–3348. https://doi.org/10.1007/S00464-014-3624-5

    Article  PubMed  Google Scholar 

  11. Sandadi S, Gadzinski JA, Lee S, Chi DS, Sonoda Y, Jewell EL, Brown CL, Gardner GJ, Barakat RR, Leitao MM (2014) Fellowship learning curve associated with completing a robotic assisted total laparoscopic hysterectomy. Gynecol Oncol 132(1):102–106. https://doi.org/10.1016/J.Ygyno.2013.11.017

    Article  PubMed  Google Scholar 

  12. Seamon LG, Fowler JM, Richardson DL, Carlson MJ, Valmadre S, Phillips GS, Cohn DE (2009) A detailed analysis of the learning curve: robotic hysterectomy and pelvic-aortic lymphadenectomy for endometrial cancer. Gynecol Oncol 114(2):162–167. https://doi.org/10.1016/J.Ygyno.2009.04.017

    Article  PubMed  Google Scholar 

  13. Balasundaram I, Aggarwal R, Darzi A (2008) Short-phase training on a virtual reality simulator improves technical performance in tele-robotic surgery. Int J Med Robot 4(2):139–145. https://doi.org/10.1002/Rcs.181

    Article  PubMed  Google Scholar 

  14. Schreuder HW, Wolswijk R, Zweemer RP, Schijven MP, Verheijen RH (2012) Training and learning robotic surgery, time for a more structured approach: a systematic review. BJOG 119(2):137–149. https://doi.org/10.1111/J.1471-0528.2011.03139.X

    Article  CAS  PubMed  Google Scholar 

  15. Andersen SA, Konge L, Cayé-Thomasen P, Sørensen MS (2015) Learning curves of virtual mastoidectomy in distributed and massed practice. JAMA Otolaryngol 141(10):913–918. https://doi.org/10.1001/Jamaoto.2015.1563

    Article  Google Scholar 

  16. Lendvay TS, Brand TC, White L, Kowalewski T, Jonnadula S, Mercer LD, Khorsand D, Andros J, Hannaford B, Satava R (2013) Virtual reality robotic surgery warm-up improves task performance in a dry laboratory environment: a prospective randomized controlled study. J Am Coll Surg 216(6):1181–1192. https://doi.org/10.1016/J.Jamcollsurg.2013.02.012

    Article  PubMed  PubMed Central  Google Scholar 

  17. Larsen CR, Soerensen JL, Grantcharov TP, Dalsgaard T, Schouenborg L, Ottosen C, Schroeder TV, Ottesen BS (2009) Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. BMJ 338:B1802. https://doi.org/10.1136/Bmj.B1802

    Article  PubMed  PubMed Central  Google Scholar 

  18. Larsen CR, Oestergaard J, Ottosen C, Soerensen JL, (2012) The efficacy of virtual reality simulation training in laparoscopy: a systematic review of randomized trials. Acta Obstet Gynecol Scand 91(9):1015–1028. https://doi.org/10.1111/J.1600-0412.2012.01482.X

    Article  PubMed  Google Scholar 

  19. Culligan P, Gurshumov E, Lewis C, Priestley J, Komar J, Salamon C (2014) Predictive validity of a training protocol using a robotic surgery simulator. Female Pelvic Med Reconstr Surg 20(1):48–51. https://doi.org/10.1097/Spv.0000000000000045

    Article  PubMed  Google Scholar 

  20. Kiely DJ, Gotlieb WH, Lau S, Zeng X, Samouelian V, Ramanakumar AV, Zakrzewski H, Brin S, Fraser SA, Korsieporn P, Drudi L (2015) Virtual reality robotic surgery simulation curriculum to teach robotic suturing: a randomized controlled trial. J Robot Surg 9(3):179–186. https://doi.org/10.1007/S11701-015-0513-4

    Article  PubMed  Google Scholar 

  21. Hung AJ, Jayaratna IS, Teruya K, Desai MM, Gill IS, Goh AC (2013) Comparative assessment of three standardized robotic surgery training methods. BJU Int 112(6):864–871. https://doi.org/10.1111/Bju.12045

    Article  PubMed  Google Scholar 

  22. Kenney PA, Wszolek MF, Gould JJ, Libertino JA, Moinzadeh A (2009) Face, content, and construct validity of dV-trainer, a novel virtual reality simulator for robotic surgery. Urology 73(6):1288–1292. https://doi.org/10.1016/J.Urology.2008.12.044

    Article  PubMed  Google Scholar 

  23. Raza SJ, Froghi S, Chowriappa A, Ahmed K, Field E, Stegemann AP, Rehman S, Sharif M, Shi Y, Wilding GE, Kesavadas T (2014) Construct validation of the key components of fundamental skills of robotic surgery (FSRS) curriculum—a multi-institution prospective study. J Surg Educ 71(3):316–324. https://doi.org/10.1016/J.Jsurg.2013.10.006

    Article  PubMed  Google Scholar 

  24. Whittaker G, Aydin A, Raison N, Kum F, Challacombe B, Khan MS, Dasgupta P, Ahmed K (2016) Validation of the robotix mentor robotic surgery simulator. J Endourol 30(3):338–346. https://doi.org/10.1089/End.2015.0620

    Article  PubMed  Google Scholar 

  25. Simbionix (2017) http://simbionix.com/simulators/robotix-mentor/robotix-library-of-modules/. Accessed 21 Feb 2017

  26. Fisher RA, Dasgupta P, Mottrie A, Volpe A, Khan MS, Challacombe B, Ahmed K (2015) An over-view of robot assisted surgery curricula and the status of their validation. Int J Surg 13:115–123. https://doi.org/10.1016/J.Ijsu.2014.11.033

    Article  PubMed  Google Scholar 

  27. Bric JD, Lumbard DC, Frelich MJ, Gould JC (2016) current state of virtual reality simulation in robotic surgery training: a review. Surg Endosc 30(6):2169–2178. https://doi.org/10.1007/S00464-015-4517-Y

    Article  PubMed  Google Scholar 

  28. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R (2013) Mastery learning for health professionals using technology-enhanced simulation: a systematic review and meta-analysis. Acad Med 88(8):1178–1186. https://doi.org/10.1097/Acm.0b013e31829a365d

    Article  PubMed  Google Scholar 

  29. Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R (2014) What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract 19(2):233–250. https://doi.org/10.1007/S10459-013-9458-4

    Article  PubMed  Google Scholar 

  30. Ghaderi I, Manji F, Park YS, Juul D, Ott M, Harris I, Farrell T (2015) Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg 261(2):251–262. https://doi.org/10.1097/Sla.0000000000000520

    Article  PubMed  Google Scholar 

  31. Larsen CR, Grantcharov T, Aggarwal R, Tully A, Sorensen JL, Dalsgaard T, Ottesen B (2006) objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator. Surg Endosc 20(9):1460–1466. https://doi.org/10.1007/S00464-005-0745-X

    Article  CAS  PubMed  Google Scholar 

  32. Tulandi T, Einarsson J (2014) The use of barbed suture for laparoscopic hysterectomy and myomectomy: a systematic review and meta-analysis. J Minim Invasive Gynecol 21(2):210–216. https://doi.org/10.1016/J.Jmig.2013.09.014

    Article  PubMed  Google Scholar 

  33. American Educational Research Association APA, National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing (2014) Standards for educational and psychological testing. American Educational Research Association, Washington, DC

    Google Scholar 

  34. Bloch R, Norman G (2012) Generalizability theory for the perplexed: a practical introduction and guide: AMEE Guide No. 68. Med Teach 34(11):960–992. https://doi.org/10.3109/0142159x.2012.703791

    Article  PubMed  Google Scholar 

  35. Downing SM, Yudkowsky R (2009) Assessment in Health Professions Education. Routledge, New York

    Google Scholar 

  36. Dyre L, Norgaard LN, Tabor A, Madsen ME, Sorensen JL, Ringsted C, Tolsgaard M (2016) Collecting validity evidence for the assessment of mastery learning in simulation-based ultrasound training. Ultraschall Med 37(4):386–392. https://doi.org/10.1055/S-0041-107976

    Article  CAS  PubMed  Google Scholar 

  37. Madsen ME, Konge L, Norgaard LN, Tabor A, Ringsted C, Klemmensen AK, Ottesen B, Tolsgaard M (2014) Assessment of performance measures and learning curves for use of a virtual-reality ultrasound simulator in transvaginal ultrasound examination. Ultrasound Obstet Gynecol 44(6):693–699. https://doi.org/10.1002/Uog.13400

    Article  CAS  PubMed  Google Scholar 

  38. Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, Smith CD, Satava RM (2005) Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg 241(2):364–372

    Article  PubMed  PubMed Central  Google Scholar 

  39. Connolly M, Seligman J, Kastenmeier A, Goldblatt M, Gould J (2014) Validation of a virtual reality-based robotic surgical skills curriculum. Surg Endosc 28(5):1691–1694. https://doi.org/10.1007/S00464-013-3373-X

    Article  PubMed  Google Scholar 

  40. Lee JY, Mucksavage P, Kerbl DC, Huynh VB, Etafy M, McDougall EM (2012) Validation study of a virtual reality robotic simulator—role as an assessment tool? J Urol 187(3):998–1002. https://doi.org/10.1016/J.Juro.2011.10.160

    Article  PubMed  Google Scholar 

  41. Sethi AS, Peine WJ, Mohammadi Y, Sundaram CP (2009) Validation of a novel virtual reality robotic simulator. J Endourol 23(3):503–508. https://doi.org/10.1089/End.2008.0250

    Article  PubMed  Google Scholar 

  42. Yule S, Flin R, Paterson-Brown S, Maran N (2006) Non-technical skills for surgeons in the operating room: a review of the literature. Surgery 139(2):140–149. https://doi.org/10.1016/J.Surg.2005.06.017

    Article  CAS  PubMed  Google Scholar 

  43. Lyons C, Goldfarb D, Jones SL, Badhiwala N, Miles B, Link R, Dunkin B (2013) Which skills really matter? Proving face, content, and construct validity for a commercial robotic simulator. Surg Endosc 27(6):2020–2030. https://doi.org/10.1007/S00464-012-2704-7

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The research group sincerely thank all doctors from the Centre for Robotic Surgery at Copenhagen University Hospital, Herlev-Gentofte and from Robotic Surgery Section at Department of Gynaecology, Copenhagen University Hospital, Rigshospitalet-Glostrup for participation.

Author information

Authors and Affiliations

  1. Centre for Robotic Surgery, Department of Gynaecology G-115, Copenhagen University Hospital, Herlev Ringvej 75, 2730, Herlev-Gentofte, Denmark

    Lisette Hvid Hovgaard & Christian Rifbjerg Larsen

  2. Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark

    Lisette Hvid Hovgaard, Lars Konge & Christian Rifbjerg Larsen

  3. The Simulation Centre at Rigshospitalet, Copenhagen Academy for Medical Education and Simulation (CAMES), Centre for HR, the Capital Region of Denmark, Copenhagen, Denmark

    Steven Arild Wuyts Andersen & Lars Konge

  4. Department of Otorhinolaryngology-Head & Neck Surgery, Copenhagen University Hospital, Rigshospitalet-Glostrup, Denmark

    Steven Arild Wuyts Andersen

  5. Endometriosis Team and Robotic Surgery Section, Department of Gynaecology, Copenhagen University Hospital, Rigshospitalet-Glostrup, Denmark

    Torur Dalsgaard

Authors
  1. Lisette Hvid Hovgaard
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven Arild Wuyts Andersen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lars Konge
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Torur Dalsgaard
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christian Rifbjerg Larsen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lisette Hvid Hovgaard.

Ethics declarations

Disclosures

Lisette H. Hovgaard and Drs. Steven A. W. Andersen, Lars Konge, Torur Dalsgaard and Christian R. Larsen have no conflicts of interest or financial ties to disclose.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 109 KB)

Supplementary material 2 (DOCX 93 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hovgaard, L.H., Andersen, S.A.W., Konge, L. et al. Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure. Surg Endosc 32, 4200–4208 (2018). https://doi.org/10.1007/s00464-018-6165-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-018-6165-5

Keyword

Search

Navigation