Neurosurgical virtual reality simulation metrics to assess psychomotor skills during brain tumor resection



   Virtual reality simulator technology together with novel metrics could advance our understanding of expert neurosurgical performance and modify and improve resident training and assessment. This pilot study introduces innovative metrics that can be measured by the state-of-the-art simulator to assess performance. Such metrics cannot be measured in an operating room and have not been used previously to assess performance.


   Three sets of performance metrics were assessed utilizing the NeuroTouch platform in six scenarios with simulated brain tumors having different visual and tactile characteristics. Tier 1 metrics included percentage of brain tumor resected and volume of simulated “normal” brain tissue removed. Tier 2 metrics included instrument tip path length, time taken to resect the brain tumor, pedal activation frequency, and sum of applied forces. Tier 3 metrics included sum of forces applied to different tumor regions and the force bandwidth derived from the force histogram.


   The results outlined are from a novice resident in the second year of training and an expert neurosurgeon. The three tiers of metrics obtained from the NeuroTouch simulator do encompass the wide variability of technical performance observed during novice/expert resections of simulated brain tumors and can be employed to quantify the safety, quality, and efficiency of technical performance during simulated brain tumor resection. Tier 3 metrics derived from force pyramids and force histograms may be particularly useful in assessing simulated brain tumor resections.


   Our pilot study demonstrates that the safety, quality, and efficiency of novice and expert operators can be measured using metrics derived from the NeuroTouch platform, helping to understand how specific operator performance is dependent on both psychomotor ability and cognitive input during multiple virtual reality brain tumor resections.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13



National Research Council


Virtual reality


Postgraduate year


Cubic centimeters




  1. 1.

    Brydges R, Kurahashi A, Brümmer V, Satterthwaite L, Classen R, Dubrowski A (2008) Developing criteria for proficiency-based training of surgical technical skills using simulation: changes in performances as a function of training year. J Am Coll Surg 206:205–211. doi:10.1016/j.jamcollsurg.2007.07.045

  2. 2.

    Aggarwal R, Grantcharov TR, Eriksen JR, Blirup D, Kristiansen VB, Funch-Jensen P, Darzi A (2006) An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 244:310–314. doi:10.1097/01.sla.0000218094.92650.44

  3. 3.

    Ericsson KA, Charness N (1994) Expert performance. Am Psychol 49:725–747

  4. 4.

    Tsuda S, Scott D, Doyle J, Jones DB (2009) Surgical skills training and simulation. Curr Probl Surg 46:271–370. doi:10.1067/j.cpsurg.2008.12.003

  5. 5.

    Jaffer A, Bednarz B, Challacombe B, Sriprasad S (2009) The assessment of surgical competency in the UK. Int J Surg 7:12–15. doi:10.1016/j.ijsu.2008.10.006

  6. 6.

    Reznick RK, MacRae H (2006) Teaching surgical skills—changes in the wind. N Eng J Med 355:2664–2669. doi:10.1056/NEJMra054785

  7. 7.

    Brightwell A, Grant J (2013) Competency-based training: who benefits? Postgrad Med J 89:107–110. doi:10.1136/postgradmedj-2012-130881

  8. 8.

    Bhatti NI, Cummings CW (2007) Competency in surgical residency training: defining and raising the bar. Acad Med 82:569–573

  9. 9.

    Gélinas-Phaneuf N, Del Maestro RF (2013) Surgical expertise in neurosurgery integrating theory into practice. Neurosurgery 73(Suppl. 1):30–38. doi:10.1227/NEU.0000000000000115

  10. 10.

    Gaba DM, DeAnda A (1998) A comprehensive anesthesia simulation environment: re-creating the operating room for research and training. Anesthesiology 69:387–394

  11. 11.

    Chung J, Sackier J (1998) A method of objectively evaluating improvements in laparoscopic skills. Surg Endosc 12:1111–1116

  12. 12.

    Derossis AM, Bothwell J, Sigman HH, Fried GM (1998) The effect of practice on performance in a laparoscopic simulator. Surg Endosc 12:1117–1120

  13. 13.

    Delp SL, Loan JP, Hoy MG, Zajac FE, Topp EL, Rosen JM (1990) An interactive graphics-based model of the lower extremity to study orthopaedic surgical procedures. IEEE Trans Biomed Eng 37:757–767

  14. 14.

    Satava RM (1993) Virtual reality surgical simulator. Surg Endosc 7:203–205

  15. 15.

    Ahlberg G, Enochsson L, Gallagher AG, Hedman L, Hogman C, McClusky DA 3rd, Ramel S, Smith CD, Arvidsson D (2007) Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg 193:797–804. doi:10.1016/j.amjsurg.2006.06.050

  16. 16.

    Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM, Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236:458–463. doi:10.1097/01.SLA.0000028969.51489

  17. 17.

    Larsen CR, Soerensen JL, Grantcharov TP, Dalsgaard T, Schouenborg L, Ottosen C, Schroeder TV, Ottesen BS (2009) Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. BMJ 338:b1802. doi:10.1136/bmj.b1802

  18. 18.

    Phillips NI, John NW (2000) Web-based surgical simulation for ventricular catheterization. Neurosurgery 46:933–937

  19. 19.

    Ferroli P, Tringali G, Acerbi F, Schiariti M, Broggi M, Aquino D, Broggi G (2013) Advanced 3-dimensional planning in neurosurgery. Neurosurgery 72(Suppl 1):54–62. doi:10.1227/NEU.0b013e3182748ee8

  20. 20.

    Neubauer A, Wolfsberger S (2013) Virtual endoscopy in neurosurgery: a review. Neurosurgery 72(Suppl 1):97–106. doi:10.1227/NEU.0b013e31827393c9

  21. 21.

    Mitha AP, Almekhlafi MA, Janjua MJ, Albuquerque FC, McDougall CG (2013) Simulation and augmented reality in endovascular neurosurgery: lessons from aviation. Neurosurgery 72(Suppl 1):107–114. doi:10.1227/NEU.0b013e31827981fd

  22. 22.

    Rousseau G, Bailes J, del Maestro RF, Cabral A, Choudhury N, Comas O, Debergue P, De Luca G, Hovdebo J, Jiang D, Laroche D, Neubauer A, Pazos V, Thibault T, DiRaddo R (2013) The development of a virtual simulator for training of neurosurgeons to perform and to perfect endoscopic endonasal transphenoidal surgery. Neurosurgery 73(Suppl. 1):85–93. doi:10.1227/NEU.0000000000000112

  23. 23.

    Kockro RA, Serra L, Tseng-Tsai Y, Yih-Yian Chan CS, Gim-Guan C, Lee E, Hoe LY, Hern N, Nowinski WL (2000) Planning and simulation of neurosurgery in a virtual reality environment. Neurosurgery 46:118–137

  24. 24.

    Luciano C, Banerjee P, Lemole G, Charbel F (2006) Second generation haptic ventriculostomy simulator using the immersive touch system. Stud Health Technol Inform 119:343–348

  25. 25.

    Banerjee PP, Luciano CJ, Lemole GM Jr, Charbel FT, Oh MY (2007) Accuracy of ventriculostomy catheter placement using a head- and hand-tracked high-resolution virtual reality simulator with haptic feedback. J Neurosurg 107:515–521. doi:10.3171/JNS-07/09/0515

  26. 26.

    Luciano CJ, Banerjee PP, Bellotte B, Oh GM, Lemole M, Jr., Charbel FT, Roitberg B (2011) Learning retention of thoracic pedicle screw placement using a high-resolution augmented reality simulator with haptic feedback. Neurosurgery 69(1 Suppl Operative):ons14–19. doi:10.1227/NEU.0b013e31821954ed

  27. 27.

    Luciano CJ, Banerjee PP, Florea L, Dawe G (2005) Design of the ImmersiveTouch\(^{TM}\): a high-performance haptic augmented virtual reality system. In: 11th International conference on human-computer interaction, Las Vegas, NV

  28. 28.

    Delorme S, Laroche D, Diraddo R, F. Del Maestro R (2012) NeuroTouch: a Physics-based virtual simulator for cranial microneurosurgery training. Neurosurgery 71(1 Suppl Operative):ons 32–42. doi:10.1227/NEU.0b013e318249c744

  29. 29.

    Choudhury N, Gelinas-Phaneuf N, Delorme S, Del Maestro R (2013) Fundamentals of Neurosurgery: virtual reality tasks for training and evaluation of technical skills. World Neurosurg 80:E9–E19. doi:10.1016/j.wneu.2012.08.022

  30. 30.

    Gélinas-Phaneuf N, Choudhury N, Al-Habib AR, Cabral A, Nadeau E, Mora V, Debergue Pazos VP, Diraddo R, Del Maestro RF (2013) Assessing performance in brain tumor resection using a novel virtual reality simulator. Int J CARS. doi:10.1007/s11548-013-0905-8

  31. 31.

    Wheatstone C (1838) Contributions to the physiology of vision.-Part the first. On some remarkable, and hitherto unobserved, phenomena of binocular vision. Philos Trans R Soc Lond 128:371–394

  32. 32.

    Kohn LT, Corrigan J, Donaldson MS (2000) To err is human: building a safer health care system. National Academy Press, Washington

  33. 33.

    Gray JD (1996) Global rating scale in residency education. Acad Med 71:555–563

  34. 34.

    Turnbull J, Gray J, MacFadyen J (1998) Improving in training evaluation programs. J Gen Intern Med 13:317–323. doi:10.1046/j.1525-1497.1998.00097.x

  35. 35.

    Katz R, Hoznek A, Salomon L, Antiphon P, de la Taille A, Abbou CC (2005) Skill assessment of urological laparoscopic surgeons: can criterion levels of surgical performance be determined using the pelvic box trainer? Eur Urol 47:482–487. doi:10.1016/j.eururo.2004.12.007

  36. 36.

    de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA (2008) The incidence and nature of in-hospital adverse events: a systemic review. Qual Saf Health Care 17:216–23. doi:10.1136/qshc.2007.023622

  37. 37.

    Gawande AA, Thomas EJ, Zinner MJ, Brennan TA (1999) The incidence and nature of surgical adverse events in Colorado and Utah in 1992. Surgery 126:66–75

  38. 38.

    Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ (2013) Surgical skills and complication rates after bariatric surgery. N Engl J Med 369:1434–1442. doi:10.1056/NEJMsa1300625

  39. 39.

    Kockro RA (2013) Neurosurgery simulators-beyond the experiment. World Neurosurg. 80:E101–E102. doi:10.1016/j.wneu.2013.02.017

Download references


This work was supported by The Di Giovanni Foundation, the Montreal English School Board, the B-Strong Foundation, the Colannini Foundation, and the Montreal Neurological Institute and Hospital. Dr. R. F. Del Maestro is the William Feindel Emeritus Professor in Neuro-Oncology at McGill University. Dr. Azarnoush held the Postdoctoral Neuro-Oncology Fellowship from the Montreal Neurological Institute and Hospital. Dr. Gélinas-Phaneuf was funded by a generous contribution from the Harold and Audrey Fisher Brain Tumour Research Award.

Author information

Correspondence to Hamed Azarnoush.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (wmv 4794 KB)

Supplementary material 2 (avi 7285 KB)

Supplementary material 1 (wmv 4794 KB)

Supplementary material 2 (avi 7285 KB)

Supplementary material 3 (avi 5134 KB)

Supplementary material 4 (doc 10 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Azarnoush, H., Alzhrani, G., Winkler-Schwartz, A. et al. Neurosurgical virtual reality simulation metrics to assess psychomotor skills during brain tumor resection. Int J CARS 10, 603–618 (2015).

Download citation


  • Performance metrics
  • Virtual reality neurosurgical simulation
  • Haptic feedback
  • Brain tumor resection
  • Neurosurgical oncology
  • NeuroTouch