Mixed reality ultrasound guidance system: a case study in system development and a cautionary tale

  • Golafsoun Ameri
  • John S. H. Baxter
  • Daniel Bainbridge
  • Terry M. Peters
  • Elvis C. S. Chen
Original Article



Real-time ultrasound has become a crucial aspect of several image-guided interventions. One of the main constraints of such an approach is the difficulty in interpretability of the limited field of view of the image, a problem that has recently been addressed using mixed reality, such as augmented reality and augmented virtuality. The growing popularity and maturity of mixed reality has led to a series of informal guidelines to direct development of new systems and to facilitate regulatory approval. However, the goals of mixed reality image guidance systems and the guidelines for their development have not been thoroughly discussed. The purpose of this paper is to identify and critically examine development guidelines in the context of a mixed reality ultrasound guidance system through a case study.


A mixed reality ultrasound guidance system tailored to central line insertions was developed in close collaboration with an expert user. This system outperformed ultrasound-only guidance in a novice user study and has obtained clearance for clinical use in humans. A phantom study with 25 experienced physicians was carried out to compare the performance of the mixed reality ultrasound system against conventional ultrasound-only guidance. Despite the previous promising results, there was no statistically significant difference between the two systems.


Guidelines for developing mixed reality image guidance systems cannot be applied indiscriminately. Each design decision, no matter how well justified, should be the subject of scientific and technical investigation. Iterative and small-scale evaluation can readily unearth issues and previously unknown or implicit system requirements.


We recommend a wary eye in development of mixed reality ultrasound image guidance systems emphasizing small-scale iterative evaluation alongside system development. Ultimately, we recommend that the image-guided intervention community furthers and deepens this discussion into best practices in developing image-guided interventions.


Ultrasound guidance Mixed reality Augmented reality Augmented virtuality System development Guidelines Internal jugular vein catheterization Central line 



The authors would like to acknowledge Jonathan McLeod, John Moore, and Adam Rankin for their invaluable discussion. This research was funded by The Canadian Foundation for Innovation, # 20994, The Natural Sciences and Engineering Council of Canada, # RPGIN 2014-04504, and The Canadian Institutes for Health Research, # FDN 143232. John S. H. Baxter is supported by the Natural Sciences and Engineering Research Council of Canada through the Canadian Graduate Scholarship program.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. 1.
    Abhari K, Baxter JS, Chen ECS, Khan AR, Peters TM, de Ribaupierre S, Eagleson R (2015) Training for planning tumour resection: augmented reality and human factors. IEEE Trans Biomed Eng 62(6):1466–1477CrossRefPubMedGoogle Scholar
  2. 2.
    Akoglu H, Piskinpasa S, Yenigun EC, Ozturk R, Dede F, Odabas AR (2012) Real-time ultrasound guided placement of temporary internal jugular vein catheters: assessment of technical success and complication rates in nephrology practice. Nephrology 17(7):603–606CrossRefPubMedGoogle Scholar
  3. 3.
    Ameri G, Baxter JS, McLeod AJ, Peters TM, Chen ECS (2015) Augmented reality ultrasound guidance for central line procedures: preliminary results. In: Workshop on augmented environments for computer-assisted interventions, Springer, pp 11–20Google Scholar
  4. 4.
    Bannon MP, Heller SF, Rivera M (2011) Anatomic considerations for central venous cannulation. Risk Manag Healthc Policy 4:27CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Baumann M, Mozer P, Daanen V, Troccaz J (2009) Prostate biopsy assistance system with gland deformation estimation for enhanced precision. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 67–74Google Scholar
  6. 6.
    Bax J, Cool D, Gardi L, Knight K, Smith D, Montreuil J, Sherebrin S, Romagnoli C, Fenster A (2008) Mechanically assisted 3D ultrasound guided prostate biopsy system. Med Phys 35(12):5397–5410CrossRefPubMedGoogle Scholar
  7. 7.
    Bichlmeier C, Wimmer F, Heining SM, Navab N (2007) Contextual anatomic mimesis hybrid in-situ visualization method for improving multi-sensory depth perception in medical augmented reality. In: 6th IEEE and ACM international symposium on mixed and augmented reality, ISMAR 2007, IEEE, pp 129–138Google Scholar
  8. 8.
    Blaivas M, Adhikari S (2009) An unseen danger: frequency of posterior vessel wall penetration by needles during attempts to place internal jugular vein central catheters using ultrasound guidance. Crit Care Med 37(8):2345–2349CrossRefPubMedGoogle Scholar
  9. 9.
    Bowdle A (2014) Vascular complications of central venous catheter placement: evidence-based methods for prevention and treatment. J Cardiothorac Vasc Anesth 28(2):358–368CrossRefPubMedGoogle Scholar
  10. 10.
    Cavanna L, Civardi G, Mordenti P, Vallisa D, Bert R, Di Nunzio C (2013) Central venous catheter care for the patients with cancer: ultrasound-guided insertion should be strongly recommended for internal jugular vein catheterization. Ann Oncol 24(11):2928–2929CrossRefPubMedGoogle Scholar
  11. 11.
    Chandler WF, Knake JE, McGillicuddy JE, Lillehei KO, Silver TM (1982) Intraoperative use of real-time ultrasonography in neurosurgery. J Neurosurg 57(2):157–163CrossRefPubMedGoogle Scholar
  12. 12.
    Chen ECS, Peters TM, Ma B (2016) Guided ultrasound calibration: where, how, and how many calibration fiducials. Int J Comput Assist Radiol Surg 11(6):889–898CrossRefPubMedGoogle Scholar
  13. 13.
    Commission IE (2006) Medical device software–software life cycle processes. IEC Standard 62304Google Scholar
  14. 14.
    Currie M, McLeod J, Patel R, Peters T, Kiaii B (2015) Augmented reality system for ultrasound guidance of transcatheter aortic valve replacement. Can J Cardiol 31(10):S182CrossRefGoogle Scholar
  15. 15.
    Denys BG, Uretsky BF (1991) Anatomical variations of internal jugular vein location: impact on central venous access. Crit Care Med 19(12):1516–1519CrossRefPubMedGoogle Scholar
  16. 16.
    Ender J, Končar-Zeh J, Mukherjee C, Jacobs S, Borger MA, Viola C, Gessat M, Fassl J, Mohr FW, Falk V (2008) Value of augmented reality-enhanced transesophageal echocardiography (TEE) for determining optimal annuloplasty ring size during mitral valve repair. Ann Thorac Surg 86(5):1473–1478CrossRefPubMedGoogle Scholar
  17. 17.
    Horn BK (1987) Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am 4:629–642CrossRefGoogle Scholar
  18. 18.
    Hung J, Lang R, Flachskampf F, Shernan SK, McCulloch ML, Adams DB, Thomas J, Vannan M, Ryan T (2007) 3d echocardiography: a review of the current status and future directions. J Am Soc Echocardiogr 20(3):213–233CrossRefPubMedGoogle Scholar
  19. 19.
    Jakola AS, Reinertsen I, Selbekk T, Solheim O, Lindseth F, Gulati S, Unsgård G (2014) Three-dimensional ultrasound-guided placement of ventricular catheters. World Neurosurg 82(3):536-e5CrossRefGoogle Scholar
  20. 20.
    Jenssen C, Brkljacic B, Hocke M, Ignee A, Piscaglia F, Radzina M, Sidhu PS, Dietrich CF (2015) EFSUMB guidelines on interventional ultrasound (INVUS), part vi. Ultraschall Med Eur J Ultrasound 36:E1–E14Google Scholar
  21. 21.
    Khaykin Y, Klemm O, Verma A (2007) First human experience with real-time integration of intracardiac echocardiography and 3D electroanatomical imaging to guide right free wall accessory pathway ablation. Europace 10(1):116–117CrossRefPubMedGoogle Scholar
  22. 22.
    Linte CA, Davenport KP, Cleary K, Peters C, Vosburgh KG, Navab N, Jannin P, Peters TM, Holmes DR, Robb RA (2013) On mixed reality environments for minimally invasive therapy guidance: systems architecture, successes and challenges in their implementation from laboratory to clinic. Comput Med Imaging Gr 37(2):83–97CrossRefGoogle Scholar
  23. 23.
    Linte CA, Moore J, Wiles AD, Wedlake C, Peters TM (2008) Virtual reality-enhanced ultrasound guidance: a novel technique for intracardiac interventions. Comput Aided Surg 13(2):82–94CrossRefPubMedGoogle Scholar
  24. 24.
    Martin C, Eon B, Auffray JP, Saux P, Gouin F (1990) Axillary or internal jugular central venous catheterization. Crit Care Med 18(4):400–402CrossRefPubMedGoogle Scholar
  25. 25.
    McGee DC, Gould MK (2003) Preventing complications of central venous catheterization. N Engl J Med 348(12):1123–1133CrossRefPubMedGoogle Scholar
  26. 26.
    McHugh M, McCaffery F, Casey V (2012) Barriers to adopting agile practices when developing medical device software. In: International conference on software process improvement and capability determination, Springer, pp 141–147Google Scholar
  27. 27.
    McHugh M, McCaffery F, Fitzgerald B, Stol KJ, Casey V, Coady G (2013) Balancing agility and discipline in a medical device software organisation. In: International conference on software process improvement and capability determination, Springer, pp 199–210Google Scholar
  28. 28.
    Milgram P, Takemura H, Utsumi A, Kishino F (1994) Augmented reality: a class of displays on the reality-virtuality continuum. Telemanipulator Telepresence Technol 2351:282–292CrossRefGoogle Scholar
  29. 29.
    Moore CL, Copel JA (2011) Point-of-care ultrasonography. N Engl J Med 364(8):749–757CrossRefPubMedGoogle Scholar
  30. 30.
    Moore JT, Chu MWA, Kiaii B, Bainbridge D, Guiraudon G, Wedlake C, Currie M, Rajchl M, Patel RV, Peters TM (2013) A navigation platform for guidance of beating heart transapical mitral valve repair. IEEE Trans Biomed Eng 60(4):1034–1040. doi: 10.1109/TBME.2012.2222405 CrossRefPubMedGoogle Scholar
  31. 31.
    Otto CM (2012) The practice of clinical echocardiography. Elsevier Health Sciences, AmsterdamGoogle Scholar
  32. 32.
    Paul RG, Price S (2014) Central venous cannulation. Medicine 42:473–474CrossRefGoogle Scholar
  33. 33.
    Paulk MC (2011) On empirical research into scrum adoption. Carnegie Mellon University, PittsburghGoogle Scholar
  34. 34.
    Peters TM, Linte CA (2016) Image-guided interventions and computer-integrated therapy: Quo vadis? Med Image Anal 33:56–63CrossRefPubMedGoogle Scholar
  35. 35.
    Pratola C, Baldo E, Artale P, Marcantoni L, Toselli T, Percoco G, Sassone B, Ferrari R (2011) Different image integration modalities to guide af ablation: impact on procedural and fluoroscopy times. Pacing Clin Electrophysiol 34(4):422–430CrossRefPubMedGoogle Scholar
  36. 36.
    Rottier PA, Rodrigues V (2008) Agile development in a medical device company. In: Conference on AGILE’08, IEEE, pp 218–223Google Scholar
  37. 37.
    Rust P, Flood D, McCaffery F (2016) Creation of an IEC 62304 compliant software development plan. J Softw Evolut Process 28(11):1005–1010CrossRefGoogle Scholar
  38. 38.
    Rust P, Flood D, McCaffery F (2016) Software process improvement roadmaps–using design patterns to aid SMEs developing medical device software in the implementation of IEC 62304. In: International conference on software process improvement and capability determination, Springer, pp 43–56Google Scholar
  39. 39.
    Sauer F, Vogt S, Khamene A (2008) Augmented reality. In: Peters T, Cleary K (eds) Image-guided interventions. Springer, Boston, pp 81–119Google Scholar
  40. 40.
    Schmidt GA, Maizel J, Slama M (2015) Ultrasound-guided central venous access: What’s new? Intensive Care Med 41(4):705–707CrossRefPubMedGoogle Scholar
  41. 41.
    Selbekk T, Jakola AS, Solheim O, Johansen TF, Lindseth F, Reinertsen I, Unsgård G (2013) Ultrasound imaging in neurosurgery: approaches to minimize surgically induced image artefacts for improved resection control. Acta Neurochir 155(6):973–980CrossRefPubMedPubMedCentralGoogle Scholar
  42. 42.
    Surry KJM, Austin HJB, Fenster A, Peters TM (2004) Poly(vinyl alcohol) cryogel phantoms for use in ultrasound and MR imaging. Phys Med Biol 49(24):5529–5546CrossRefPubMedGoogle Scholar
  43. 43.
    Troianos CA, Kuwik RJ, Pasqual JR, Lim AJ, Odasso DP (1996) Internal jugular vein and carotid artery anatomic relation as determined by ultrasonography. Anesthesiology 85(1):43–48Google Scholar
  44. 44.
    Unsgaard G, Rygh O, Selbekk T, Müller T, Kolstad F, Lindseth F, Hernes TN (2006) Intra-operative 3D ultrasound in neurosurgery. Acta Neurochir 148(3):235–253CrossRefPubMedGoogle Scholar
  45. 45.
    Van Krevelen D, Poelman R (2010) A survey of augmented reality technologies, applications and limitations. Int J Virtual Real 9(2):1Google Scholar
  46. 46.
    Wei Z, Wan G, Gardi L, Mills G, Downey D, Fenster A (2004) Robot-assisted 3D-trus guided prostate brachytherapy: system integration and validation. Med Phys 31(3):539–548CrossRefPubMedGoogle Scholar
  47. 47.
    Weiner MM, Geldard P, Mittnacht AJ (2013) Ultrasound-guided vascular access: a comprehensive review. J Cardiothorac Vasc Anesth 27(2):345–360CrossRefPubMedGoogle Scholar
  48. 48.
    Yaniv Z, Linte CA (2016) Applications of augmented reality in the operating room. In: Barfield W (ed) Fundamentals of wearable computers and augmented reality. CRC Press, USA, pp 485–518Google Scholar

Copyright information

© CARS 2017

Authors and Affiliations

  1. 1.Robarts Research InstituteWestern UniversityLondonCanada
  2. 2.London Health Sciences CentreLondonCanada

Personalised recommendations