Holographic Visualisation and Interaction of Fused CT, PET and MRI Volumetric Medical Imaging Data Using Dedicated Remote GPGPU Ray Casting
Medical experts commonly use imaging including Computed Tomography (CT), Positron-Emission Tomography (PET) and Magnetic Resonance Imaging (MRI) for diagnosis or to plan a surgery. These scans give a highly detailed representation of the patient anatomy, but the usual Three-Dimensional (3D) separate visualisations on screens does not provide an convenient and performant understanding of the real anatomical complexity. This paper presents a computer architecture allowing medical staff to visualise and interact in real-time holographic fused CT, PET, MRI of patients. A dedicated workstation with a wireless connection enables real-time General-Purpose Processing on Graphics Processing Units (GPGPU) ray casting computation through the mixed reality (MR) headset. The hologram can be manipulated with hand gestures and voice commands through the following interaction features: instantaneous visualisation and manipulation of 3D scans with a frame rate of 30 fps and a delay lower than 120 ms. These performances give a seamless interactive experience for the user .
KeywordsAugmented and mixed reality Medical application Medical visualisation MRI scan PET scan CT scan GPGPU ray casting HoloLens Hologram
- 1.Bach, B., Sicat, R., Beyer, J., Cordeil, M., Pfister, H.: The hologram in my hand: how effective is interactive exploration of 3D visualizations in immersive tangible augmented reality? IEEE TVCG 24(1), 457–467 (2018)Google Scholar
- 4.Egger, J., et al.: HTC Vive MeVisLab integration via OpenVR for med. app. PLOS ONE 12(3), 1–14 (2017)Google Scholar
- 8.Morley, C., Choudhry, O., Kelly, S., Phillips, J., Ahmed, F.. In: SIIM Scientific Session: Poster & Demostrations (2017)Google Scholar
- 9.Qian, L., et al.: Technical Note: Towards Virtual Monitors for Image Guided Interventions - Real-time Streaming to Optical See-Through Head-Mounted Displays (2017)Google Scholar
- 10.Raaen, K., Kjellmo, I.: Measuring latency in VR systems, pp. 457–462 (2015)Google Scholar