A Framework to Visualize and Interact with Multimodal Medical Images

  • Isabel Manssour
  • Sérgio Furuie
  • Luciana Nedel
  • Carla Freitas
Part of the Eurographics book series (EUROGRAPH)


The simultaneous use of images obtained from different sources is common in medical diagnosis. However, even though the quality of these images has been improving, the integration of multimodality data into a unique 3D representation is still non-trivial. To overcome this problem, multimodal visualization techniques provide better insight by fording suitable strategies to integrate different characteristics of multiple data sets into a single visual representation. This paper describes a framework for interactive multimodal visualization of 3D medical images, focusing on the multimodal visualization model and requirements for developing such systems. A short overview of multimodal visualization systems and techniques is also presented.


Mutual Information Unify Modeling Language View Class Graphic Hardware Volume Visualization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    J. B. A. Maintz and M. A. Viergever, Survey of Medical Image Registration. Medical Image Analysis, 2 (1), 1998, 1–36.CrossRefGoogle Scholar
  2. 2.
    J. P. W. Pluim and J. B. A. Maintz, Registration of Multimodal medical images. Available at 2000 ).
  3. 3.
    G. Q. Maguire Jr., M. E. Noz, H. Rusinek et al. Graphics Applied to Medical Image Registration. IEEE Computer Graphics & Applications, March, 1991, 20–28.Google Scholar
  4. 4.
    R. Stokking, Integrated Visualization of Functional and Anatomical Brain Images(Netherlands: Universiteit Utrecht, 1998, Ph.D. Thesis).Google Scholar
  5. 5.
    R. A. Robb, Visualization Methods for Analysis of Multimodality Images. In: R. W. Thatcher, M. Hallet, T. Zeffiro et al. (eds). Functional Neuroimaging: Technical Foundations. Academic Press, 1994.Google Scholar
  6. 6.
    Computer Graphics and Image Processing Group, VPat - Visualization and interaction with Virtual Patients. Available at (July 2000).
  7. 7.
    M. R. M. Silva, Ray Casting Based Volume Visualization Techniques(Porto Alegre: Instituto de Informatica/UFRGS, 2000, Master Thesis).Google Scholar
  8. 8.
    L. G. Brown. A Survey of Image Registration Techniques. ACM Computing Surveys, 24 (4), 1992, 325–376.CrossRefGoogle Scholar
  9. 9.
    J. K. Udupa and G. T. Herman, 3D Imaging in Medicine (CRC Press LLC, 2000, 2th ed.).Google Scholar
  10. 10.
    K. J. Zuiderveld, A. H. J. Koning, R. Stokking, et al. Multimodality Visualization of Medical Volume Data. Computers&Graphics, 20 (6), 1996, 775–791.Google Scholar
  11. 11.
    A. Collignon, Multi-Modality Medical Image Registration by Maximization of Mutual Information(Belgium: Katholieke Universiteit Leuven, 1998, Ph.D. Thesis).Google Scholar
  12. 12.
    F. Maes, Segmentation and Registration of Multimodal Medical Images(Belgium: Katholieke Universiteit Leuven, 1998, Ph.D. Thesis).Google Scholar
  13. 13.
    J. B. A. Maintz, Retrospective Registration of Tomographic Brain Images(Netherlands: Universiteit Utrecht, 1996, Ph.D. Thesis).Google Scholar
  14. 14.
    A. Collignon, F. Maes, D. Delaere, et al. Automated multi-modality image registration based on information theory. Information Processing in Medical Imaging. Kluwer Academic Publishers, 1995, p. 263–274.Google Scholar
  15. 15.
    P. Viola and W. M. Wells III, Alignment by maximization of mutual information. In Proceedings of International Conference On Computer Vision, pages 16–23. IEEE Computer Society Press, 1995.Google Scholar
  16. 16.
    D. N. Levin, X. Hu, K. K. Tan et al. The Brain: Integrated Three-Dimensional Display of MRI and PET Images. Radiology, 172 (3), 1989, 783–789.Google Scholar
  17. 17.
    K. J. Zuiderveld, Visualization of Multimodality Medical Volume Data using Object-Oriented Methods(Netherlands: Universiteit Utrecht, 1995. Ph.D. Thesis).Google Scholar
  18. 18.
    W. Cai and G. Sakas. Data Intermixing and Multi-Volume Rendering. Computer Graphics Forum, Cambridge, 18(3), 1999, C359–C368. Proceedings EUROGRAPHICS, Milan, Italy.Google Scholar
  19. 19.
    M. Levoy, Efficient Ray Tracing of Volume Data, ACM Transactions on Graphics, 9 (3), 1990, 245–261.MATHCrossRefGoogle Scholar
  20. 20.
    H. J. Noordmans, Interactive Analysis of 3D Microscope Images(Netherlands: Universiteit Utrecht, 1997, Ph.D. Thesis).Google Scholar
  21. 21.
    B. Lichtenbelt, R. Crane and S. Naqvi, Introduction to Volume Rendering(Prentice Hall, 1998 ).Google Scholar
  22. 22.
    P. Hastreiter and T. Ertl, Integrated Registration and Visualization of Medical Image Data. In Proceeding of Computer Graphics International’98, (Hannover, Germany, 1998), pages 78–85. IEEE Computer Society Press, 1998.Google Scholar
  23. 23.
    C. Rezk-Salama, K. Engel, M. Bauer et al. Interactive Volume Rendering on Standard PC Graphics Hardware Using Multi-Textures and Multi-Stage Rasterization. In Proceedings of SIGGraph/Eurographics Workshop on Graphics Hardware 2000(Interlaken, Switzerland, August 21–22, 2000 ).Google Scholar
  24. 24.
    F. Buschmann, R. Meunier, H. Rohnert et al. Pattern-Oriented Software Architecture, A System of Patterns (John Wiley & Sons, 1996)Google Scholar
  25. 25.
    C. Larman, Applying UML and Patterns - An Introduction to Object-Oriented Analysis and Design(Prentice-Hall, 1997 ).Google Scholar
  26. 26.
    B. Spitzak, FLTK - The Fast Light Tool Kit Home Page. Available at (March 2000).
  27. 27.
    E. Gamma, R. Helm, R. Johnson et al. Design patterns: elements of reusable object-oriented software(Addison-Wesley, 1995 ).Google Scholar
  28. 28.
    R. Mullick, N. Bryan and J. Butman. Confocal Volume Rendering: Fast Segmentation-Free Visualization of Internal Structures. In Proceedings of SPIE 00–Annual Meeting International Symposium on Optical Science and Technology( San Diego, California, 2000 ), pages 144–154.Google Scholar
  29. 29.
    CPAD - Research Center in High Performance Computing. Available at (April 2001).
  30. 30.
    R. Buyya. High Performance Cluster Computing Architectures and System(Prentice Hall, 1999 ).Google Scholar
  31. 31.
    O. Sommer, A. Dietz, R. Westermann et al. An Interactive Visualization and Navigation Tool for Medical Volume Data. In Proceedings of WSCG’98(Plzen - Bory, Czech Republic, February 9–13, 1998 ).Google Scholar
  32. 32.
    R. A. Robb, Visualization Methods for Analysis of Multimodality Images. In: R. W. Thatcher, M. Hallet, T. Zeffiro et al. (eds). Functional Neuroimaging: Technical Foundations. San Diego, CA: Academic Press, 1994.Google Scholar
  33. 33.
    C. Johnson, S. G. Parker, C. Hansen et al. Interactive Simulation and Visualization. Computer, 32 (12), 1999, 59–65.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag/Wien 2001

Authors and Affiliations

  • Isabel Manssour
    • 1
    • 2
  • Sérgio Furuie
    • 3
  • Luciana Nedel
    • 1
    • 2
  • Carla Freitas
    • 1
  1. 1.UFRGS - Universidade Federal do Rio Grande do Sul, II/PGCCPorto AlegreBrazil
  2. 2.PUCRS - Pontifícia Universidade Católica do Rio Grande do Sul, FACINPorto AlegreBrazil
  3. 3.InCor - Instituto do CoraçãoHC-FMUSP, Unidade de Pesquisa e DesenvolvimentoSão PauloBrazil

Personalised recommendations