Intelligent Support for Surgeons in the Operating Room

  • Rainer MalakaEmail author
  • Frank Dylla
  • Christian Freksa
  • Thomas Barkowsky
  • Marc Herrlich
  • Ron Kikinis


Modern technology gives surgeons the possibility to plan operations using complex 3D information tools providing data integration, analysis and visualization. However, in the operating room, most of the tools are not at hand. It might be useful to access the data and to visualize certain surgical procedures during the actual surgery. We investigate such situations and look for novel solutions for intra-operative support for surgeons to access 3D information: what they need, when they need it. We integrate medical image processing, cognitive modeling and human- computer interaction in order to anticipate the surgeons’ needs. We address three issues for developing such systems: how to identify what information the surgeon needs; how to adapt pre- and intra-procedure information to the surgical situation; how to present the relevant information to the surgeon. This paper presents the vision and preliminary results of a collaborative research project.


Human-computer interaction Surgery Cognitive systems 3D medical data 



The research reported on in this contribution is funded within the Creative Unit “Intra-Operative Information: What Surgeons Need, When They Need It” in the context of the Excellence Initiative at the University of Bremen.


  1. 1.
    Schenk, A., Haemmerich, D., Preusser, T.: Planning of image-guided interventions in the Liver. IEEE Pulse 2(5), 48–55 (2011)CrossRefGoogle Scholar
  2. 2.
    Peterhans, M., Oliveira, T., Banz, V., Candinas, D., Weber, S.: Computer-assisted liver surgery: clinical applications and technological trends. Crit. Rev. Biomed. Eng. 40(3), 199–220 (2012)CrossRefGoogle Scholar
  3. 3.
    Heizmann, O., Zidowitz, S., Bourquain, H., Potthast, S., Peitgen, H.-O., Oertli, D., Kettelhack, C.: Assessment of intraoperative liver deformation: prospective clinical Study. World J. Surg. 34(8), 1887–1893 (2010)CrossRefGoogle Scholar
  4. 4.
    Nadin, M.: Anticipation and dynamics: Rosen’s anticipation in the perspective of time. Int. J. Gen. Syst. 39(1), 3–33 (2010)CrossRefzbMATHGoogle Scholar
  5. 5.
    Nadin, M.: Not everything we know we learned. In: Butz, M. (ed.) Adaptive behavior in anticipatory learning systems. LNAI 2684, pp. 23–43. BertelsmanSpringer, Heidelberg (2003)CrossRefGoogle Scholar
  6. 6.
    Schenk, A., Köhn, A., Matsuyama, R., Endo, I.: Transfer of liver surgery planning into the operating room: Initial experience with the iPad. 10th Congress European-African Hepato Pancreato Biliary Association (E-AHPBA), Belgrade, Serbia (2013)Google Scholar
  7. 7.
    O’Hara, K., Gonzalez, G., Sellen, A., Penney, G., Varnavas, A., Mentis, H., Criminisi, A., Corish, R., Rouncefield, M., Dastur, N., Carrel, T.: Touchless interaction in surgery. Commun. ACM 57(1), 70–77 (2014)CrossRefGoogle Scholar
  8. 8.
    Sweller, J.: Instructional design in technical areas. Camberwell, Australian Council for Educational Research (1999)Google Scholar
  9. 9.
    Dix, A., Finlay, J., Abowd, G., Beale, R.: Human Computer Interaction, 3rd edn. Pearson (2003)Google Scholar
  10. 10.
    Hassenzahl, M., Tractinsky, N.: User experience-a research agenda. Behav. Inf. Technol. 25(2), 91–97 (2006)CrossRefGoogle Scholar
  11. 11.
    Anderson, J.R.: Cognitive Psychology and its Implications. Worth Publishers (2009)Google Scholar
  12. 12.
    Kahneman, D.: Attention and Effort. Prentice-Hall, New Jersey (1973)Google Scholar
  13. 13.
    Richardson, A.: Verbalizer-visualizer: a cognitive style dimension. J. Mental Imagery 1, 109–126 (1977)Google Scholar
  14. 14.
    Kozhevnikov, M., Kosslyn, S.M., Shephard, J.: Spatial versus object visualizers: A new characterization of visual cognitive style. Mem. Cogn. 44(3), 710–726 (2005)CrossRefGoogle Scholar
  15. 15.
    Elting, C., Zwickel, J., Malaka, R.: What are multimodalities made of? Modeling output in a multimodal dialogue system. In: International Conference on Intelligent User Interfaces IUI, pp. 13–16 (2002)Google Scholar
  16. 16.
    Porzel, R., Scheffler, A., Malaka, R.: How entrainment increases dialogical efficiency. In Proceedings of Workshop on Effective Multimodal Dialogue Interfaces at IUI 2006, Sydney (2006)Google Scholar
  17. 17.
    Gurevych, I., Porzel, R., Malaka, R.: Modeling domain knowledge: Know-how and know-what. In: SmartKom: Foundations of Multimodal Dialogue Systems, pp. 71–84. Springer, Berlin (2006)Google Scholar
  18. 18.
    Porzel, R., Gurevych, I., Malaka, R.: In context: Integrating domain-and situation-specific knowledge. In: SmartKom: Foundations of Multimodal Dialogue Systems, pp. 269–84. Springer, Berlin (2006)Google Scholar
  19. 19.
    Porzel, R., Zorn, H.-P., Loos, B., Malaka, R.: Towards a separation of pragmatic knowledge and contextual information. In: Proceedings of ECAI-06 Workshop on Contexts and Ontologies (2006)Google Scholar
  20. 20.
    Malaka, R.: Intelligent User Interfaces for Ubiquitous Computing. In: Handbook of Research on Ubiquitous Computing Technology for Real Time Enterprises, pp. 470–486 (2008)Google Scholar
  21. 21.
    Renz, J., Rauh, R., Knauff, M.: Towards Cognitive Adequacy of Topological Spatial Relations. In: Freksa, C., Brauer, W., Habel, C., Wender, K.F. (eds.) Spatial Cognition II, Integrating Abstract Theories, Empirical Studies, Formal Methods, and Practical Applications, pp. 184–197. Springer, London, UK, (2000)Google Scholar
  22. 22.
    Freksa, C.: Qualitative spatial reasoning. In: Mark, D.M., Frank, A. (eds.) Cognitive and Linguistic Aspects of Geographic Space, pp. 361–372. Kluwer, Dordrecht (1991)CrossRefGoogle Scholar
  23. 23.
    Dylla, F., Mossakowski, T., Schneider, T., Wolter, D.: Algebraic properties of qualitative spatio-temporal calculi. In: Conference On Spatial Information Theory (COSIT- 13). North Yorkshire, UK, Springer (2013)Google Scholar
  24. 24.
    Cohn, A. G., Renz, J.: Qualitative spatial reasoning. In: Harmelen, F., Lifschitz, V., Porter B. (eds.) Handbook of Knowledge Representation. Elsevier (2007)Google Scholar
  25. 25.
    Ligozat, G.: Qualitative Spatial and Temporal Reasoning. Wiley-ISTE (2011)Google Scholar
  26. 26.
    Walther-Franks, B., Herrlich, M., Malaka, R.: A multi-touch system for 3D modelling and animation. In: Smart Graphics. vol. 6815 of Lecture Notes in Computer Science, Chap. 5, pp. 48–59. Springer, Berlin (2011)Google Scholar
  27. 27.
    Walther-Franks, B., Malaka, R.: An interaction approach to computer animation. Entertainment Comput. 5(4), 271–283 (2014)CrossRefGoogle Scholar
  28. 28.
    Freksa, C.: Understanding cognition through synthesis and analysis. Int. J. Softw. Inform. 7(1), 3–18 (2013)Google Scholar
  29. 29.
    Herrlich, M., Benker, J., Black, D., Dylla, F., Malaka, R.: Tool-mounted ring displays for intraoperative navigation. In: Proceedings of CURAC 2015, Bremen, Germany (2015)Google Scholar
  30. 30.
    Fitzke, T., Krail, N., Kroll, F., Ohlrogge, L., Schröder, F., Spillner, L. Voll, A., Dylla, F., Herrlich, M., Malaka, R.: Fußbasierte interaktionen mit computersystemen im operationssaal. In: Proceedings of CURAC 2015, Bremen, Germany (2015)Google Scholar
  31. 31.
    Ritter, F., Porzel, R., Mostajeran Gourtani, F., Malaka, R.: Brain-computer-interface for software control in medical interventions. In: Proceedings of CURAC 2015, Bremen, Germany (2015)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • Rainer Malaka
    • 1
    • 4
    Email author
  • Frank Dylla
    • 2
    • 4
  • Christian Freksa
    • 2
    • 4
  • Thomas Barkowsky
    • 2
    • 4
  • Marc Herrlich
    • 1
    • 4
  • Ron Kikinis
    • 3
    • 4
    • 5
    • 6
  1. 1.TZI, Digital Media LabUniversity of BremenBremenGermany
  2. 2.Cognitive SystemsUniversity of BremenBremenGermany
  3. 3.Medical Image ComputingUniversity of BremenBremenGermany
  4. 4.Creative Unit Intra-Operative InformationUniversity of BremenBremenGermany
  5. 5.Surgical Planning LaboratoryBrigham and Women’s Hospital and Harvard Medical SchoolBostonUSA
  6. 6.Fraunhofer MEVISBremenGermany

Personalised recommendations