A Multimodal Multi-device Discourse and Dialogue Infrastructure for Collaborative Decision-Making in Medicine

  • Daniel Sonntag
  • Christian Schulz
Conference paper


The dialogue components we developed provide the infrastructure of the disseminated industrial prototype RadSpeech—a semantic speech dialogue system for radiologists. The major contribution of this paper is the description of a new speech-based interaction scenario of RadSpeech where two radiologists use two independent but related mobile speech devices (iPad and iPhone) and collaborate via a connected large screen installation using related speech commands. With traditional user interfaces, users may browse or explore patient data, but little to no help is given when it comes to structuring the collaborative user input and annotate radiology images in real-time with ontology-based medical annotations. A distinctive feature is that the interaction design includes the screens of the mobile devices for touch screen interaction for more complex tasks rather than the simpler ones such as a mere remote control of the image display on the large screen.


Mobile Device Image Annotation Dialogue System Dialogue Session Type Feature Structure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Fensel, D., Hendler, J.A., Lieberman, H., Wahlster, W. (eds.): Spinning the Semantic Web: Bringing the World Wide Web to Its Full Potential. MIT Press, Cambridge (2003)Google Scholar
  2. 2.
    Hall, F.M.: The radiology report of the future. Radiology 251(2), 313–316 (2009)CrossRefGoogle Scholar
  3. 3.
    Mejino, J.L., Rubin, D.L., Brinkley, J.F.: FMA-RadLex: An application ontology of radiological anatomy derived from the foundational model of anatomy reference ontology. In: Proceedings of AMIA Symposium, pp. 465–469, 2008.
  4. 4.
    Möller, M., Ernst, P., Dengel, A., Sonntag, D.: Representing the international classification of diseases version 10 in OWL. In: Proceedings of the International Conference on Knowledge Engineering and Ontology Development (KEOD), Valencia, Spain, 2010Google Scholar
  5. 5.
    Oviatt, S.: Ten myths of multimodal interaction. Commun. ACM 42(11), 74–81 (1999). Google Scholar
  6. 6.
    Pfleger, N.: FADE: an integrated approach to multimodal fusion and discourse processing. In: Proceedings of the Dotoral Spotlight at ICMI 2005, Trento, Italy, 2005Google Scholar
  7. 7.
    Pieraccini, R., Huerta, J.: Where do we go from here? research and commercial spoken dialog systems. In: Proceedings of the 6th SIGDdial Worlshop on Discourse and Dialogue, pp. 1–10, 2005Google Scholar
  8. 8.
    Reithinger, N., Fedeler, D., Kumar, A., Lauer, C., Pecourt, E., Romary, L.: MIAMM: A multimodal dialogue system using haptics. In: van Kuppevelt, J., Dybkjaer, L., Bernsen, N.O. (eds.) Advances in Natural Multimodal Dialogue Systems. Springer, Berlin (2005)Google Scholar
  9. 9.
    Reithinger, N., Sonntag, D.: An integration framework for a mobile multimodal dialogue system accessing the semantic web. In: Proceedings of Interspeech, Lisbon, Portugal, pp. 841–844 (2005)Google Scholar
  10. 10.
    Sonntag, D.: Ontologies and Adaptivity in Dialogue for Question Answering. AKA and IOS Press, Heidelberg (2010)MATHGoogle Scholar
  11. 11.
    Sonntag, D., Deru, M., Bergweiler, S.: Design and implementation of combined mobile and touchscreen-based multimodal web 3.0 interfaces. In: Proceedings of the International Conference on Artificial Intelligence (ICAI), pp. 974–979 (2009)Google Scholar
  12. 12.
    Sonntag, D., Möller, M.: Unifying semantic annotation and querying in biomedical image repositories. In: Proceedings of International Conference on Knowledge Management and Information Sharing (KMIS) (2009)Google Scholar
  13. 13.
    Sonntag, D., Reithinger, N., Herzog, G., Becker, T.: a discourse and dialogue infrastructure for industrial dissemination. Proceedings of IWSDS—Spoken Dialogue Systems for Ambient Environment, Chapter. Lecture Notes in Artificial Intelligence, pp. 132–143. Springer, Berlin (2010)Google Scholar
  14. 14.
    Sonntag, D., Sonnenberg, G., Nesselrath, R., Herzog, G.: Supporting a rapid dialogue engineering process. In: Proceedings of the First International Workshop On Spoken Dialogue Systems Technology (IWSDS) (2009)Google Scholar
  15. 15.
    Wahlster, W.: SmartKom: symmetric multimodality in an adaptive and reusable dialogue shell. In: Krahl, R., Günther, D. (eds.) Proceedings of the Human Computer Interaction Status Conference 2003, pp. 47–62. DLR, Berlin, Germany (2003)Google Scholar
  16. 16.
    Weiss, D.L., Langlotz, C.: Structured reporting: Patient care enhancement or productivity nightmare? Radiology 249(3), 739–747 (2008)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.German Research Center for AI (DFKI)SaarbrueckenGermany

Personalised recommendations