Achieving Virtual Presence with a Semi-Autonomous Robot Through a Multi-Reality and Speech Control Interface

  • Kristian T. Simsarian
  • Ivan Bretan
  • Niklas Frost
  • Jussi Karlgren
  • Emmanuel Frécon
  • Lars Jonsson
  • Lennart E. Fahlén
  • Tomas Axling
Part of the Eurographics book series (EUROGRAPH)


This paper describes a model for a complex human-machine system where a human operator controls a remote robot through the mediation of a distributed virtual environment with a language interface. The system combines speech controlled graphical immersive environments with the live video from a robot working in a real environment. The worlds are synchronized and updated based on operator selections, commands and robot actions. This system allows the user to have a powerful tool with a high level of abstraction to create and control autonomous robots, thus making possible the realization of single and possibly multiple real-world autonomous robot applications.


Virtual Reality Mobile Robot Virtual Environment Augmented Reality Virtual World 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Magnus Andersson, Lennart E. Fahlén, and Torleif Söderlund. A virtual environment user interface for a robotic assistive device. In Proceedings of the second European Conference on the Advancement of Rehabilitation Technology, pages 33–57, Stockholm, May 1993.Google Scholar
  2. [2]
    Ronald Arkin. Reactive control as a substrate for telerobotic systems. IEEE AES Systems Magazine, pages 24–31, June 1991.Google Scholar
  3. [3]
    Ronald Arkin and Khaled S. Ali. Integration of reactive and telerobotic control in multi-agent robotic systems. In Proceedings of Third International Conference on Simulation of Adaptive Behavior: From Animals to Animals, Brighten, UK, 1994.Google Scholar
  4. [4]
    W.A. Aviles, T.W. Hughes, H.R. Everett, A.Y. Martin, and A.H. Koyamatsu. Issues in mobile robotics: The unmanned ground vehicle program teleoperated vehicle (tov). In SPIE Vol. 1388 Mobile Robots V, pages 587–597, 1990.Google Scholar
  5. [5]
    S Benford, J. Bowers, L. Fahlén, and C. Greenhalg. Managing mutual awareness in collaborative virtual environments. In Proceedings of VRST’94, ACM, Singapore, 1994.Google Scholar
  6. [6]
    S. Benford and L. Fahlen. A spatial model of interaction in large virtual environments. In Third European Conference on Computer-Supported Cooperative Work, pages 109–124. Kluwer Academic Publishers, 1993.Google Scholar
  7. [7]
    Steve Benford, John Bowers, Lennart E. Fahlen, Chris Greenhalgh, John Mariani, and Tom Rodden. Networked virtual reality and co-operative work. To appear in Presence, 1995.Google Scholar
  8. [8]
    Alan W. Biermann, Bruce W. Ballard, and Anne H. Sigmon. An experimental study of natural language programming. International journal of man-machine studies, 18:71–87, 1983.CrossRefGoogle Scholar
  9. [9]
    Edwin Bos, Carla Huls, and Wim Claassen. Edward: full integration of language and action in a multimodal user interface. International Journal of Human-Computer Studies, 40:473–495, 1994.CrossRefGoogle Scholar
  10. [10]
    S. Bouffouix and M. Bogaert. Real time navigation and obstacle avoidance for teleoperated vehicles. In SPIE Vol. 1831 Mobile Robots VII, pages 265–275, 1992.Google Scholar
  11. [11]
    Ivan Bretan, Niklas Frost, and Jussi Karlgren. Using surface syntax in interactive interfaces. In The 10th Nordic Conference of Computational Linguistics, University of Helsinki, 1995.Google Scholar
  12. [12]
    Christer Carlsson and Olof Hagsand. DIVE — a platform for multi-user virtual environments. Computers and Graphics, 17(6), 1993.Google Scholar
  13. [13]
    R. Chandrasekar and S. Ramani. Interactive communication of sentential structure and content: an alternative approach to man-machine communication. International Journal of Man-Machine Studies, 30:121–148, 1989.CrossRefGoogle Scholar
  14. [14]
    Philip R. Cohen. The role of natural language in a multimodal interface. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST), pages pp. 143–150, Monterey, CA, 1992.Google Scholar
  15. [15]
    Steven Feiner, Blair MacIntyre, and Doree Seligmann. Knowledge-based augmented reality. Communications of the ACM, 36(7):52–62, July 1993.CrossRefGoogle Scholar
  16. [16]
    Bretan I. Natural Language in Model World Interfaces. Licentiate Thesis, Department of Computer and Systems Sciences. The Royal Institute of Technology and Stockholm University, Stockholm Sweden, 1995.Google Scholar
  17. [17]
    Jussi Karlgren, Ivan Bretan, Niklas Frost, and Lars Jonsson. Interaction models, reference, and interactivity for speech interfaces to virtual environments. Proceedings of Second Eurographics Workshop on Virtual Environments — Realism and Real Time, 1995.Google Scholar
  18. [18]
    Fred Karlsson. Constraint grammar for parsing running text. In Karlgren, editor, Thirteenth International Conference On Computational Linguistics (COLING — 90), University of Helsinki, Helsinki, 1990.Google Scholar
  19. [19]
    Fred Karlsson, Atro Voutilainen, Juha Heikkila, and Arto Anttila (eds.). Constraint Grammar. Mouton de Gruyter, Berlin, 1995.Google Scholar
  20. [20]
    Jacqueline H. Kim, Richard J. Weidner, and Allan L. Sacks. Using virtual reality for science mission planning: A mars pathfinder case. In ISMCR 1994: Topical Workshop on Virtual Reality, pages 37–42, Houston, 1994. NASA Conference publication 10163.Google Scholar
  21. [21]
    Eric Krotkov. Mobile robot localization using a single image. In IEEE Proceedings of Robotics and Automation, pages 978–983, 1989.Google Scholar
  22. [22]
    M. Mallem, F. Chavand, and E. Colle. Computer-assisted visual perception in teleoperated robotics. Robotica (10), pages 99–103, 1992.CrossRefGoogle Scholar
  23. [23]
    Paul Milgram and David Drascic. Enhancement of 3-d video displays by means of superimposed stereo-graphics. In Proceedings of the Human Factors Society 35th Annual Meeting, pages 1457–1461, 1991.Google Scholar
  24. [24]
    NASA. Proceedings of the NASA Conference on Space Telerobotics. JPL Publication 89-7, Vol 1-5, Pasadena, Ca, 1989.Google Scholar
  25. [25]
    Jane Robinson. Dependency structures and transformational rules. Language, 46:259–285, 1970.CrossRefGoogle Scholar
  26. [26]
    Eric Rose, David Breen, Klaus H. Ahlers, Chris Compton, Mihran Tuceryan, Ross Whitaker, and Douglas Greer. Annotating real-world objects using augmented vision. Technical report, European Computer-Industry Research Center GmbH, Arabellastrasse 17 D-81925 Munich, 1994.Google Scholar
  27. [27]
    Ben Shneiderman. Natural vs. precise concise languages for human operation of computers: Research issues and experimental approaches. In Proceedings of the 18th Annual Meeting of the Association for Computational Linguistics, Philadelphia, 1980.Google Scholar
  28. [28]
    K. T. Simsarian, N. Nandhakumar, and T. J. Olson. Mobile robot self-localization from range data using view-invariant regions. In IEEE Proceedings of the 5th International Symposium on Intelligent Control, pages 1038–1043, 1990.Google Scholar
  29. [29]
    P.C Woodland, J.J. Odell V. Valtchev, and S.J. Young. Large vocabulary continuous speech recognition using htk. In Proceedings of ICASSP’94, Adelaide, 1994.Google Scholar

Copyright information

© Springer-Verlag/Wien 1996

Authors and Affiliations

  • Kristian T. Simsarian
    • 1
  • Ivan Bretan
    • 1
  • Niklas Frost
    • 1
  • Jussi Karlgren
    • 1
  • Emmanuel Frécon
    • 1
  • Lars Jonsson
    • 1
  • Lennart E. Fahlén
    • 1
  • Tomas Axling
    • 1
  1. 1.Swedish Institute of Computer ScienceKista StockholmSweden

Personalised recommendations