Mobile Reality: A PDA-Based Multimodal Framework Synchronizing a Hybrid Tracking Solution with 3D Graphics and Location-Sensitive Speech Interaction

  • Stuart Goose
  • Heiko Wanning
  • Georg Schneider
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2498)


A maintenance engineer who talks to pumps and pipes may not seem like the ideal person to entrust with keeping a factory running smoothly, but we hope that our Mobile Reality framework will enable such behavior in the future to be anything but suspicious! Described in this paper is how the Mobile Reality framework, running entirely on a Pocket PC, synchronizes a hybrid tracking solution to offer the user a seamless, location-dependent, mobile multimodal interface. The user interface juxtaposes a three-dimensional graphical view with a context-sensitive speech dialog centered upon objects located in the immediate vicinity of the mobile user. In addition, support for collaboration enables shared VRML browsing with annotation and a full-duplex voice channel.


Speech Recognition Augmented Reality Multimodal Interface Proximity Sensor Mobile Reality 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aalburg, S., Beaugeant, C., Stan, S., Fingscheidt, T., Balan, R. and Rosca, J., Single and Two-Channel Noise Reduction for Robust Speech Recognition, to appear in ISCA Workshop Multi-Modal Dialogue in Mobile Environments, June 2002.Google Scholar
  2. 2.
    Abowd, G., Atkeson, C., Dey, A., Hong, J., Long, S., Kooper R. and Pinkerton, M., Cyberguide: A Mobile Context-Aware Tourguide, ACM Wireless Networks Volume 3, pages 421–433, November 1997.Google Scholar
  3. 3.
    Beigl, M., Memoclip: A Location-based Rembrance Applicance, Journal of Personal Technologies, 4(4):230–234, Springer Press, 2000.CrossRefGoogle Scholar
  4. 4.
    Dai, F., Virtual Reality for Industrial Applications, Springer-Verlag, 1998.Google Scholar
  5. 5.
    Eyeled GmbH, Saarbrücken, Germany
  6. 6.
    Fitzmaurice, G., Zhai, Z. and Chignell, M. Virtual Reality for Palmtop Computers, ACM Transactions on Office Information Systems, 11(3):197–218, July, 1993.Google Scholar
  7. 7.
    Goose, S., Gruber, I., Sudarsky, S., Hampel, K., Baxter, B. and Navab, N., 3D Interaction and Visualization in the Industrial Environment, Proceedings of the 9th International Conference on Human Computer Interaction, New Orleans, USA, Volume 1, pages 31–35, August, 2001.Google Scholar
  8. 8.
    Grasso, M., Ebert, D. and Finin, T., The Integrality of Speech in Multimodal Interfaces, ACM Transactions on Computer Human Interaction, 5(4):303–325, December 1998.Google Scholar
  9. 9.
    Hinkley, K, Pierce, J., Sinclair, M. and Horvitz, E., Sensing Techniques for Mobile Interaction, ACM UIST, San Diego, USA, November 2000.Google Scholar
  10. 10.
    Hull, R., Neaves, P. and Bedford-Roberts, J. Towards Situated Computing, Proceeding of IEEE First International Symposium on Wearable Computing, Cambridge, USA, pages 146–153, October 1997.Google Scholar
  11. 11.
    Infrared Data Association:
  12. 12.
    Klinker, G., Reicher, T. and Bruegge, B., Distributed User Tracking Concepts for Augmented Reality Applications, Proceedings of ISAR 2000, Munich, Germany, pages 37–44, October, 2000.Google Scholar
  13. 13.
    Kortuem, G., Segall, Z., Thompson, T., Close Encounters: Supporting Mobile Collaboration through Interchange of User Profiles, Proceedings of the First International Symposium on Handheld and Ubiquitous Computing, Karlsruhe, Germany, pages 171–185, September 1999.Google Scholar
  14. 14.
    Lewis, R. and Séquin, C. H., Generation of Three-Dimensional Building Models from Two-Dimensional Architectural Plans, Computer-Aided Design, 30:10, pages 765–779, 1998.zbMATHCrossRefGoogle Scholar
  15. 15.
    Luff, P. and Heath, C., Mobility in Collaboration, Proceedings of CSCW’ 98, Seattle, USA, November 1998.Google Scholar
  16. 16.
    Marmasse, N. and Schmandt, C., Location-aware Information Delivery with ComMotion, Proceedings of the Second International Symposium on Handheld and Ubiquitous Computing, Bristol, U.K., pages 157–171, September 2000.Google Scholar
  17. 17.
  18. 18.
    Mynatt, E., Back, M., Want, R., Baer, M. and Ellis, J., Designing Audio Aura, ACM International Conference on Computer Human Interaction, Los Angeles, USA, pages 566–573, 1998.Google Scholar
  19. 19.
    Newman, J., Ingram, D. and Hopper, A., Augmented Reality in a Wide Area Sentient Environment, Proceedings of ISAR 2000, New York, USA, pages 77–86, October, 2001.Google Scholar
  20. 20.
    Nilsson, J., Sokoler, T., Binder, T. and Wetcke, N., Beyond the Control Room: Mobile Devices for Spatially Distributed Interaction on Industrial Process Plants, Proceedings of the Second International Symposium on Handheld and Ubiquitous Computing, Bristol, U.K., pages 30–45, September 2000.Google Scholar
  21. 21.
  22. 22.
    Rekimoto, J. and Ayatsuka, Y., Cybercode: Designing Augmented Reality Environments With Visual Tags, Designing Augmented Reality Environments, 2000.Google Scholar
  23. 23.
    Ressler, S. and Wang, Q., Making VRML Accessible for People with Disabilities, ASSETS 98, Marina del Rey, USA, pages 50–55, April 1998.Google Scholar
  24. 24.
    Saffo, P., Sensors: The Next Wave of Infortech Innovation, Institute for the Future: 1997 Ten-Year Forecast, pages 115–122.Google Scholar
  25. 25.
  26. 26.
    Smailagic, A. and Bennington, B., Wireless and Mobile Computing in Training Maintenance and Diagnosis, IEEE Vehicular Technology Conference, Phoenix, AZ, May 1997.Google Scholar
  27. 27.
  28. 28.
    Wan, D., Magic Medicine Cabinet: A Situated Portal for Consumer Healthcare, Proceedings of the First International Symposium on Handheld and Ubiquitous Computing, Karlsruhe, Germany, pages 352–355, September 1999.Google Scholar
  29. 29.
    Want, R., Hopper, A., Falcao, V. and Gibbons, J., The Active Badge Location System, ACM Transactions on Information Systems, 10(1):91–102, 1992.CrossRefGoogle Scholar
  30. 30.
    Zhai, S., Milgram, P. and Drasic, D., An Evaluation of four 6 Degree-of-Freedom Input Techniques, ACM Conference on Human Factors in Computer Systems, Asmterdam, Netherlands, 1993.Google Scholar
  31. 31.
    Zhang, X. and Navab, N., Tracking and Pose Estimation for Computer Assisted Localization in Industrial Environments, IEEE Workshop on Applications of Computer Vision, pages 241–221, 2000.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Stuart Goose
    • 1
  • Heiko Wanning
    • 2
  • Georg Schneider
    • 3
  1. 1.Multimedia Technology DepartmentSiemens Corporate Research, Inc.PrincetonUSA
  2. 2.Computer Science DepartmentUniversity of SaarlandesSaarbrückenGermany
  3. 3.Computer Science DepartmentFachhochschule TrierTrierGermany

Personalised recommendations