A Modality Replacement Framework for the Communication between Blind and Hearing Impaired People
This paper presents a multimodal framework for the communication between blind and hearing impaired people. The algorithms that are developed are based on the concept of modality replacement, which is the use of information originating from various modalities to compensate for the missing input modality of the system or the users. Spatial information is conveyd in the blind user’s terminal through the haptic modality utilizing an efficient haptic rendering scheme, while verbal information is presented in the hearing impaired user’s terminal through sign language synthesis. All technologies are integrated in a virtual treasure hunting game, where a blind and a hearing impaired user have to collaborate so as to navigate in the virtual environment and solve the riddle of the game. Usability evaluation of this framework has shown significant impact of the proposed system for the disabled users.
KeywordsModality replacement multimodal interfaces haptics sign language virtual reality
Unable to display preview. Download preview PDF.
- 2.Bologna, G., Deville, B., Pun, T., Vinckenbosch, M.: Transforming 3d coloured pixels into musical instrument notes for vision substitution applications. Eurasip Journal of Image and Video Processing, Special Issue: Image and Video Processing for Disability 24, 14–27 (2007)Google Scholar
- 3.Lumsden, J., Brewster, S.A.: A paradigm shift: Alternative interaction techniques for use with mobile & wearable devices. In: Proc. 13th Annual IBM Centers for Advanced Studies Conference CASCON 2003, Toronto, Canada, pp. 97–100 (2003)Google Scholar
- 7.Papadogiorgaki, M.G., Grammalidis, N., Makris, L., Strintzis, M.G.: Gesture synthesis from sign language notation using MPEG-4 humanoid animation parameters and inverse kinematics. In: 2nd International Conference on Intelligent Environments (IE 2006), Athens, Greece, July 5-6 (2006)Google Scholar
- 8.Raman, T.V.: Multimodal Interaction Design Principles For Multimodal Interaction. In: CHI 2003, pp. 5–10. Fort Lauderdale, USA (2003)Google Scholar