For the people who are totally or partially unable to move or control their limbs and cannot rely on verbal communication, it is very important to obtain an interface capable of interpreting their limited voluntary movements, in order to allow communications with friends, relatives and care providers, or to send commands to a system. This paper presents a real time software application for disabled subjects, suffering from both motor and speech impairments, that provides message composition and speech synthesis functionalities based on face detection and head tracking. The proposed application runs on portable devices equipped with Android Operating System, and relies upon the O.S.’s native computer vision primitives, without resorting to any external software library. This way, the available camera sensors are exploited, and the device computational requirements accomplished. Experimental results show the effectiveness of the application in recognizing the user’s movements, and the reliability of the message composition and speech synthesis functionalities.
This is a preview of subscription content, log in to check access.
Buy single article
Instant unlimited access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Mads B, Gimpel G, Hedman J (2009) The user experience of smart phones: a consumption values approach. In: Proc. 8th Global Mobility Roundtable, GMR, Cairo
World Health Organization (2009) Dept. of Violence and Injury Prevention, Global status report on road safety: time for action, World Health Organization
Wood E, Willoughby T, Rushing A, Bechtel L, Gilbert J (2005) Use of computer input devices by older adults. J Appl Gerontol 24(5):419–438
Spinsante S, Gambi E (2012) Remote health monitoring by OSGi technology and digital TV integration. IEEE Trans Consum Electron 58(4):1434–1441
Mertens A, Koch-Korfges D, Jochems N, Schlick CM (2010) Touchscreen-based input technique for people with intention tremor. In: Proc. of 3rd Conference on Human System Interactions (HSI). pp 236–240
Spinsante S, Gambi E (2012) Home automation systems control by head tracking in AAL applications. In: Proc. of IEEE 1st ESTEL Conference, Rome, April 2012
Ren J, Rahman M, Kehtarnavaz N, Estevez L (2010) Real-time head pose estimation on mobile platforms. J Syst, Cybern Inf 8(3):56–62
Lupu RG, Ungureanu F, Bozomitu RG (2012) Mobile embedded system for human computer communication in assistive technology. In: Proc. IEEE International Conference on Intelligent Computer Communication and Processing (ICCP), 2012, pp 209–212, Aug 30 2012–Sept
Takahashi K, Mitsukura Y (2013) Head pose tracking system using a mobile device. In: Proc. 2013 IEEE Int. Symposium on Robot and Human Interactive Communication, pp 461–466, 26–29 Aug 2013
Montanini L, Cippitelli E, Gambi E, Spinsante S (2014) Real time message composition through head movements on portable Android devices. In: Proc. IEEE 2014 Int. Conf. on Consumer Electronics, pp 526–527, Jan 10–13, 2014, Las Vegas, USA
Haraikawa T, Oyamada A, Ito H, Oikawa S, Fukui Y (2014) A cost-effective solution for realizing talking appliances for the visually impaired. In: 2014 IEEE International Conference on Consumer Electronics (ICCE), pp 317–318, 10–13 Jan 2014
Ivanov R (2014) Blind-environment interaction through voice augmented objects. J Multimodal User Interfaces 8(4):345–365
Batliner A, Hacker C, Nth E (2008) To talk or not to talk with a computer. J Multimodal User Interfaces 2(3–4):171–186
Chandramouli C, Agarwal V (2009) Speech Recognition based Computer Keyboard Replacement for the Quadriplegics, Paraplegics, Paralytics and Amputees, IEEE International Workshop on Medical Measurements and Applications, pp 241–245, 29–30 May 2009
Kathirvelan J, Anilkumar R, Alex ZC, Fazul A (2012) Development of low cost automatic wheelchair controlled by oral commands using standalone controlling system, IEEE International Conference on Computational Intelligence & Computing Research (ICCIC), pp 1–4, 18–20 Dec 2012
McFarland DJ, Wolpaw JR (2011) Brain-computer interfaces for communication and control. Commun ACM 54(5):60–66
Donegan M, Cotmore S, Holmqvist E, Buchholz M, Lundalv M, Pasian V, Farinetti L, Corno F (2009) Deliverable 3.6: Final User Trials Report. Communication by Gaze Interaction (COGAIN) IST-2003-511598. http://wiki.cogain.org/index.php/File:COGAIN-D3.6
Beukelman DR, Yorkston KM, Reichle J (2000) Augmentative and alternative communication for adults with acquired neurologic disorders. Paul H Brookes, Baltimore, MD
Kumar N, Kohlbecher S, Schneider E (2009) A novel approach to video-based pupil tracking. In: Proc. IEEE International Conference on Systems, Man and Cybernetics, 2009. pp 1255–1262, 11–14 Oct 2009
Rantanen V, Vanhala T, Tuisku O, Niemenlehto P-H, Verho J, Surakka V, Juhola M, Lekkala J (2011) A wearable, wireless gaze tracker with integrated selection command source for human-computer interaction, IEEE Trans. On Inf Tech Biomed 15(5):795–801
Lupu RG, Ungureanu F, Siriteanu V (2013) Eye tracking mouse for human computer interaction. In: Proc. 2013 E-Health and Bioengineering Conference, pp 1–4, 21–23 Nov 2013
La Cascia M, Sclaroff S (1999) Fast, reliable head tracking under varying illumination. In: Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol 1
Malassiotis S, Strintzis MG (2003) Real-time head tracking and 3D pose estimation from range data. In: Proc. of International Conference on Image Processing, vol 2, pp 859–862
Yun F, Huang TS (2007) hMouse: head tracking driven virtual computer mouse. In: Proc. of IEEE Workshop on Applications of Computer Vision
Morency LP, Sidner C, Lee C, Darrell T (2005) Contextual recognition of head gestures. In: Proceedings of the International Conference on Multimodal Interactions, Oct 46, 2005, Trento, Italy
Song Y, Luo Y, Lin J (2011) Detection of movements of head and mouth to provide computer access for disabled, 2011 International Conference on Technologies and Applications of Artificial Intelligence (TAAI), pp 223–226, 11–13 Nov 2011
Bastos-Filho T, Ferreira A, Cavalieri D, Silva R, Muller S, Perez E (2013) Multi-modal interface for communication operated by eye blinks, eye movements, head movements, blowing/sucking and brain waves, 2013 ISSNIP Biosignals and Biorobotics Conference (BRC), pp 1–6, 18–20 Feb 2013
Morency LP, Rahimi A, Darrell T (2003) Adaptive view-based appearance model. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition
Face Tracking on Android O.S. demo video clip, available at: http://youtu.be/racDJvUJKL4
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Montanini, L., Cippitelli, E., Gambi, E. et al. Low complexity head tracking on portable android devices for real time message composition. J Multimodal User Interfaces 9, 141–151 (2015). https://doi.org/10.1007/s12193-015-0174-7
- Face detection
- Head tracking
- Android O.S.
- Human-system interface