Low complexity head tracking on portable android devices for real time message composition

  • 299 Accesses

  • 2 Citations


For the people who are totally or partially unable to move or control their limbs and cannot rely on verbal communication, it is very important to obtain an interface capable of interpreting their limited voluntary movements, in order to allow communications with friends, relatives and care providers, or to send commands to a system. This paper presents a real time software application for disabled subjects, suffering from both motor and speech impairments, that provides message composition and speech synthesis functionalities based on face detection and head tracking. The proposed application runs on portable devices equipped with Android Operating System, and relies upon the O.S.’s native computer vision primitives, without resorting to any external software library. This way, the available camera sensors are exploited, and the device computational requirements accomplished. Experimental results show the effectiveness of the application in recognizing the user’s movements, and the reliability of the message composition and speech synthesis functionalities.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11


  1. 1.

    Mads B, Gimpel G, Hedman J (2009) The user experience of smart phones: a consumption values approach. In: Proc. 8th Global Mobility Roundtable, GMR, Cairo

  2. 2.

    World Health Organization (2009) Dept. of Violence and Injury Prevention, Global status report on road safety: time for action, World Health Organization

  3. 3.

    Wood E, Willoughby T, Rushing A, Bechtel L, Gilbert J (2005) Use of computer input devices by older adults. J Appl Gerontol 24(5):419–438

  4. 4.

    Spinsante S, Gambi E (2012) Remote health monitoring by OSGi technology and digital TV integration. IEEE Trans Consum Electron 58(4):1434–1441

  5. 5.

    Mertens A, Koch-Korfges D, Jochems N, Schlick CM (2010) Touchscreen-based input technique for people with intention tremor. In: Proc. of 3rd Conference on Human System Interactions (HSI). pp 236–240

  6. 6.

    Spinsante S, Gambi E (2012) Home automation systems control by head tracking in AAL applications. In: Proc. of IEEE 1st ESTEL Conference, Rome, April 2012

  7. 7.

    Ren J, Rahman M, Kehtarnavaz N, Estevez L (2010) Real-time head pose estimation on mobile platforms. J Syst, Cybern Inf 8(3):56–62

  8. 8.

    Lupu RG, Ungureanu F, Bozomitu RG (2012) Mobile embedded system for human computer communication in assistive technology. In: Proc. IEEE International Conference on Intelligent Computer Communication and Processing (ICCP), 2012, pp 209–212, Aug 30 2012–Sept

  9. 9.

    Takahashi K, Mitsukura Y (2013) Head pose tracking system using a mobile device. In: Proc. 2013 IEEE Int. Symposium on Robot and Human Interactive Communication, pp 461–466, 26–29 Aug 2013

  10. 10.

    Montanini L, Cippitelli E, Gambi E, Spinsante S (2014) Real time message composition through head movements on portable Android devices. In: Proc. IEEE 2014 Int. Conf. on Consumer Electronics, pp 526–527, Jan 10–13, 2014, Las Vegas, USA

  11. 11.

    Haraikawa T, Oyamada A, Ito H, Oikawa S, Fukui Y (2014) A cost-effective solution for realizing talking appliances for the visually impaired. In: 2014 IEEE International Conference on Consumer Electronics (ICCE), pp 317–318, 10–13 Jan 2014

  12. 12.

    Ivanov R (2014) Blind-environment interaction through voice augmented objects. J Multimodal User Interfaces 8(4):345–365

  13. 13.

    Batliner A, Hacker C, Nth E (2008) To talk or not to talk with a computer. J Multimodal User Interfaces 2(3–4):171–186

  14. 14.

    Chandramouli C, Agarwal V (2009) Speech Recognition based Computer Keyboard Replacement for the Quadriplegics, Paraplegics, Paralytics and Amputees, IEEE International Workshop on Medical Measurements and Applications, pp 241–245, 29–30 May 2009

  15. 15.

    Kathirvelan J, Anilkumar R, Alex ZC, Fazul A (2012) Development of low cost automatic wheelchair controlled by oral commands using standalone controlling system, IEEE International Conference on Computational Intelligence & Computing Research (ICCIC), pp 1–4, 18–20 Dec 2012

  16. 16.

    McFarland DJ, Wolpaw JR (2011) Brain-computer interfaces for communication and control. Commun ACM 54(5):60–66

  17. 17.

    Donegan M, Cotmore S, Holmqvist E, Buchholz M, Lundalv M, Pasian V, Farinetti L, Corno F (2009) Deliverable 3.6: Final User Trials Report. Communication by Gaze Interaction (COGAIN) IST-2003-511598.

  18. 18.

    Beukelman DR, Yorkston KM, Reichle J (2000) Augmentative and alternative communication for adults with acquired neurologic disorders. Paul H Brookes, Baltimore, MD

  19. 19.

    Kumar N, Kohlbecher S, Schneider E (2009) A novel approach to video-based pupil tracking. In: Proc. IEEE International Conference on Systems, Man and Cybernetics, 2009. pp 1255–1262, 11–14 Oct 2009

  20. 20.

    Rantanen V, Vanhala T, Tuisku O, Niemenlehto P-H, Verho J, Surakka V, Juhola M, Lekkala J (2011) A wearable, wireless gaze tracker with integrated selection command source for human-computer interaction, IEEE Trans. On Inf Tech Biomed 15(5):795–801

  21. 21.

    Lupu RG, Ungureanu F, Siriteanu V (2013) Eye tracking mouse for human computer interaction. In: Proc. 2013 E-Health and Bioengineering Conference, pp 1–4, 21–23 Nov 2013

  22. 22.

    La Cascia M, Sclaroff S (1999) Fast, reliable head tracking under varying illumination. In: Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol 1

  23. 23.

    Malassiotis S, Strintzis MG (2003) Real-time head tracking and 3D pose estimation from range data. In: Proc. of International Conference on Image Processing, vol 2, pp 859–862

  24. 24.

    Yun F, Huang TS (2007) hMouse: head tracking driven virtual computer mouse. In: Proc. of IEEE Workshop on Applications of Computer Vision

  25. 25.

    Morency LP, Sidner C, Lee C, Darrell T (2005) Contextual recognition of head gestures. In: Proceedings of the International Conference on Multimodal Interactions, Oct 46, 2005, Trento, Italy

  26. 26.

    Song Y, Luo Y, Lin J (2011) Detection of movements of head and mouth to provide computer access for disabled, 2011 International Conference on Technologies and Applications of Artificial Intelligence (TAAI), pp 223–226, 11–13 Nov 2011

  27. 27.

    Bastos-Filho T, Ferreira A, Cavalieri D, Silva R, Muller S, Perez E (2013) Multi-modal interface for communication operated by eye blinks, eye movements, head movements, blowing/sucking and brain waves, 2013 ISSNIP Biosignals and Biorobotics Conference (BRC), pp 1–6, 18–20 Feb 2013

  28. 28.

    Morency LP, Rahimi A, Darrell T (2003) Adaptive view-based appearance model. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition

  29. 29.

    Face Tracking on Android O.S. demo video clip, available at:

Download references

Author information

Correspondence to Laura Montanini.

Electronic supplementary material

Below is the link to the electronic supplementary material.



Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Montanini, L., Cippitelli, E., Gambi, E. et al. Low complexity head tracking on portable android devices for real time message composition. J Multimodal User Interfaces 9, 141–151 (2015).

Download citation


  • Face detection
  • Head tracking
  • Android O.S.
  • Human-system interface