Skip to main content
Log in

Toward the gestural interface: comparative analysis between touch user interfaces versus gesture-based user interfaces on mobile devices

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

Human–computer interaction is transformed as well as the technology used to achieve it. This paper discusses the ease of use of a classic touch screen user interface (TUI) in contrast to a gesture-based user interface (GBUI) applied to a mobile device. Two basic positions are used for comparison: “normal” when individual is standing and “special” when individual is lying down. The first represents a common position for healthy individuals, while the second is used to represent sick, disabled or comfortable individuals. Two software applications have been developed in order to set an interaction test for TUI and GBUI, which was performed by 25 users. The interaction tests show that GBUI has a very reasonable average accuracy of 88.5% in special position and 89.8% in normal position, while TUI presents an average accuracy of 96.5% in special position and 97.4% in normal position. Therefore, GBUI could be a complementary form of interaction to TUI that can be useful to any user, especially for cases of illness or simple comfort of the user. As study case, we have developed a software solution called SICLLE (emergency call control interactive system) that manages phone calls in a smartphone and applies the concept of gestural control tree to set a gestural navigation through a gestural language from a gestural vocabulary. An architecture for mobile devices supporting GBUI has been proposed and used in the implementation of SICLLE. SICLLE allows the personalization of the gestures, which optimizes the gestural dictionary and achieves an easier learning and use. In addition, SICLLE incorporates a hearing guide which replaces the typical visual control, reaching total acceptance in the experiment conducted with real users.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. MYO is a trademark of Thalmic Labs Inc.

  2. (https://www.thalmic.com/)

References

  1. Abid, M.R., Shi, F., Petriu, E.M.: Dynamic hand gesture recognition from bag-of-features and local part model. In: Proceedings of 2012 IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE 2012), IEEE, pp. 78–82 (2012). doi:10.1109/HAVE.2012.6374443

  2. AdnanIbraheem, N., Zaman Khan, R.: Survey on various gesture recognition technologies and techniques. Int. J. Comput. Appl. 50(7), 38–44 (2012). doi:10.5120/7786-0883, http://research.ijcaonline.org/volume50/number7/pxc3880883.pdf

  3. Al-Timemy, A.H., Bugmann, G., Escudero, J., Outram, N.: Classification of finger movements for the dexterous hand prosthesis control with surface electromyography. IEEE J. Biomed. Health Inform. 17(3), 608–618 (2013). doi:10.1109/JBHI.2013.2249590

    Article  Google Scholar 

  4. Blazquez Cano, M., Perry, P., Ashman, R., Waite, K.: The influence of image interactivity upon user engagement when using mobile touch screens. Comput. Hum. Behav. 1–7 (2017). doi:10.1016/j.chb.2017.03.042, http://linkinghub.elsevier.com/retrieve/pii/S0747563217301991

  5. Buchner, H., Petersen, E., Eger, M., Rostalski, P.: Convolutive blind source separation on surface EMG signals for respiratory diagnostics and medical ventilation control. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, pp. 3626–3629 (2016). doi:10.1109/EMBC.2016.7591513, http://ieeexplore.ieee.org/document/7591513/

  6. Card, S.K., Newell, A., Moran, T.P.: The Psychology of Human–Computer Interaction. Lawrence Erlbaum Associates Inc., Hillsdale, NJ (1983)

    Google Scholar 

  7. Chen, C., Jafari, R., Kehtarnavaz, N.: Improving human action recognition using fusion of depth camera and inertial sensors. IEEE Trans. Hum. Mach. Syst. 45(1), 51–61 (2015). doi:10.1109/THMS.2014.2362520, http://ieeexplore.ieee.org/document/6934998/

  8. Chen, X., Wang, Z.J.: Pattern recognition of number gestures based on a wireless surface EMG system. Biomed. Signal Process. Control 8(2), 184–192 (2013). doi:10.1016/j.bspc.2012.08.005

    Article  Google Scholar 

  9. Chittaro, L.: Visualizing information on mobile devices. Computer 39(3), 40–45 (2006). doi:10.1109/MC.2006.109, http://ieeexplore.ieee.org/document/1607948/

  10. de Waard, D., Westerhuis, F., Joling, D., Weiland, S., Stadtbäumer, R., Kaltofen, L.: Visual map and instruction-based bicycle navigation: a comparison of effects on behaviour. Ergonomics 0139(April), 1–14 (2017). doi:10.1080/00140139.2017.1282628

    Article  Google Scholar 

  11. Dinh, D.L., Kim, J.T., Kim, T.S.: Hand gesture recognition and interface via a depth imaging sensor for smart home appliances. Energy Procedia 62(62), 576–582 (2014). doi:10.1016/j.egypro.2014.12.419

    Article  Google Scholar 

  12. Erden, F., Çetin, A.E.: Hand gesture based remote control system using infrared sensors and a camera. IEEE Trans. Consum. Electron. 60(4), 675–680 (2014). doi:10.1109/TCE.2014.7027342

    Article  Google Scholar 

  13. Fernandez, R.A.S., Sanchez-Lopez, J.L., Sampedro, C., Bavle, H., Molina, M., Campoy, P.: Natural user interfaces for human-drone multi-modal interaction. In: 2016 International Conference on Unmanned Aircraft Systems (ICUAS), IEEE, pp. 1013–1022 (2016). doi:10.1109/ICUAS.2016.7502665

  14. Fitts, P.M.: The amplitude of movement. J. Exp. Psychol. 47, 381–391 (1954)

    Article  Google Scholar 

  15. Flores-Abad, A., Ma, O., Pham, K., Ulrich, S.: A review of space robotics technologies for on-orbit servicing. Prog. Aerosp. Sci. 68, 1–26 (2014). doi:10.1016/j.paerosci.2014.03.002, http://linkinghub.elsevier.com/retrieve/pii/S0376042114000347

  16. Flowers, K.A., Robertson, C.: The effect of Parkinson’s disease on the ability to maintain a mental set. J. Neurol. Neurosurg. Psychiatry 48(6), 517–529 (1985). doi:10.1136/jnnp.48.6.517

    Article  Google Scholar 

  17. Francese, R., Passero, I., Tortora, G.: Wiimote and Kinect. In: Proceedings of the International Working Conference on Advanced Visual Interfaces—AVI ’12, ACM Press, New York, New York, USA, pp 116–123 (2012). doi:10.1145/2254556.2254580

  18. Garawi, S., Istepanian, R., Abu-Rgheff, M.: 3G wireless communications for mobile robotic tele-ultrasonography systems. IEEE Commun. Mag. 44(4), 91–96 (2006). doi:10.1109/MCOM.2006.1632654

    Article  Google Scholar 

  19. Gokgoz, E., Subasi, A.: Effect of multiscale PCA de-noising on EMG signal classification for diagnosis of neuromuscular disorders. J. Med. Syst. 38(4), 31–38 (2014). doi:10.1007/s10916-014-0031-3

    Article  Google Scholar 

  20. Guerrero, F.N., Spinelli, E.M., Haberman, M.A.: Analysis and simple circuit design of double differential EMG active electrode. IEEE Trans. Biomed. Circuits Syst. 10(3), 787–795 (2016). doi:10.1109/TBCAS.2015.2492944

    Article  Google Scholar 

  21. Hargrove, L., Englehart, K., Hudgins, B.: A comparison of surface and intramuscular myoelectric signal classification. IEEE Trans. Biomed. Eng. 54(5), 847–853 (2007). doi:10.1109/TBME.2006.889192, http://ieeexplore.ieee.org/document/4154997/

  22. Hasan, H., Abdul-Kareem, S.: Static hand gesture recognition using neural networks. Artif. Intell. Rev. 41(2), 147–181 (2014). doi:10.1007/s10462-011-9303-1

    Article  Google Scholar 

  23. Heo, H., Lee, E.C., Park, K.R., Kim, C.J., Whang, M.: A realistic game system using multi-modal user interfaces. IEEE Trans. Consum. Electron. 56(3), 1364–1372 (2010). doi:10.1109/TCE.2010.5606271

    Article  Google Scholar 

  24. Hooper, C.J., Dix, A.: Web science and human–computer interaction. Interactions 20(3), 52–57 (2013). doi:10.1145/2451856.2451868

    Article  Google Scholar 

  25. Hug, F.: Can muscle coordination be precisely studied by surface electromyography? J. Electromyogr. Kinesiol. 21(1), 1–12 (2011). doi:10.1016/j.jelekin.2010.08.009

    Article  Google Scholar 

  26. Ismirle, J., OBara, I., Swierenga, S.J., Jackson, J.E.: Touchscreen voting interface design for persons with disabilities: insights from usability evaluation of mobile voting prototype. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 60(1), 780–784 (2016). doi:10.1177/1541931213601179

  27. Jaimes, A., Sebe, N.: Multimodal humancomputer interaction: a survey. Comput. Vis. Image Underst. 108(1–2), 116–134 (2007). doi:10.1016/j.cviu.2006.10.019

    Article  Google Scholar 

  28. Jin, Z.X, Plocher, T., Kiff, L.: Touch screen user interfaces for older adults: button size and spacing. In: Stephanidis, C. (ed.) Universal Access in Human Computer Interaction. Coping with Diversity, Lecture Notes in Computer Science, vol. 4554, Springer, Berlin, pp. 933–941, (2007). doi:10.1007/978-3-540-73279-2_104, http://www.springerlink.com/index/10.1007/978-3-540-73279-2, http://link.springer.com/10.1007/978-3-540-73279-2

  29. Jung, Y.: What a smartphone is to me: understanding user values in using smartphones. Inf. Syst. J. 24(4), 299–321 (2014). doi:10.1111/isj.12031

  30. Kane, S.K, Jayant, C., Wobbrock, J.O, Ladner, R.E.: Freedom to roam: a study of mobile device adoption and accessibility for people with visual and motor disabilities. In: Proceeding of the Eleventh International ACM SIGACCESS Conference on Computers and Accessibility—ASSETS ’09, ACM Press, New York, NY, USA, pp. 115–122 (2009). doi:10.1145/1639642.1639663

  31. Kaushik, D.M., Jain, R.: Gesture based interaction NUI: an overview. Int. J. Eng. Trends Technol. 9(12), 633–636 (2014). doi:10.14445/22315381/IJETT-V9P319, https://arxiv.org/abs/1404.2364

  32. Kim, G.J.: Human-Computer Interaction: Fundamentals and Practice. CRC Press, Boca Raton (2015)

    Book  Google Scholar 

  33. Klboz, N.Ç., Güdükbay, U.: A hand gesture recognition technique for human–computer interaction. J. Vis. Commun. Image Represent. 28, 97–104 (2015). doi:10.1016/j.jvcir.2015.01.015, http://linkinghub.elsevier.com/retrieve/pii/S10473203

  34. Kühnel, C., Westermann, T., Hemmert, F., Kratz, S., Müller, A., Möller, S.: I’m home: defining and evaluating a gesture set for smart-home control. Int. J. Hum. Comput. Stud. 69(11), 693–704 (2011). doi:10.1016/j.ijhcs.2011.04.005, http://linkinghub.elsevier.com/retrieve/pii/S1071581911000668

  35. Lara, O.D., Labrador, M.A.: A survey on human activity recognition using wearable sensors. IEEE Commun. Surv. Tutor. 15(3), 1192–1209 (2013). doi:10.1109/SURV.2012.110112.00192

    Article  Google Scholar 

  36. LaViola, J.J.: 3D gestural interaction: the state of the field. ISRN Artif. Intell. 2013(2), 1–18 (2013). doi:10.1155/2013/514641

    Article  Google Scholar 

  37. Lee, M., Hong, Y., Lee, S., Won, J., Yang, J., Park, S., Chang, K.T., Hong, Y.: The effects of smartphone use on upper extremity muscle activity and pain threshold. J. Phys. Ther. Sci. 27(6), 1743–1745 (2015). doi:10.1589/jpts.27.1743, https://www.jstage.jst.go.jp/article/jpts/27/6/27_jpts-2015-015/_article

  38. Leitão, R.A.: Creating mobile gesture-based interaction design patterns for older adults: a study of tap and swipe gestures with Portuguese seniors. Ph.D. thesis, Universidade do Porto (2012) http://repositorio-aberto.up.pt/handle/10216/68413

  39. Li, G., Li, Y., Zhang, Z., Geng, Y., Zhou, R.: Selection of sampling rate for EMG pattern recognition based prosthesis control. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, IEEE, vol. 2010, pp. 5058–5061 (2010). doi:10.1109/IEMBS.2010.5626224

  40. Li, C., Ma, H., Yang, C., Fu, M. Teleoperation of a virtual iCub robot under framework of parallel system via hand gesture recognition. In: 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), IEEE, pp 1469–1474 (2014). doi:10.1109/FUZZ-IEEE.2014.6891887, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6891887

  41. Lv, Z., Feng, S., Feng, L., Li, H.: Extending touch-less interaction on vision based wearable device. In: 2015 IEEE Virtual Reality (VR), IEEE, pp 231–232 (2015). doi:10.1109/VR.2015.7223380, http://ieeexplore.ieee.org/document/7223380/, https://arxiv.org/abs/1504.01025

  42. MacKenzie, I.S.: Fitts’ law as a research and design tool in human–computer interaction. Hum. Comput. Interact. 7(1):91–139 (1992). doi:10.1207/s15327051hci07013, http://portal.acm.org/citation.cfm?id=1461857

  43. Mehler, A., Lücking, A., Abrami, G.: WikiNect: image schemata as a basis of gestural writing for kinetic museum wikis. Univers. Access Inf. Soc. 14(3), 333–349 (2015). doi:10.1007/s10209-014-0386-8

  44. Merletti, R., Botter, A., Troiano, A., Merlo, E., Minetto, M.A.: Technology and instrumentation for detection and conditioning of the surface electromyographic signal: state of the art. Clin. Biomech. 24(2), 122–134 (2009). doi:10.1016/j.clinbiomech.2008.08.006

  45. Min, B.W: Improvement of mobile U-health services system. Commun. Comput. Inf. Sci. 262 CCIS(PART 1), 44–51 (2011). doi:10.1007/978-3-642-27204-2_6

  46. Mitra, S., Acharya, T.: Gesture recognition: a survey. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 37(3), 311–324 (2007). doi:10.1109/TSMCC.2007.893280, http://ieeexplore.ieee.org/document/4154947/

  47. Nguyen, Q., Premaratne, P.: Consumer electronics control system based on hand gesture moment invariants. IET Comput. Vis. 1(1), 35–41 (2007). doi:10.1049/iet-cvi:20060198, http://ieeexplore.ieee.org/document/4159604/?arnumber=4159604

  48. Nielsen, J.: Noncommand user interfaces. Commun. ACM 36(4), 83–99 (1993). doi:10.1145/255950.153582

    Article  MathSciNet  Google Scholar 

  49. Nielsen, M., Störring, M., Moeslund, T.B., Granum, E.: A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI. In: Gesture-Based Communication in Human–Computer Interaction, pp. 409–420 (2004). doi:10.1007/978-3-540-24598-8 38

  50. Ohn-Bar, E., Trivedi, M.M.: Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans. Intell. Transp. Syst. 15(6), 2368–2377 (2014). doi:10.1109/TITS.2014.2337331, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6883176

  51. Page, T.: Touchscreen and perceived usability: a comparison of attitudes between older and youger mobile device. I-Manag. J. Mob. Appl. Techonol. 3(1), 1–16 (2016). https://search.proquest.com/docview/1853319087?accountid=14542, http://hdl.handle.net/11025/1847

  52. Peters, A., Morrison, J.H.: Cerebral Cortex: Neurodegenerative and Age-Related Changes in Structure and Function of Cerebral Cortex, vol. 14. Springer, Berlin (2012)

    Google Scholar 

  53. Phinyomark, A., Phukpattaranont, P., Limsakul, C.: Feature reduction and selection for EMG signal classification. Expert Syst. Appl. 39(8), 7420–7431 (2012). doi:10.1016/j.eswa.2012.01.102

    Article  Google Scholar 

  54. Pomboza-Junez, G., Holgado-Terriza, J.A.: Control of home devices based on hand gestures. In: 2015 IEEE 5th International Conference on Consumer Electronics—Berlin (ICCE-Berlin), IEEE, pp 510-514 (2015). DOI:10.1109/ICCE-Berlin.2015.7391325

  55. Pomboza-Junez, G., Holgado-Terriza, J.A.: Hand gesture recognition based on sEMG signals using Support Vector Machines. In: 2016 IEEE 6th International Conference on Consumer Electronics—Berlin (ICCE-Berlin), IEEE, pp. 174–178 (2016). doi:10.1109/ICCE-Berlin.2016.7684748

  56. Premaratne, P., Ajaz, S., Premaratne, M.: Hand gesture tracking and recognition system using Lucas-Kanade algorithms for control of consumer electronics. Neurocomputing 116, 242–249 (2013). doi:10.1016/j.neucom.2011.11.039

    Article  Google Scholar 

  57. Prieto De Lope, R.P., Medina Medina, N.: A comprehensive taxonomy for serious games. J. Educ. Comput. Res. (2016). doi:10.1177/0735633116681301

    Article  Google Scholar 

  58. Quigley, M., Goodrich, M., Beard, R.: Semi-autonomous human–UAV interfaces for fixed-wing mini-UAVs. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. 04CH37566), IEEE, vol. 3, pp. 2457–2462 (2004). doi:10.1109/IROS.2004.1389777

  59. Raez, M.B.I., Hussain, M.S., Mohd-Yasin, F., Reaz, M., Hussain, M.S., Mohd-Yasin, F.: Techniques of EMG signal analysis: detection, processing, classification and applications. Biol. Proced. Online 8(1), 11–35 (2006). doi:10.1251/bpo115

    Article  Google Scholar 

  60. Rainoldi, A., Melchiorri, G., Caruso, I.: A method for positioning electrodes during surface EMG recordings in lower limb muscles. J. Neurosci. Methods 134(1), 37–43 (2004). doi:10.1016/j.jneumeth.2003.10.014

    Article  Google Scholar 

  61. Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2015). doi:10.1007/s10462-012-9356-9

    Article  Google Scholar 

  62. Rempel, D., Camilleri, M.J., Lee, D.L.: The design of hand gestures for human-computer interaction: Lessons from sign language interpreters. Int. J. Hum. Comput. Stud. 72(10–11), 728–735 (2014). doi:10.1016/j.ijhcs.2014.05.003

    Article  Google Scholar 

  63. Rezende, A., Schneider, G., Prakash, P.: Method and apparatus for customizing a display screen of a user interface. https://www.google.com/patents/US9304668, US Patent 9,304,668 (2016)

  64. Robertson, C., Flowers, K.A.: Motor set in Parkinson’s disease. J. Neurol. Neurosurg. Psychiatry 53(7), 583–592 (1990). doi:10.1136/jnnp.53.7.583

    Article  Google Scholar 

  65. Sanchez-Nielsen, E., Hernandez-Tejera, M., Anton-Canalís, L. (2004) Hand gesture recognition for human–machine interaction. J. WSCG 12(1-3), 395–402, http://hdl.handle.net/11025/1847

  66. Sanna, A., Lamberti, F., Paravati, G., Manuri, F.: A Kinect-based natural interface for quadrotor control. Entertain. Comput. 4(3), 179–186 (2013). doi:10.1016/j.entcom.2013.01.001

    Article  Google Scholar 

  67. Schaller, R.: Moore’s law: past, present and future. IEEE Spectr. 34(6), 52–59 (1997). doi:10.1109/6.591665, http://ieeexplore.ieee.org/document/591665/

  68. Stergiopoulou, E., Papamarkos, N.: Hand gesture recognition using a neural network shape fitting technique. Eng. Appl. Artif. Intell. 22(8), 1141–1158 (2009). doi:10.1016/j.engappai.2009.03.008, http://linkinghub.elsevier.com/retrieve/pii/S0952197609000694

  69. Turk, M.: Multimodal human–computer interaction. In: Real-Time Vision for Human–Computer Interaction, Springer-Verlag, New York, pp. 269–283 (2005). doi:10.1007/0-387-27890-7

  70. Turk, M.: Multimodal interaction: a review. Pattern Recognit. Lett. 36(1), 189–195 (2014). doi:10.1016/j.patrec.2013.07.003

    Article  MathSciNet  Google Scholar 

  71. Von Neumann, J., Kurzweil, R.: The Computer and the Brain. Yale University Press, New Haven (2012)

    Google Scholar 

  72. Walker, F.O.: Huntington’s disease. The Lancet 369(9557), 218–228 (2007). doi:10.1016/S0140-6736(07)60111-1. 1111.6189v1

    Article  Google Scholar 

  73. Wang, D., Xiang, Z., Fesenmaier, D.R.: Smartphone use in everyday life and travel. J. Travel Res. 55(1), 52–63 (2016). doi:10.1177/0047287514535847

    Article  Google Scholar 

  74. Wheeler, K., Chang, M., Knuth, K.: Gesture-based control and EMG decomposition. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 36(4), 503–514 (2006). doi:10.1109/TSMCC.2006.875418

    Article  Google Scholar 

  75. Yang, S.W., Lin, C.S., Lin, S.K., Lee, C.H.: Design of virtual keyboard using blink control method for the severely disabled. Comput. Methods Programs Biomed. 111(2), 410–418 (2013). doi:10.1016/j.cmpb.2013.04.012

    Article  Google Scholar 

  76. Zhou, J., Rau, P.L.P., Salvendy, G.: Older adults’ use of smart phones: an investigation of the factors influencing the acceptance of new functions. Behav. Inf. Technol. 33(6), 552–560 (2014). doi:10.1080/0144929X.2013.780637

    Article  Google Scholar 

  77. Zimmerman, T.G., Lanier, J., Blanchard, C., Bryson, S., Harvill, Y.: A hand gesture interface device. ACM SIGCHI Bull. 17(SI), 189–192 (1986). doi:10.1145/30851.275628

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Secretariat of Higher Education, Science, Technology and Innovation SENESCYT of Ecuador, and the Spanish Ministry of Science and Innovation, as part of the DISPERSA Project (TIN2015-67149-C3-3-R), and by the Andalusia Research Program under the project P11-TIC-7486 co-financed by FEDER (European Regional Development Fund ERDF).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Juan A. Holgado-Terriza.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pomboza-Junez, G., Holgado-Terriza, J.A. & Medina-Medina, N. Toward the gestural interface: comparative analysis between touch user interfaces versus gesture-based user interfaces on mobile devices. Univ Access Inf Soc 18, 107–126 (2019). https://doi.org/10.1007/s10209-017-0580-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-017-0580-6

Keywords

Navigation