Abstract
Human–computer interaction is transformed as well as the technology used to achieve it. This paper discusses the ease of use of a classic touch screen user interface (TUI) in contrast to a gesture-based user interface (GBUI) applied to a mobile device. Two basic positions are used for comparison: “normal” when individual is standing and “special” when individual is lying down. The first represents a common position for healthy individuals, while the second is used to represent sick, disabled or comfortable individuals. Two software applications have been developed in order to set an interaction test for TUI and GBUI, which was performed by 25 users. The interaction tests show that GBUI has a very reasonable average accuracy of 88.5% in special position and 89.8% in normal position, while TUI presents an average accuracy of 96.5% in special position and 97.4% in normal position. Therefore, GBUI could be a complementary form of interaction to TUI that can be useful to any user, especially for cases of illness or simple comfort of the user. As study case, we have developed a software solution called SICLLE (emergency call control interactive system) that manages phone calls in a smartphone and applies the concept of gestural control tree to set a gestural navigation through a gestural language from a gestural vocabulary. An architecture for mobile devices supporting GBUI has been proposed and used in the implementation of SICLLE. SICLLE allows the personalization of the gestures, which optimizes the gestural dictionary and achieves an easier learning and use. In addition, SICLLE incorporates a hearing guide which replaces the typical visual control, reaching total acceptance in the experiment conducted with real users.
Similar content being viewed by others
Notes
MYO is a trademark of Thalmic Labs Inc.
References
Abid, M.R., Shi, F., Petriu, E.M.: Dynamic hand gesture recognition from bag-of-features and local part model. In: Proceedings of 2012 IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE 2012), IEEE, pp. 78–82 (2012). doi:10.1109/HAVE.2012.6374443
AdnanIbraheem, N., Zaman Khan, R.: Survey on various gesture recognition technologies and techniques. Int. J. Comput. Appl. 50(7), 38–44 (2012). doi:10.5120/7786-0883, http://research.ijcaonline.org/volume50/number7/pxc3880883.pdf
Al-Timemy, A.H., Bugmann, G., Escudero, J., Outram, N.: Classification of finger movements for the dexterous hand prosthesis control with surface electromyography. IEEE J. Biomed. Health Inform. 17(3), 608–618 (2013). doi:10.1109/JBHI.2013.2249590
Blazquez Cano, M., Perry, P., Ashman, R., Waite, K.: The influence of image interactivity upon user engagement when using mobile touch screens. Comput. Hum. Behav. 1–7 (2017). doi:10.1016/j.chb.2017.03.042, http://linkinghub.elsevier.com/retrieve/pii/S0747563217301991
Buchner, H., Petersen, E., Eger, M., Rostalski, P.: Convolutive blind source separation on surface EMG signals for respiratory diagnostics and medical ventilation control. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, pp. 3626–3629 (2016). doi:10.1109/EMBC.2016.7591513, http://ieeexplore.ieee.org/document/7591513/
Card, S.K., Newell, A., Moran, T.P.: The Psychology of Human–Computer Interaction. Lawrence Erlbaum Associates Inc., Hillsdale, NJ (1983)
Chen, C., Jafari, R., Kehtarnavaz, N.: Improving human action recognition using fusion of depth camera and inertial sensors. IEEE Trans. Hum. Mach. Syst. 45(1), 51–61 (2015). doi:10.1109/THMS.2014.2362520, http://ieeexplore.ieee.org/document/6934998/
Chen, X., Wang, Z.J.: Pattern recognition of number gestures based on a wireless surface EMG system. Biomed. Signal Process. Control 8(2), 184–192 (2013). doi:10.1016/j.bspc.2012.08.005
Chittaro, L.: Visualizing information on mobile devices. Computer 39(3), 40–45 (2006). doi:10.1109/MC.2006.109, http://ieeexplore.ieee.org/document/1607948/
de Waard, D., Westerhuis, F., Joling, D., Weiland, S., Stadtbäumer, R., Kaltofen, L.: Visual map and instruction-based bicycle navigation: a comparison of effects on behaviour. Ergonomics 0139(April), 1–14 (2017). doi:10.1080/00140139.2017.1282628
Dinh, D.L., Kim, J.T., Kim, T.S.: Hand gesture recognition and interface via a depth imaging sensor for smart home appliances. Energy Procedia 62(62), 576–582 (2014). doi:10.1016/j.egypro.2014.12.419
Erden, F., Çetin, A.E.: Hand gesture based remote control system using infrared sensors and a camera. IEEE Trans. Consum. Electron. 60(4), 675–680 (2014). doi:10.1109/TCE.2014.7027342
Fernandez, R.A.S., Sanchez-Lopez, J.L., Sampedro, C., Bavle, H., Molina, M., Campoy, P.: Natural user interfaces for human-drone multi-modal interaction. In: 2016 International Conference on Unmanned Aircraft Systems (ICUAS), IEEE, pp. 1013–1022 (2016). doi:10.1109/ICUAS.2016.7502665
Fitts, P.M.: The amplitude of movement. J. Exp. Psychol. 47, 381–391 (1954)
Flores-Abad, A., Ma, O., Pham, K., Ulrich, S.: A review of space robotics technologies for on-orbit servicing. Prog. Aerosp. Sci. 68, 1–26 (2014). doi:10.1016/j.paerosci.2014.03.002, http://linkinghub.elsevier.com/retrieve/pii/S0376042114000347
Flowers, K.A., Robertson, C.: The effect of Parkinson’s disease on the ability to maintain a mental set. J. Neurol. Neurosurg. Psychiatry 48(6), 517–529 (1985). doi:10.1136/jnnp.48.6.517
Francese, R., Passero, I., Tortora, G.: Wiimote and Kinect. In: Proceedings of the International Working Conference on Advanced Visual Interfaces—AVI ’12, ACM Press, New York, New York, USA, pp 116–123 (2012). doi:10.1145/2254556.2254580
Garawi, S., Istepanian, R., Abu-Rgheff, M.: 3G wireless communications for mobile robotic tele-ultrasonography systems. IEEE Commun. Mag. 44(4), 91–96 (2006). doi:10.1109/MCOM.2006.1632654
Gokgoz, E., Subasi, A.: Effect of multiscale PCA de-noising on EMG signal classification for diagnosis of neuromuscular disorders. J. Med. Syst. 38(4), 31–38 (2014). doi:10.1007/s10916-014-0031-3
Guerrero, F.N., Spinelli, E.M., Haberman, M.A.: Analysis and simple circuit design of double differential EMG active electrode. IEEE Trans. Biomed. Circuits Syst. 10(3), 787–795 (2016). doi:10.1109/TBCAS.2015.2492944
Hargrove, L., Englehart, K., Hudgins, B.: A comparison of surface and intramuscular myoelectric signal classification. IEEE Trans. Biomed. Eng. 54(5), 847–853 (2007). doi:10.1109/TBME.2006.889192, http://ieeexplore.ieee.org/document/4154997/
Hasan, H., Abdul-Kareem, S.: Static hand gesture recognition using neural networks. Artif. Intell. Rev. 41(2), 147–181 (2014). doi:10.1007/s10462-011-9303-1
Heo, H., Lee, E.C., Park, K.R., Kim, C.J., Whang, M.: A realistic game system using multi-modal user interfaces. IEEE Trans. Consum. Electron. 56(3), 1364–1372 (2010). doi:10.1109/TCE.2010.5606271
Hooper, C.J., Dix, A.: Web science and human–computer interaction. Interactions 20(3), 52–57 (2013). doi:10.1145/2451856.2451868
Hug, F.: Can muscle coordination be precisely studied by surface electromyography? J. Electromyogr. Kinesiol. 21(1), 1–12 (2011). doi:10.1016/j.jelekin.2010.08.009
Ismirle, J., OBara, I., Swierenga, S.J., Jackson, J.E.: Touchscreen voting interface design for persons with disabilities: insights from usability evaluation of mobile voting prototype. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 60(1), 780–784 (2016). doi:10.1177/1541931213601179
Jaimes, A., Sebe, N.: Multimodal humancomputer interaction: a survey. Comput. Vis. Image Underst. 108(1–2), 116–134 (2007). doi:10.1016/j.cviu.2006.10.019
Jin, Z.X, Plocher, T., Kiff, L.: Touch screen user interfaces for older adults: button size and spacing. In: Stephanidis, C. (ed.) Universal Access in Human Computer Interaction. Coping with Diversity, Lecture Notes in Computer Science, vol. 4554, Springer, Berlin, pp. 933–941, (2007). doi:10.1007/978-3-540-73279-2_104, http://www.springerlink.com/index/10.1007/978-3-540-73279-2, http://link.springer.com/10.1007/978-3-540-73279-2
Jung, Y.: What a smartphone is to me: understanding user values in using smartphones. Inf. Syst. J. 24(4), 299–321 (2014). doi:10.1111/isj.12031
Kane, S.K, Jayant, C., Wobbrock, J.O, Ladner, R.E.: Freedom to roam: a study of mobile device adoption and accessibility for people with visual and motor disabilities. In: Proceeding of the Eleventh International ACM SIGACCESS Conference on Computers and Accessibility—ASSETS ’09, ACM Press, New York, NY, USA, pp. 115–122 (2009). doi:10.1145/1639642.1639663
Kaushik, D.M., Jain, R.: Gesture based interaction NUI: an overview. Int. J. Eng. Trends Technol. 9(12), 633–636 (2014). doi:10.14445/22315381/IJETT-V9P319, https://arxiv.org/abs/1404.2364
Kim, G.J.: Human-Computer Interaction: Fundamentals and Practice. CRC Press, Boca Raton (2015)
Klboz, N.Ç., Güdükbay, U.: A hand gesture recognition technique for human–computer interaction. J. Vis. Commun. Image Represent. 28, 97–104 (2015). doi:10.1016/j.jvcir.2015.01.015, http://linkinghub.elsevier.com/retrieve/pii/S10473203
Kühnel, C., Westermann, T., Hemmert, F., Kratz, S., Müller, A., Möller, S.: I’m home: defining and evaluating a gesture set for smart-home control. Int. J. Hum. Comput. Stud. 69(11), 693–704 (2011). doi:10.1016/j.ijhcs.2011.04.005, http://linkinghub.elsevier.com/retrieve/pii/S1071581911000668
Lara, O.D., Labrador, M.A.: A survey on human activity recognition using wearable sensors. IEEE Commun. Surv. Tutor. 15(3), 1192–1209 (2013). doi:10.1109/SURV.2012.110112.00192
LaViola, J.J.: 3D gestural interaction: the state of the field. ISRN Artif. Intell. 2013(2), 1–18 (2013). doi:10.1155/2013/514641
Lee, M., Hong, Y., Lee, S., Won, J., Yang, J., Park, S., Chang, K.T., Hong, Y.: The effects of smartphone use on upper extremity muscle activity and pain threshold. J. Phys. Ther. Sci. 27(6), 1743–1745 (2015). doi:10.1589/jpts.27.1743, https://www.jstage.jst.go.jp/article/jpts/27/6/27_jpts-2015-015/_article
Leitão, R.A.: Creating mobile gesture-based interaction design patterns for older adults: a study of tap and swipe gestures with Portuguese seniors. Ph.D. thesis, Universidade do Porto (2012) http://repositorio-aberto.up.pt/handle/10216/68413
Li, G., Li, Y., Zhang, Z., Geng, Y., Zhou, R.: Selection of sampling rate for EMG pattern recognition based prosthesis control. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, IEEE, vol. 2010, pp. 5058–5061 (2010). doi:10.1109/IEMBS.2010.5626224
Li, C., Ma, H., Yang, C., Fu, M. Teleoperation of a virtual iCub robot under framework of parallel system via hand gesture recognition. In: 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), IEEE, pp 1469–1474 (2014). doi:10.1109/FUZZ-IEEE.2014.6891887, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6891887
Lv, Z., Feng, S., Feng, L., Li, H.: Extending touch-less interaction on vision based wearable device. In: 2015 IEEE Virtual Reality (VR), IEEE, pp 231–232 (2015). doi:10.1109/VR.2015.7223380, http://ieeexplore.ieee.org/document/7223380/, https://arxiv.org/abs/1504.01025
MacKenzie, I.S.: Fitts’ law as a research and design tool in human–computer interaction. Hum. Comput. Interact. 7(1):91–139 (1992). doi:10.1207/s15327051hci07013, http://portal.acm.org/citation.cfm?id=1461857
Mehler, A., Lücking, A., Abrami, G.: WikiNect: image schemata as a basis of gestural writing for kinetic museum wikis. Univers. Access Inf. Soc. 14(3), 333–349 (2015). doi:10.1007/s10209-014-0386-8
Merletti, R., Botter, A., Troiano, A., Merlo, E., Minetto, M.A.: Technology and instrumentation for detection and conditioning of the surface electromyographic signal: state of the art. Clin. Biomech. 24(2), 122–134 (2009). doi:10.1016/j.clinbiomech.2008.08.006
Min, B.W: Improvement of mobile U-health services system. Commun. Comput. Inf. Sci. 262 CCIS(PART 1), 44–51 (2011). doi:10.1007/978-3-642-27204-2_6
Mitra, S., Acharya, T.: Gesture recognition: a survey. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 37(3), 311–324 (2007). doi:10.1109/TSMCC.2007.893280, http://ieeexplore.ieee.org/document/4154947/
Nguyen, Q., Premaratne, P.: Consumer electronics control system based on hand gesture moment invariants. IET Comput. Vis. 1(1), 35–41 (2007). doi:10.1049/iet-cvi:20060198, http://ieeexplore.ieee.org/document/4159604/?arnumber=4159604
Nielsen, J.: Noncommand user interfaces. Commun. ACM 36(4), 83–99 (1993). doi:10.1145/255950.153582
Nielsen, M., Störring, M., Moeslund, T.B., Granum, E.: A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI. In: Gesture-Based Communication in Human–Computer Interaction, pp. 409–420 (2004). doi:10.1007/978-3-540-24598-8 38
Ohn-Bar, E., Trivedi, M.M.: Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans. Intell. Transp. Syst. 15(6), 2368–2377 (2014). doi:10.1109/TITS.2014.2337331, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6883176
Page, T.: Touchscreen and perceived usability: a comparison of attitudes between older and youger mobile device. I-Manag. J. Mob. Appl. Techonol. 3(1), 1–16 (2016). https://search.proquest.com/docview/1853319087?accountid=14542, http://hdl.handle.net/11025/1847
Peters, A., Morrison, J.H.: Cerebral Cortex: Neurodegenerative and Age-Related Changes in Structure and Function of Cerebral Cortex, vol. 14. Springer, Berlin (2012)
Phinyomark, A., Phukpattaranont, P., Limsakul, C.: Feature reduction and selection for EMG signal classification. Expert Syst. Appl. 39(8), 7420–7431 (2012). doi:10.1016/j.eswa.2012.01.102
Pomboza-Junez, G., Holgado-Terriza, J.A.: Control of home devices based on hand gestures. In: 2015 IEEE 5th International Conference on Consumer Electronics—Berlin (ICCE-Berlin), IEEE, pp 510-514 (2015). DOI:10.1109/ICCE-Berlin.2015.7391325
Pomboza-Junez, G., Holgado-Terriza, J.A.: Hand gesture recognition based on sEMG signals using Support Vector Machines. In: 2016 IEEE 6th International Conference on Consumer Electronics—Berlin (ICCE-Berlin), IEEE, pp. 174–178 (2016). doi:10.1109/ICCE-Berlin.2016.7684748
Premaratne, P., Ajaz, S., Premaratne, M.: Hand gesture tracking and recognition system using Lucas-Kanade algorithms for control of consumer electronics. Neurocomputing 116, 242–249 (2013). doi:10.1016/j.neucom.2011.11.039
Prieto De Lope, R.P., Medina Medina, N.: A comprehensive taxonomy for serious games. J. Educ. Comput. Res. (2016). doi:10.1177/0735633116681301
Quigley, M., Goodrich, M., Beard, R.: Semi-autonomous human–UAV interfaces for fixed-wing mini-UAVs. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. 04CH37566), IEEE, vol. 3, pp. 2457–2462 (2004). doi:10.1109/IROS.2004.1389777
Raez, M.B.I., Hussain, M.S., Mohd-Yasin, F., Reaz, M., Hussain, M.S., Mohd-Yasin, F.: Techniques of EMG signal analysis: detection, processing, classification and applications. Biol. Proced. Online 8(1), 11–35 (2006). doi:10.1251/bpo115
Rainoldi, A., Melchiorri, G., Caruso, I.: A method for positioning electrodes during surface EMG recordings in lower limb muscles. J. Neurosci. Methods 134(1), 37–43 (2004). doi:10.1016/j.jneumeth.2003.10.014
Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2015). doi:10.1007/s10462-012-9356-9
Rempel, D., Camilleri, M.J., Lee, D.L.: The design of hand gestures for human-computer interaction: Lessons from sign language interpreters. Int. J. Hum. Comput. Stud. 72(10–11), 728–735 (2014). doi:10.1016/j.ijhcs.2014.05.003
Rezende, A., Schneider, G., Prakash, P.: Method and apparatus for customizing a display screen of a user interface. https://www.google.com/patents/US9304668, US Patent 9,304,668 (2016)
Robertson, C., Flowers, K.A.: Motor set in Parkinson’s disease. J. Neurol. Neurosurg. Psychiatry 53(7), 583–592 (1990). doi:10.1136/jnnp.53.7.583
Sanchez-Nielsen, E., Hernandez-Tejera, M., Anton-Canalís, L. (2004) Hand gesture recognition for human–machine interaction. J. WSCG 12(1-3), 395–402, http://hdl.handle.net/11025/1847
Sanna, A., Lamberti, F., Paravati, G., Manuri, F.: A Kinect-based natural interface for quadrotor control. Entertain. Comput. 4(3), 179–186 (2013). doi:10.1016/j.entcom.2013.01.001
Schaller, R.: Moore’s law: past, present and future. IEEE Spectr. 34(6), 52–59 (1997). doi:10.1109/6.591665, http://ieeexplore.ieee.org/document/591665/
Stergiopoulou, E., Papamarkos, N.: Hand gesture recognition using a neural network shape fitting technique. Eng. Appl. Artif. Intell. 22(8), 1141–1158 (2009). doi:10.1016/j.engappai.2009.03.008, http://linkinghub.elsevier.com/retrieve/pii/S0952197609000694
Turk, M.: Multimodal human–computer interaction. In: Real-Time Vision for Human–Computer Interaction, Springer-Verlag, New York, pp. 269–283 (2005). doi:10.1007/0-387-27890-7
Turk, M.: Multimodal interaction: a review. Pattern Recognit. Lett. 36(1), 189–195 (2014). doi:10.1016/j.patrec.2013.07.003
Von Neumann, J., Kurzweil, R.: The Computer and the Brain. Yale University Press, New Haven (2012)
Walker, F.O.: Huntington’s disease. The Lancet 369(9557), 218–228 (2007). doi:10.1016/S0140-6736(07)60111-1. 1111.6189v1
Wang, D., Xiang, Z., Fesenmaier, D.R.: Smartphone use in everyday life and travel. J. Travel Res. 55(1), 52–63 (2016). doi:10.1177/0047287514535847
Wheeler, K., Chang, M., Knuth, K.: Gesture-based control and EMG decomposition. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 36(4), 503–514 (2006). doi:10.1109/TSMCC.2006.875418
Yang, S.W., Lin, C.S., Lin, S.K., Lee, C.H.: Design of virtual keyboard using blink control method for the severely disabled. Comput. Methods Programs Biomed. 111(2), 410–418 (2013). doi:10.1016/j.cmpb.2013.04.012
Zhou, J., Rau, P.L.P., Salvendy, G.: Older adults’ use of smart phones: an investigation of the factors influencing the acceptance of new functions. Behav. Inf. Technol. 33(6), 552–560 (2014). doi:10.1080/0144929X.2013.780637
Zimmerman, T.G., Lanier, J., Blanchard, C., Bryson, S., Harvill, Y.: A hand gesture interface device. ACM SIGCHI Bull. 17(SI), 189–192 (1986). doi:10.1145/30851.275628
Acknowledgements
This work was supported by the Secretariat of Higher Education, Science, Technology and Innovation SENESCYT of Ecuador, and the Spanish Ministry of Science and Innovation, as part of the DISPERSA Project (TIN2015-67149-C3-3-R), and by the Andalusia Research Program under the project P11-TIC-7486 co-financed by FEDER (European Regional Development Fund ERDF).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pomboza-Junez, G., Holgado-Terriza, J.A. & Medina-Medina, N. Toward the gestural interface: comparative analysis between touch user interfaces versus gesture-based user interfaces on mobile devices. Univ Access Inf Soc 18, 107–126 (2019). https://doi.org/10.1007/s10209-017-0580-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10209-017-0580-6