Nowadays, authentication is required for both physical access to buildings and internal access to computers and systems. Biometrics are one of the emerging technologies used to protect these highly sensitive structures. However, biometric systems based on a single trait enclose several problems such as noise sensitivity and vulnerability to spoof attacks. In this regard, we present in this paper a fully unobtrusive and robust multimodal authentication system that automatically authenticates a user by the way he/she answers his/her phone, after extracting ear and arm gesture biometric modalities from this single action. To deal the challenges facing ear and arm gesture authentication systems in real-world applications, we propose a new method based on image fragmentation that makes the ear recognition more robust in relation to occlusion. The ear feature extraction process has been made locally using Local Phase Quantization (LPQ) in order to get robustness with respect to pose and illumination variation. We also propose a set of four statistical metrics to extract features from arm gesture signals. The two modalities are combined on score-level using a weighted sum. In order to evaluate our contribution, we conducted a set of experiments to demonstrate the contribution of each of the two biometrics and the advantage of their fusion on the overall performance of the system. The multimodal biometric system achieves an equal error rate (EER) of 5.15%.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Abate AF, Nappi M, Ricciardi S (2019) I-Am: Implicitly authenticate me person authentication on mobile devices through ear shape and arm gesture. IEEE Trans Syst Man Cybern Syst 49(3):469–481. https://doi.org/10.1109/TSMC.2017.2698258
Abozaid A, Haggag A, Kasban H, Eltokhy M (2018) Multimodal biometric scheme for human authentication technique based on voice and face recognition fusion. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-7012-3
Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: Application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041. https://doi.org/10.1109/TPAMI.2006.244
Akhtar Z, Sandeep K, Nasir A (2011) Spoof attacks on multimodal biometric systems. Int Conf Inf Netw Technol 4:46–51
Akhtar Z, Buriro A, Crispo B, Falk T. H. (2017) Multimodal smartphone user authentication using touchstroke, phone-movement and face patterns. In: Global conference on signal and information processing (GlobalSIP), pp 1368–1372. https://doi.org/10.1109/GlobalSIP.2017.8309185
Aronowitz H, Li M, Toledo-Ronen O, Harary S, Geva A, Ben-David S, Rendel A, Hoory R, Ratha N, Pankanti S, Nahamoo D (2014) Multi-modal biometrics for mobile authentication. IEEE Int Jt Conf Biom, pp 1–8. https://doi.org/10.1109/BTAS.2014.6996269
Arteaga-Falconi JS, Al Osman H, El Saddik A (2018) ECG and fingerprint bimodal authentication. Sustain Cities Soc 40:274–283. https://doi.org/10.1016/j.scs.2017.12.023
Barra S, Fenu G, De Marsico M, Castiglione A, Nappi M (2018) Have you permission to answer this phone?. In: 2018 International workshop on biometrics and forensics (IWBF), pp 1–7. https://doi.org/10.1109/IWBF.2018.8401563
Buriro A, Crispo B, Zhauniarovich Y (2017) Please hold on: Unobtrusive user authentication using smartphone’s built-in sensors. In: IEEE International conference on identity, security and behavior analysis (ISBA). https://doi.org/10.1109/ISBA.2017.7947684. IEEE, New Delhi India, pp 1–8
Chhabria SA, Dharaskar RV, Thakare VM (2013) Survey of fusion techniques for design of efficient multimodal systems. In: International conference on machine intelligence research and advancement, pp 486–492. https://doi.org/10.1109/ICMIRA.2013.103
Dhvani S, Vinayak H (2016) IoT based biometrics implementation on raspberry Pi. Procedia Comput Sci 79:328–336. https://doi.org/10.1016/j.procs.2016.03.043
Dornaika F, Moujahid A, El Merabet Y, Ruichek Y (2017) A comparative study of image segmentation algorithms and descriptors for building detection. In: Handbook of neural computation, pp 591–606. https://doi.org/10.1016/B978-0-12-811318-9.00032-6
Emersic Z, Struc V, Peer P (2017) Ear recognition: More than a survey. Neurocomputing 255:26–39. https://doi.org/10.1016/j.neucom.2016.08.139
Gofman M, Mitra S (2016) Multimodal biometrics for enhanced mobile device security. Commun ACM 59(4):58–65. https://doi.org/10.1145/2818990
Gofman M, Mitra S, Cheng K, Smith N (2015) Quality-based score-level fusion for secure and robust multimodal biometrics-based authentication on consumer mobile devices. In: ICSEA 2015 : The tenth international conference on software engineering advances, pp 274–276
Gofman M, Mitra S, Smith N (2016) Hidden Markov models for feature-level fusion of biometrics on mobile devices. In: IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), pp 1–2. https://doi.org/10.1109/AICCSA.2016.7945755
Gofman M, Sandico N, Mitra S, Suo E, Muhi S, Vu T (2018) multimodal biometrics via discriminant correlation analysis on mobile devices. In: Proceedings of the international conference on security and management (SAM). The steering committee of The World congress in computer science, Computer engineering and applied computing (WorldComp)
Habib K, Arild T, Leister W (2014) A novel authentication framework based on biometric and radio fingerprinting for the IoT in eHealth. In: Proceedings of international conference on smart systems devices and technologies (SMART), pp 32–37. https://doi.org/10.13140/2.1.3944.1286
Introduction to USTB ear image databases (2002) University of Science and Technology of Beijing, http://www1.ustb.edu.cn/resb/en/doc/Imagedb123introen.pdf
Kandgaonkar TV, Mente RS, Shinde AR, Raut SD (2015) Ear biometrics: A survey on ear image databases and techniques for ear detection and recognition. IBMRD’s J Manag Res 4(1):88–103. https://doi.org/10.17697/ibmrd/2015/v4i1/60357
Kannala J, Rahtu E (2012) BSIf: binarized statistical image features. In: Proceedings of the International Conference on Pattern Recognition (ICPR), pp 1363–1366
Kim DJ, Kil-Ram H, Kwang-Seok H (2008) An implementation of multimodal speaker verification system using teeth image and voice on mobile environment. J Inst Electron Eng Korea CI 45(5):162–172
Krizaj J, Struc V, Pavesic N (2010) Adaptation of SIFT features for robust face recognition. Int Conf Image Anal Recog, pp 394–404. https://doi.org/10.1007/978-3-642-13772-3∖_40
Kumar A, Wu C (2012) Automated human identification using ear imaging. Pattern Recognit 45(3):956–968. https://doi.org/10.1016/j.patcog.2011.06.005
Macek N, Franc I, Bogdanoski M, Mirkovic A (2016) Multimodal biometric authentication in IoT: Single camera case study. In: 8th International conference on business information security (BISEC’2016), pp 33–37
Mahmoud RO, Selim MM, Muhi OA (2020) Fusion time reduction of a feature level based multimodal biometric authentication system. Int J Sociotechnol Knowl Dev 12(1):67–83. https://doi.org/10.4018/IJSKD.2020010104
Malhotra A, Sankaran A, Mittal A, Vatsa M, Singh R (2017) Chapter 6 - Fingerphoto authentication using smartphone camera captured under varying environmental conditions. In: Human recognition in unconstrained environments, pp 119–144. https://doi.org/10.1016/B978-0-08-100705-1.00006-3
Manjunath BS, Ma WY (1996) Texture features for browsing and retrieval of image data. IEEE Trans Pattern Anal Mach Intell 18(8):837–848. https://doi.org/10.1109/34.531803
Mohanta TK, Mohapatra S (2014) Development of multimodal biometric framework for smartphone authentication system. Int J Comput Appl 102 (7):6–11. https://doi.org/10.5120/17825-8597
Ojala T, Pietikainen M, Harwood D (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recogn 29(1):51–59. https://doi.org/10.1016/0031-3203(95)00067-4
Ojansivu V, Heikkila J (2008) Blur insensitive texture classification using local phase quantization. In: Image and Signal Processing: 3rd International Conference, ICISP 2008. https://doi.org/10.1007/978-3-540-69905-7_27. Springer, Berlin, pp 236–243
Ojansivu V, Rahtu E, Heikkila J (2008) Rotation invariant local phase quantization for blur insensitive texture analysis. In: 19th International conference on pattern recognition, pp 236–243. https://doi.org/10.1109/ICPR.2008.4761377
Olazabal O, Gofman M, Bai Y, Choi Y, Sandico N, Mitra S, Pham K (2019) Multimodal Biometrics for Enhanced IoT Security. In: 2019 IEEE 9th annual computing and communication workshop and conference (CCWC), pp 886–893. https://doi.org/10.1109/CCWC.2019.8666599
Paul PP, Gavrilova ML, Alhajj R (2014) Decision fusion for multimodal biometrics using social network analysis. IEEE Trans Syst Man Cybern Syst 44(11):1522–1533. https://doi.org/10.1109/TSMC.2014.2331920
Pflug A, Paul PN, Busch C (2014) A comparative study on texture and surface descriptors for ear biometrics. In: 2014 International carnahan conference on security technology (ICCST), pp 1–8. https://doi.org/10.1109/BTAS.2014.6996239
Pflug A, Busch C, Ross A (2014) 2D ear classification based on unsupervised clustering. In: Proceedings of the international joint conference on biometrics, pp 1–8. https://doi.org/10.1109/BTAS.2014.6996239
Rodrigues RN, Lee LL, Venu G (2009) Robustness of multimodal biometric fusion methods against spoof attacks. J Vis Lang Comput 20(3):169–179. https://doi.org/10.1016/j.jvlc.2009.01.010
Rodrigues RN, Niranjan K, Venu G (2010) Evaluation of biometric spoofing in a multimodal system. In: 2010 Fourth IEEE international conference on biometrics: Theory applications and systems (BTAS). https://doi.org/10.1109/BTAS.2010.5634531
Sequeira AF, Monteiro JC, Rebelo A, Oliveira HP (2014) MobBIO: A multimodal database captured with a portable handheld device. In: 2014 International conference on computer vision theory and applications (VISAPP), vol 3
Sitova Z, Sedenka J, Yang Q, Peng G, Zhou G, Gasti P, Balagani KS (2016) HMOG: New behavioral biometric features for continuous authentication of smartphone users. IEEE Trans Inf Forensic Secur 11 (5):877–892. https://doi.org/10.1109/TIFS.2015.2506542
Vu NS, Caplier A (2010) Face recognition with patterns of oriented edge magnitudes. Comp Vision pp 313–326. https://doi.org/10.1007/978-3-642-15549-9∖_23
Wu L, Yang J, Zhou M, Chen Y, Wang Q (2020) LVID: A multimodal biometrics authenticationsystem on smartphones. IEEE Trans Inf Forensic Secur 15:1–6. https://doi.org/10.1109/TIFS.2019.2944058
Zhang Q, Li H, Sun Z, Tan T (2018) Deep feature fusion for iris and periocular biometrics on mobile devices. IEEE Trans Inf Forensic Secur 13(11):2897–2912. https://doi.org/10.1109/TIFS.2018.2833033
Zhang Y, Mu Z, Yuan L, Yu C (2018) Ear verification under uncontrolled conditions with convolutional neural networks. IET Biometrics 7(3):185–198. https://doi.org/10.1049/iet-bmt.2017.0176
This work was carried out in the framework of research activities of the laboratory LIMED, which is affiliated to the Faculty of Exact Sciences of the University of Bejaia, with collaboration with LIGM, University of Gustave-Eiffel.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Conflict of interests
The authors declare that they have no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Cherifi, F., Amroun, K. & Omar, M. Robust multimodal biometric authentication on IoT device through ear shape and arm gesture. Multimed Tools Appl (2021). https://doi.org/10.1007/s11042-021-10524-9
- Multimodal authentication
- Arm gesture
- Score-level fusion