Robust multimodal biometric authentication on IoT device through ear shape and arm gesture


Nowadays, authentication is required for both physical access to buildings and internal access to computers and systems. Biometrics are one of the emerging technologies used to protect these highly sensitive structures. However, biometric systems based on a single trait enclose several problems such as noise sensitivity and vulnerability to spoof attacks. In this regard, we present in this paper a fully unobtrusive and robust multimodal authentication system that automatically authenticates a user by the way he/she answers his/her phone, after extracting ear and arm gesture biometric modalities from this single action. To deal the challenges facing ear and arm gesture authentication systems in real-world applications, we propose a new method based on image fragmentation that makes the ear recognition more robust in relation to occlusion. The ear feature extraction process has been made locally using Local Phase Quantization (LPQ) in order to get robustness with respect to pose and illumination variation. We also propose a set of four statistical metrics to extract features from arm gesture signals. The two modalities are combined on score-level using a weighted sum. In order to evaluate our contribution, we conducted a set of experiments to demonstrate the contribution of each of the two biometrics and the advantage of their fusion on the overall performance of the system. The multimodal biometric system achieves an equal error rate (EER) of 5.15%.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. 1.

    Abate AF, Nappi M, Ricciardi S (2019) I-Am: Implicitly authenticate me person authentication on mobile devices through ear shape and arm gesture. IEEE Trans Syst Man Cybern Syst 49(3):469–481.

    Article  Google Scholar 

  2. 2.

    Abozaid A, Haggag A, Kasban H, Eltokhy M (2018) Multimodal biometric scheme for human authentication technique based on voice and face recognition fusion. Multimed Tools Appl.

  3. 3.

    Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: Application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041.

    Article  Google Scholar 

  4. 4.

    Akhtar Z, Sandeep K, Nasir A (2011) Spoof attacks on multimodal biometric systems. Int Conf Inf Netw Technol 4:46–51

    Google Scholar 

  5. 5.

    Akhtar Z, Buriro A, Crispo B, Falk T. H. (2017) Multimodal smartphone user authentication using touchstroke, phone-movement and face patterns. In: Global conference on signal and information processing (GlobalSIP), pp 1368–1372.

  6. 6.

    Aronowitz H, Li M, Toledo-Ronen O, Harary S, Geva A, Ben-David S, Rendel A, Hoory R, Ratha N, Pankanti S, Nahamoo D (2014) Multi-modal biometrics for mobile authentication. IEEE Int Jt Conf Biom, pp 1–8.

  7. 7.

    Arteaga-Falconi JS, Al Osman H, El Saddik A (2018) ECG and fingerprint bimodal authentication. Sustain Cities Soc 40:274–283.

    Article  Google Scholar 

  8. 8.

    Barra S, Fenu G, De Marsico M, Castiglione A, Nappi M (2018) Have you permission to answer this phone?. In: 2018 International workshop on biometrics and forensics (IWBF), pp 1–7.

  9. 9.

    Buriro A, Crispo B, Zhauniarovich Y (2017) Please hold on: Unobtrusive user authentication using smartphone’s built-in sensors. In: IEEE International conference on identity, security and behavior analysis (ISBA). IEEE, New Delhi India, pp 1–8

  10. 10.

    Chhabria SA, Dharaskar RV, Thakare VM (2013) Survey of fusion techniques for design of efficient multimodal systems. In: International conference on machine intelligence research and advancement, pp 486–492.

  11. 11.

    Dhvani S, Vinayak H (2016) IoT based biometrics implementation on raspberry Pi. Procedia Comput Sci 79:328–336.

    Article  Google Scholar 

  12. 12.

    Dornaika F, Moujahid A, El Merabet Y, Ruichek Y (2017) A comparative study of image segmentation algorithms and descriptors for building detection. In: Handbook of neural computation, pp 591–606.

  13. 13.

    Emersic Z, Struc V, Peer P (2017) Ear recognition: More than a survey. Neurocomputing 255:26–39.

    Article  Google Scholar 

  14. 14.

    Gofman M, Mitra S (2016) Multimodal biometrics for enhanced mobile device security. Commun ACM 59(4):58–65.

    Article  Google Scholar 

  15. 15.

    Gofman M, Mitra S, Cheng K, Smith N (2015) Quality-based score-level fusion for secure and robust multimodal biometrics-based authentication on consumer mobile devices. In: ICSEA 2015 : The tenth international conference on software engineering advances, pp 274–276

  16. 16.

    Gofman M, Mitra S, Smith N (2016) Hidden Markov models for feature-level fusion of biometrics on mobile devices. In: IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), pp 1–2.

  17. 17.

    Gofman M, Sandico N, Mitra S, Suo E, Muhi S, Vu T (2018) multimodal biometrics via discriminant correlation analysis on mobile devices. In: Proceedings of the international conference on security and management (SAM). The steering committee of The World congress in computer science, Computer engineering and applied computing (WorldComp)

  18. 18.

    Habib K, Arild T, Leister W (2014) A novel authentication framework based on biometric and radio fingerprinting for the IoT in eHealth. In: Proceedings of international conference on smart systems devices and technologies (SMART), pp 32–37.

  19. 19.

    Introduction to USTB ear image databases (2002) University of Science and Technology of Beijing,

  20. 20.

    Kandgaonkar TV, Mente RS, Shinde AR, Raut SD (2015) Ear biometrics: A survey on ear image databases and techniques for ear detection and recognition. IBMRD’s J Manag Res 4(1):88–103.

    Google Scholar 

  21. 21.

    Kannala J, Rahtu E (2012) BSIf: binarized statistical image features. In: Proceedings of the International Conference on Pattern Recognition (ICPR), pp 1363–1366

  22. 22.

    Kim DJ, Kil-Ram H, Kwang-Seok H (2008) An implementation of multimodal speaker verification system using teeth image and voice on mobile environment. J Inst Electron Eng Korea CI 45(5):162–172

    Google Scholar 

  23. 23.

    Krizaj J, Struc V, Pavesic N (2010) Adaptation of SIFT features for robust face recognition. Int Conf Image Anal Recog, pp 394–404.∖_40

  24. 24.

    Kumar A, Wu C (2012) Automated human identification using ear imaging. Pattern Recognit 45(3):956–968.

    Article  Google Scholar 

  25. 25.

    Macek N, Franc I, Bogdanoski M, Mirkovic A (2016) Multimodal biometric authentication in IoT: Single camera case study. In: 8th International conference on business information security (BISEC’2016), pp 33–37

  26. 26.

    Mahmoud RO, Selim MM, Muhi OA (2020) Fusion time reduction of a feature level based multimodal biometric authentication system. Int J Sociotechnol Knowl Dev 12(1):67–83.

    Article  Google Scholar 

  27. 27.

    Malhotra A, Sankaran A, Mittal A, Vatsa M, Singh R (2017) Chapter 6 - Fingerphoto authentication using smartphone camera captured under varying environmental conditions. In: Human recognition in unconstrained environments, pp 119–144.

  28. 28.

    Manjunath BS, Ma WY (1996) Texture features for browsing and retrieval of image data. IEEE Trans Pattern Anal Mach Intell 18(8):837–848.

    Article  Google Scholar 

  29. 29.

    Mohanta TK, Mohapatra S (2014) Development of multimodal biometric framework for smartphone authentication system. Int J Comput Appl 102 (7):6–11.

    Google Scholar 

  30. 30.

    Ojala T, Pietikainen M, Harwood D (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recogn 29(1):51–59.

    Article  Google Scholar 

  31. 31.

    Ojansivu V, Heikkila J (2008) Blur insensitive texture classification using local phase quantization. In: Image and Signal Processing: 3rd International Conference, ICISP 2008. Springer, Berlin, pp 236–243

  32. 32.

    Ojansivu V, Rahtu E, Heikkila J (2008) Rotation invariant local phase quantization for blur insensitive texture analysis. In: 19th International conference on pattern recognition, pp 236–243.

  33. 33.

    Olazabal O, Gofman M, Bai Y, Choi Y, Sandico N, Mitra S, Pham K (2019) Multimodal Biometrics for Enhanced IoT Security. In: 2019 IEEE 9th annual computing and communication workshop and conference (CCWC), pp 886–893.

  34. 34.

    Paul PP, Gavrilova ML, Alhajj R (2014) Decision fusion for multimodal biometrics using social network analysis. IEEE Trans Syst Man Cybern Syst 44(11):1522–1533.

    Article  Google Scholar 

  35. 35.

    Pflug A, Paul PN, Busch C (2014) A comparative study on texture and surface descriptors for ear biometrics. In: 2014 International carnahan conference on security technology (ICCST), pp 1–8.

  36. 36.

    Pflug A, Busch C, Ross A (2014) 2D ear classification based on unsupervised clustering. In: Proceedings of the international joint conference on biometrics, pp 1–8.

  37. 37.

    Rodrigues RN, Lee LL, Venu G (2009) Robustness of multimodal biometric fusion methods against spoof attacks. J Vis Lang Comput 20(3):169–179.

    Article  Google Scholar 

  38. 38.

    Rodrigues RN, Niranjan K, Venu G (2010) Evaluation of biometric spoofing in a multimodal system. In: 2010 Fourth IEEE international conference on biometrics: Theory applications and systems (BTAS).

  39. 39.

    Sequeira AF, Monteiro JC, Rebelo A, Oliveira HP (2014) MobBIO: A multimodal database captured with a portable handheld device. In: 2014 International conference on computer vision theory and applications (VISAPP), vol 3

  40. 40.

    Sitova Z, Sedenka J, Yang Q, Peng G, Zhou G, Gasti P, Balagani KS (2016) HMOG: New behavioral biometric features for continuous authentication of smartphone users. IEEE Trans Inf Forensic Secur 11 (5):877–892.

    Article  Google Scholar 

  41. 41.

    Vu NS, Caplier A (2010) Face recognition with patterns of oriented edge magnitudes. Comp Vision pp 313–326.∖_23

  42. 42.

    Wu L, Yang J, Zhou M, Chen Y, Wang Q (2020) LVID: A multimodal biometrics authenticationsystem on smartphones. IEEE Trans Inf Forensic Secur 15:1–6.

    Article  Google Scholar 

  43. 43.

    Zhang Q, Li H, Sun Z, Tan T (2018) Deep feature fusion for iris and periocular biometrics on mobile devices. IEEE Trans Inf Forensic Secur 13(11):2897–2912.

    Article  Google Scholar 

  44. 44.

    Zhang Y, Mu Z, Yuan L, Yu C (2018) Ear verification under uncontrolled conditions with convolutional neural networks. IET Biometrics 7(3):185–198.

    Article  Google Scholar 

Download references


This work was carried out in the framework of research activities of the laboratory LIMED, which is affiliated to the Faculty of Exact Sciences of the University of Bejaia, with collaboration with LIGM, University of Gustave-Eiffel.


This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information



Corresponding author

Correspondence to Feriel Cherifi.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cherifi, F., Amroun, K. & Omar, M. Robust multimodal biometric authentication on IoT device through ear shape and arm gesture. Multimed Tools Appl (2021).

Download citation


  • Biometrics
  • Multimodal authentication
  • Ear
  • Arm gesture
  • Score-level fusion