Evaluation Methodologies

  • Ivana ChingovskaEmail author
  • André Anjos
  • Sébastien Marcel
Part of the Advances in Computer Vision and Pattern Recognition book series (ACVPR)


Anti-spoofing systems, regardless of the technique, biometric mode or degree of independence of external equipment, are most commonly treated as binary classification systems. The two classes that they differentiate are genuine accesses and spoofing attacks. From this perspective, their evaluation is equivalent to the established evaluation standards for the binary classification systems. However, the anti-spoofing systems are designed to operate in conjunction with recognition systems and as such can affect their performance. From the point of view of a recognition system, the spoofing attacks are a separate class that they need to detect and reject. As the problem of spoofing attacks detection grows to this pseudo-ternary status, the evaluation methodologies for the recognition systems need to be revised and updated. Consequentially, the database requirements for spoofing databases become more specific. The focus of this chapter is the task of biometric verification and its scope is threefold: first, it gives the definition of the spoofing detection problem from the two perspectives. Second, it states the database requirements for a fair and unbiased evaluation. Finally, it gives an overview of the existing evaluation techniques for anti-spoofing systems and verification systems under spoofing attacks.


Spoofing Attack Binary Classification System Genuine Access Spoofing Databases Biometric Verification System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors would like to thank the projects BEAT ( and TABULA RASA ( both funded under the 7th Framework Programme of the European Union (EU) (grant agreement number 284989 and 257289) respectively.


  1. 1.
    Rodrigues RN, Ling LL, Govindaraju V (2009) Robustness of multimodal biometric fusion methods against spoofing attacks. J Vis Lang Computing 20(3):169–179CrossRefGoogle Scholar
  2. 2.
    Johnson PA, Tan B, Schuckers S (2010) Multimodal fusion vulnerability to non-zero (spoof) imposters. In: IEEE international workshop information forensics and securityGoogle Scholar
  3. 3.
    Akhtar Z, Fumera G, Marcialis GL, Roli F (2011) Robustness evaluation of biometric systems under spoof attacks. In: 16th International conference on image analysis and processing, pp 159–168Google Scholar
  4. 4.
    Akhtar Z, Fumera G, Marcialis GL, Roli F (2011) Robustness analysis of likelihood ration score fusion rule for multi-modal biometric systems under spoof attacks. In: 45th IEEE international carnahan conference on security technology, pp 237–244Google Scholar
  5. 5.
    Akhtar Z, Fumera G, Marcialis GL, Roli F (2012) Evaluation of serial and parallel multibiometric systems under spoofing attacks. In: 5th IEEE international conference on biometrics: theory, applications and systemsGoogle Scholar
  6. 6.
    Villalba J, Lleida E (2011) Preventing replay attacks on speaker verification systems. In: 2011 IEEE international carnahan conference on security technology (ICCST), pp 1–8Google Scholar
  7. 7.
    Marasco E, Johnson P, Sansone C, Schuckers, S (2011) Increase the security of multibiometric systems by incorporating a spoofing detection algorithm in the fusion mechanism. In: Proceedings of the 10th international conference on multiple classifier systems, pp 309–318Google Scholar
  8. 8.
    Marasco E, Ding Y, Ross A (2012) Combining match scores with liveness values in a fingerprint verification system. In: 5th IEEE international conference on biometrics: theory, applications and systemsGoogle Scholar
  9. 9.
    Chingovska I, Anjos A, Marcel S (2013) Anti-spoofing in action: joint operation with a verification system. In: Proceedings of IEEE conference on computer vision and pattern recognition, workshop on biometricsGoogle Scholar
  10. 10.
    Pan G, Sun L, Wu Z, Lao S (2007) Eyeblink-based anti-spoofing in face recognition from a generic webcamera. In: IEEE 11th international conference on computer vision, 2007. ICCV 2007, pp 1–8Google Scholar
  11. 11.
    Bao W, Li H, Li N, Jiang W (2009) A liveness detection method for face recognition based on optical flow field. In: International conference on Image analysis and signal processing, 2009. IASP 2009, pp 233–236Google Scholar
  12. 12.
    yan J, Zhang Z, Lei Z, Yi D, Li SZ (2012) Face liveness detection by exploring multiple scenic clues. In: 12th International conference on control, automation, robotics and vision (ICARCV 2012), ChinaGoogle Scholar
  13. 13.
    Tan X, Li Y, Liu J, Jiang L (2010) Face liveness detection from a single image with sparse low rank bilinear discriminative model. In: Proceedings of European conference on computer vision (ECCV), LNCS, vol 6316. Springer, pp 504–517Google Scholar
  14. 14.
    Galbally J, Alonso-Fernandez F, Fierrez J, Ortega-Garcia J (2012) A high performance fingerprint liveness detection method based on quality related features. Future Gener Comput Syst 28(1):311–321CrossRefGoogle Scholar
  15. 15.
    Yambay D, Ghiani L, Denti P, Marcialis G, Roli F, Schuckers S (2012) LivDet 2011—fingerprint liveness detection competition 2011. In: 2012 5th IAPR international conference on biometrics (ICB), pp 208–215Google Scholar
  16. 16.
    Mansfield AJ, Wayman JL, Dr Rayner D, Wayman JL (2002) Best practices in testing and reporting performance. NPL Report CMSCGoogle Scholar
  17. 17.
    Galbally-Herrero J, Fierrez-Aguilar J, Rodriguez-Gonzalez JD, Alonso-Fernandez F, Ortega-Garcia J, Tapiador M (2006) On the vulnerability of fingerprint verification systems to fake fingerprints attacks. In: IEEE international carnahan conference on security technology, pp 169–179Google Scholar
  18. 18.
    Ruiz-Albacete V, Tome-Gonzalez P, Alonso-Fernandez F, Galbally J, Fierrez J, Ortega-Garcia J (2008) Direct attacks using fake images in iris verification. In: Proceedings of COST 2101 workshop on biometrics and identity management, BIOID, Springer, pp 181–190Google Scholar
  19. 19.
    Galbally J, Fierrez J, Alonso-Fernandez F, Martinez-Diaz M (2011) Evaluation of direct attacks to fingerprint verification systems. Telecommunication systems, special issue on biometrics 47(3):243–254CrossRefGoogle Scholar
  20. 20.
    Hastie T, Tibshirani R, Friedman JH (2001) The elements of statistical learning: data mining, inference, and prediction: with 200 full-color illustrations. Springer, New YorkGoogle Scholar
  21. 21.
    Lui YM, Bolme D, Phillips P, Beveridge J, Draper B (2012) Preliminary studies on the good, the bad, and the ugly face recognition challenge problem. In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW), pp 9–16Google Scholar
  22. 22.
    Marcialis GL, Lewicke A, Tan B, Coli P, Grimberg D, Congiu A, Tidu A, Roli F, Schuckers S (2009) First international fingerprint liveness detection competition - livdet 2009. In: Proceeding of IAPR international conference on image analysis and processing (ICIAP), LNCS, vol 5716. pp 12–23Google Scholar
  23. 23.
    Ghiani L, Yambay D, Mura V, Tocco S, Marcialis G, Roli F, Schuckers S (2013) Livdet 2013—fingerprint liveness detection competition. In: IEEE International conference on biometrics (ICB)Google Scholar
  24. 24.
    Zhiwei Z, Yan J, Liu S, Lei Z, Yi D, Li SZ (2012) A face antispoofing database with diverse attacks. In: Proceedings of IAPR international conference on biometrics (ICB), pp 26–31Google Scholar
  25. 25.
    Chingovska I, Anjos A, Marcel S (2012) On the effectiveness of local binary patterns in face anti-spoofing. In: Proceedings of IEEE international conference of the biometrics special interest group (BIOSIG), pp 1–7Google Scholar
  26. 26.
    Peixoto B, Michelassi C, Rocha A (2011) Face liveness detection under bad illumination conditions. In: 2011 18th IEEE International conference on image processing (ICIP), pp 3557–3560Google Scholar
  27. 27.
    Poh N, Bengio S (2006) Database, protocols and tools for evaluating score-level fusion algorithms in biometric authentication. Pattern Recognit J 39(2):223–233CrossRefGoogle Scholar
  28. 28.
    Martin A, Doddington G, Kamm T, Ordowski M (1997) The det curve in assessment of detection task performance. In: Proceedings of eurospeech, pp 1895–1898Google Scholar
  29. 29.
    Bengio S, Keller M, Mariéthoz J (2003) The expected performance curve. Technical report IDIAP-RR-85-2003, IDIAPGoogle Scholar
  30. 30.
    Wang L, Ding X, Fang C (2009) Face live detection method based on physiological motion analysis. Tsinghua Sci Technol 14(6):685–690CrossRefGoogle Scholar
  31. 31.
    Zhang Z, Yi D, Lei Z, Li S (2011) Face liveness detection by learning multispectral reflectance distributions. In: 2011 IEEE international conference on automatic face gesture recognition and workshops (FG 2011), pp 436–441Google Scholar
  32. 32.
    Johnson P, Lazarick R, Marasco E, Newton E, Ross A, Schuckers S (2012) Biometric liveness detection: framework and metrics. In: International biometric performance conferenceGoogle Scholar
  33. 33.
    Gao X, Tsong Ng T, Qiu B, Chang SF (2010) Single-view recaptured image detection based on physics-based features. In: IEEE international conference on multimedia and expo (ICME), SingaporeGoogle Scholar
  34. 34.
    Tronci R, Muntoni D, Fadda G, Pili M, Sirena N, Murgia G, Ristori M, Ricerche S, Roli F (2011) Fusion of multiple clues for photo-attack detection in face recognition systems. In: Proceedings of the 2011 international joint conference on biometrics, IJCB ’11, IEEE Computer Society, pp 1–6Google Scholar
  35. 35.
    Jain AK, Flynn P, Ross AA (eds) (2008) Handbook of biometrics. Springer, New YorkGoogle Scholar
  36. 36.
    Adler A, Schuckers S (2009) Security and liveness, overview. In: Jain AK, Li SZ (eds) Encyclopedia of biometrics. Springer, New YorkGoogle Scholar
  37. 37.
    Galbally J, Cappelli R, Lumini A, de Rivera GG, Maltoni D, Fiérrez J, Ortega-Garcia J, Maio D (2010) An evaluation of direct attacks using fake fingers generated from iso templates. Pattern Recogn Lett 31(8):725–732CrossRefGoogle Scholar
  38. 38.
    Rodrigues R, Kamat N, Govindaraju V (2010) Evaluation of biometric spoofing in a multimodal system. In: 2010 Fourth IEEE international conference on biometrics: theory applications and systems (BTAS)Google Scholar
  39. 39.
    Matsumoto T, Matsumoto H, Yamada K, Hoshino S (2002) Impact of artifical “gummy” fingers on fingerprint systems. In: SPIE Proceedings: optical security and counterfeit deterrence techniques, vol. 4677Google Scholar
  40. 40.
    Patrick P, Aversano G, Blouet R, Charbit M, Chollet G (2005) Voice forgery using alisp: indexation in a client memory. In: Proceedings. (ICASSP ’05). IEEE international conference on acoustics, speech, and signal processing, 2005, vol 1, pp 17–20Google Scholar
  41. 41.
    Alegre F, Vipperla R, Evans N, Fauve B (2012) On the vulnerability of automatic speaker recognition to spoofing attacks with artificial signals. In: Signal processing conference (EUSIPCO), 2012 proceedings of the 20th European, pp 36–40Google Scholar
  42. 42.
    Bonastre JF, Matrouf D, Fredouille C (2007) Artificial impostor voice transformation effects on false acceptance rates. In: INTERSPEECH, pp 2053–2056Google Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  • Ivana Chingovska
    • 1
    Email author
  • André Anjos
    • 1
  • Sébastien Marcel
    • 1
  1. 1.Idiap Research InstituteMartignySwitzerland

Personalised recommendations