System Evaluation: The Statistical Basis of Biometric Systems

Some biometric systems perform well, and others perform poorly. This chapter examines in detail the methods for establishing and reporting system accuracy. The manner in which a system is evaluated depends on how it operates. There are two fundamental types of biometric systems: verification systems and identification systems. Verification systems are the subject of Sect. 7.1. The section begins with a review of statistical theory, which is at the heart of the biometric matching process. After these foundations have been laid, the discussion proceeds to develop the rates and graphs used for reporting verification system performance. This is followed by an examination of identification systems in Sect. 7.2. This section differentiates between verification and identification systems, and demonstrates how identification is actually composed of a series of verifications. Section 7.3 examines the role of chance in biometric analysis. As will be seen, the reliability of performance testing actually depends on the size of the experiment. Finally, in Sect. 7.4 other quantitative measures of system performance are explored.

By the end of this chapter, you should have a firm understanding of:
  • The statistical basis of verification decisions (Sect. 7.1.1).

  • The difference between verification and identification systems. (Sect. 7.2.1).

  • The common performance measures and graphs used for verification (Sect. 7.1) and identification (Sect. 7.2) systems, and when they are applicable.

  • The dependence of identification performance on the number of enrollments in a system (Sect.

  • The role of statistical uncertainty in biometric matching, and its impact on the way performance is measured and evaluations are conducted (Sect. 7.3).

  • Enrollment and acquisition errors, and how they impact system performance (Sect. 7.4.1). 11


Receiver Operating Characteristic Curve False Alarm Rate Candidate List Equal Error Rate Biometric System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agresti, A., Coull, B.: Approximate is better than ’exact’for intervalestimation of binomial proportions. The American Statistician 52, 119–126 (1998)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Bolle, R.M., Connell, J.H., Pankanti, S., Ratha, N.K., Senior, A.W.: The relation between the ROCcurve and the CMC.In: AUTOID ’05: Proceedings of the Fourth IEEE Workshop on Automatic Identification Advanced Technologies, pp. 15–20 (2005)Google Scholar
  3. 3.
    Bolle, R.M., Ratha, N.K., Pankanti, S.: Error analysis of pattern recognition systems: the subsets bootstrap. Comput. Vis. Image Underst. 93(1), 1–33 (2004)CrossRefGoogle Scholar
  4. 4.
    Brown, L., Cai, T., DasGupta, A.: Interval estimation for a binomial propor-tion.Statistical Science 16(2), 101–133 (2001)Google Scholar
  5. 5.
    Cappelli, R., Maio, D., Maltoni, D.: Indexing Fingerprint databases for efficient 1:N matching. In: Proceedings of ICARCV2000 (2000)Google Scholar
  6. 6.
    Clopper, C., Pearson, S.: The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26, 404–413 (1934)Google Scholar
  7. 7.
    Dass, S.C., Zhu, Y., Jain, A.: Validating a biometric authentication system: Sample size requirements. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 1902–1319 (2006)CrossRefGoogle Scholar
  8. 8.
    Fawcett, T.: ROC graphs: Notes and practical considerationsfor researchers (2004)Google Scholar
  9. 9.
    Grother, P.J., Phillips, P.J.: Models of large population recognition performance. In: CVPR04, pp. II: 68–75 (2004)Google Scholar
  10. 10.
    Hube, J.: Using biometric verification to estimate identification performance. In: Proceedings of 2006 Biometrics Symposium (2006)Google Scholar
  11. 11.
    ISO: Information technology –biometric performance testing and reporting – part 1: Principles and framework (ISO/IEC 19795-1:2006) (2006)Google Scholar
  12. 12.
    Johnson, A., Sun, J., Bobick, A.: Predicting large population data cumulative match characteristic performance from small population data. In: Proceedings of AVBPA 2003 (2003)Google Scholar
  13. 13.
    Jovanovic, B., Levy, P.: A look at the rule of three. The American Statistician. 51(2), 137–139 (1997)CrossRefGoogle Scholar
  14. 14.
    Maio, D., Maltoni, D., Cappelli, R., Wayman, J.L., Jain, A.K.: FVC2000: Fingerprint verification competition. IEEE Trans. Pattern Anal. Mach. Intell. 24(3), 402–412 (2002)CrossRefGoogle Scholar
  15. 15.
    Mansfield, A.J., Wayman, J.L.: Best practices in testingand reporting performance of biometric devices, v2.01. Tech. Rep. NPL Report CMSC 14/02, National Physical Laboratory (2002)Google Scholar
  16. 16.
    Martin, A., Doddington, G., Kamm, T., Ordowski, M., Przybocki, M.: The DET curve in assessment of detection task performance. In: Proc. Eurospeech ’97, pp. 1895–1898. Rhodes, Greece (1997)Google Scholar
  17. 17.
    Payton, M.E., Greenstone, M.H., Schenker, N.: Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance? Journal of Insect Science 3(34) (2003)Google Scholar
  18. 18.
    Phillips, P.J., Grother, P., Micheals, R., Blackburn, D., Tabassi, E., Bone, J.: FRVT 2002 evaluation report. Tech. Rep. NISTIR 6965 (2003)Google Scholar
  19. 19.
    Phillips, P.J., Scruggs, W.T., O’Toole, A.J., Flynn, P.J., Bowyer, K.W., Schott, C.L., Sharpe, M.: FRVT 2006 and ice 2006 large-scale results. Tech. Rep. NISTIR 7408, National Institute of Standards and Technology (????)Google Scholar
  20. 20.
    Reynolds, D.A., Doddington, G.R., Przybocki, M.A., Martin, A.F.: The NIST speaker recognition evaluation -overview methodology, systems, results, perspective. Speech Commun. 31(2-3), 225–254 (2000)Google Scholar
  21. 21.
    Schuckers, M.: Estimation and sample size calculations for correlated binary error rates of biometric identification rates.Proceedings of the American Statistical Association: Biometrics Section (2003)Google Scholar
  22. 22.
    Schuckers, M.: Using the beta-binomial distribution to assess performance of a biometric identification device. International Journal of Image and Graphics 3(3) (2003)Google Scholar

Copyright information

© Springer-Verlag US 2009

Personalised recommendations