System Evaluation: The Statistical Basis of Biometric Systems
Some biometric systems perform well, and others perform poorly. This chapter examines in detail the methods for establishing and reporting system accuracy. The manner in which a system is evaluated depends on how it operates. There are two fundamental types of biometric systems: verification systems and identification systems. Verification systems are the subject of Sect. 7.1. The section begins with a review of statistical theory, which is at the heart of the biometric matching process. After these foundations have been laid, the discussion proceeds to develop the rates and graphs used for reporting verification system performance. This is followed by an examination of identification systems in Sect. 7.2. This section differentiates between verification and identification systems, and demonstrates how identification is actually composed of a series of verifications. Section 7.3 examines the role of chance in biometric analysis. As will be seen, the reliability of performance testing actually depends on the size of the experiment. Finally, in Sect. 7.4 other quantitative measures of system performance are explored.
The statistical basis of verification decisions (Sect. 7.1.1).
The difference between verification and identification systems. (Sect. 7.2.1).
The common performance measures and graphs used for verification (Sect. 7.1) and identification (Sect. 7.2) systems, and when they are applicable.
The dependence of identification performance on the number of enrollments in a system (Sect. 220.127.116.11).
The role of statistical uncertainty in biometric matching, and its impact on the way performance is measured and evaluations are conducted (Sect. 7.3).
Enrollment and acquisition errors, and how they impact system performance (Sect. 7.4.1). 11
KeywordsReceiver Operating Characteristic Curve False Alarm Rate Candidate List Equal Error Rate Biometric System
Unable to display preview. Download preview PDF.
- 2.Bolle, R.M., Connell, J.H., Pankanti, S., Ratha, N.K., Senior, A.W.: The relation between the ROCcurve and the CMC.In: AUTOID ’05: Proceedings of the Fourth IEEE Workshop on Automatic Identification Advanced Technologies, pp. 15–20 (2005)Google Scholar
- 4.Brown, L., Cai, T., DasGupta, A.: Interval estimation for a binomial propor-tion.Statistical Science 16(2), 101–133 (2001)Google Scholar
- 5.Cappelli, R., Maio, D., Maltoni, D.: Indexing Fingerprint databases for efficient 1:N matching. In: Proceedings of ICARCV2000 (2000)Google Scholar
- 6.Clopper, C., Pearson, S.: The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26, 404–413 (1934)Google Scholar
- 8.Fawcett, T.: ROC graphs: Notes and practical considerationsfor researchers (2004)Google Scholar
- 9.Grother, P.J., Phillips, P.J.: Models of large population recognition performance. In: CVPR04, pp. II: 68–75 (2004)Google Scholar
- 10.Hube, J.: Using biometric verification to estimate identification performance. In: Proceedings of 2006 Biometrics Symposium (2006)Google Scholar
- 11.ISO: Information technology –biometric performance testing and reporting – part 1: Principles and framework (ISO/IEC 19795-1:2006) (2006)Google Scholar
- 12.Johnson, A., Sun, J., Bobick, A.: Predicting large population data cumulative match characteristic performance from small population data. In: Proceedings of AVBPA 2003 (2003)Google Scholar
- 15.Mansfield, A.J., Wayman, J.L.: Best practices in testingand reporting performance of biometric devices, v2.01. Tech. Rep. NPL Report CMSC 14/02, National Physical Laboratory (2002)Google Scholar
- 16.Martin, A., Doddington, G., Kamm, T., Ordowski, M., Przybocki, M.: The DET curve in assessment of detection task performance. In: Proc. Eurospeech ’97, pp. 1895–1898. Rhodes, Greece (1997)Google Scholar
- 17.Payton, M.E., Greenstone, M.H., Schenker, N.: Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance? Journal of Insect Science 3(34) (2003)Google Scholar
- 18.Phillips, P.J., Grother, P., Micheals, R., Blackburn, D., Tabassi, E., Bone, J.: FRVT 2002 evaluation report. Tech. Rep. NISTIR 6965 (2003)Google Scholar
- 19.Phillips, P.J., Scruggs, W.T., O’Toole, A.J., Flynn, P.J., Bowyer, K.W., Schott, C.L., Sharpe, M.: FRVT 2006 and ice 2006 large-scale results. Tech. Rep. NISTIR 7408, National Institute of Standards and Technology (????)Google Scholar
- 20.Reynolds, D.A., Doddington, G.R., Przybocki, M.A., Martin, A.F.: The NIST speaker recognition evaluation -overview methodology, systems, results, perspective. Speech Commun. 31(2-3), 225–254 (2000)Google Scholar
- 21.Schuckers, M.: Estimation and sample size calculations for correlated binary error rates of biometric identification rates.Proceedings of the American Statistical Association: Biometrics Section (2003)Google Scholar
- 22.Schuckers, M.: Using the beta-binomial distribution to assess performance of a biometric identification device. International Journal of Image and Graphics 3(3) (2003)Google Scholar