Advertisement

Performance Testing and Reporting

The evaluation of system performance and assessment of overall system reliability are important to the implementation and use of all biometric systems. Almost all biometric systems are ‘black boxes’ to their users, and it is only through appropriate testing that a system’s accuracy and performance can be determined. However, this testing process is not always straightforward, as there are many factors that influence performance, and published error rates often provide only approximate guidance. Issues related to the misunderstanding of test results are frequently the reason why some biometric implementations have failed to meet expectations. Properly conducted testing can provide crucial information for the planning of a system’s functionality, and the introduction of controls to maximize results.

The goals of this chapter are to:
  • Discuss the evaluation process (Sect. 5.1).

  • Describe the components of a test plan (Sect. 5.2).

  • Introduce the different types of evaluation (Sect. 5.2.1).

  • Present the fundamental techniques for assessment, including establishing ground truth and data size (Sect. 5.3.1).

  • Discuss issues related to the test set (Sect. 5.3).

  • Present the elements of proper reporting of test results (Sect. 5.4).

Keywords

Ground Truth Performance Test Testing Process Test Plan Biometric System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Performix biometric research and analysis software. http://www.biometix.com/performix.htm(2007)
  2. 2.
    Cappelli, R., Erol, A., Maio, D., Maltoni, D.: Synthetic Fingerprint-image generation. Pattern Recognition, 2000. Proceedings. 15th International Conference on 3 (2000)Google Scholar
  3. 3.
    ISO: Information technology –biometric performance testing and reporting – part 1: Principles and framework (ISO/IEC 19795-1:2006) (2006)Google Scholar
  4. 4.
    Jovanovic, B., Levy, P.: A look at the rule of three.The American Statistician 51(2), 137–139 (1997)CrossRefGoogle Scholar
  5. 5.
    Maio, D., Maltoni, D., Cappelli, R., Wayman, J.L., Jain, A.K.: FVC2000: Fingerprint verification competition. IEEE Trans. Pattern Anal. Mach. Intell. 24(3), 402–412 (2002)CrossRefGoogle Scholar
  6. 6.
    Mansfield, A., Wayman, J.: Best practices in testing and reporting performance of biometric devices. NPL Report CMSC 14(02) (2002)Google Scholar
  7. 7.
    Mansfield, T., Kelly, G., Chandler, D., Kane, J.: Biometric product testing final report. Contract 92, 4009,309 (2001)Google Scholar
  8. 8.
    O’Toole, A.J., Phillips, P.J., Jiang, F., Ayyad, J., Penard, N., Abdi, H.: Face recognition algorithms surpass humans matching faces over changes in illumination. In: IEEE Transactions on Pattern and Machine Intelligence, vol. 29, pp. 1642–1646 (2007)CrossRefGoogle Scholar
  9. 9.
    Phillips, P.J., Scruggs, W.T., O’Toole, A.J., Flynn, P.J., Bowyer, K.W., Schott, C.L., Sharpe, M.: FRVT 2006 and ice 2006 large-scale results. Tech. Rep. NISTIR 7408, National Institute of Standards and Technology (????)Google Scholar
  10. 10.
    Reynolds, D.A., Doddington, G.R., Przybocki, M.A., Martin, A.F.: The NIST speaker recognition evaluation -overview methodology, systems, results, perspective. Speech Commun. 31(2-3), 225–254 (2000)Google Scholar
  11. 11.
    Schuckers, M.: Estimation and sample size calculations for correlated binary error rates of biometric identification rates.Proceedings of the American Statistical Association: Biometrics Section (2003)Google Scholar

Copyright information

© Springer-Verlag US 2009

Personalised recommendations