Abstract
In the evaluation of speaker recognition systems—an important part of speaker classification [1], the trade-off between missed speakers and false alarms has always been an important diagnostic tool. NIST has defined the task of speaker detection with the associated Detection Cost Function (DCF) to evaluate performance, and introduced the DET-plot [2] as a diagnostic tool. Since the first evaluation in 1996, these evaluation tools have been embraced by the research community. Although it is an excellent measure, the DCF has the limitation that it has parameters that imply a particular application of the speaker detection technology.
In this chapter we introduce an evaluation measure that instead averages detection performance over application types. This metric, , was first introduced in 2004 by one of the authors [3]. Here we introduce the subject with a minimum of mathematical detail, concentrating on the various interpretations of and its practical application.
We will emphasize the difference between discrimination abilities of a speaker detector (‘the position/shape of the DET-curve’), and the calibration of the detector (‘how well was the threshold set’). If speaker detectors can be built to output well-calibrated log-likelihood-ratio scores, such detectors can be said to have an application-independent calibration. The proposed metric can properly evaluate the discrimination abilities of the log-likelihood-ratio scores, as well as the quality of the calibration.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Martin, A.: Evaluations of Automatic Speaker Classification Systems. In: Müller, C. (ed.) Speaker Classification I. LNCS(LNAI), vol. 4343, Springer, Heidelberg (2007)
Martin, A., Doddington, G., Kamm, T., Ordowski, M., Przybocki, M.: The DET curve in assessment of detection task performance. In: Proc. Eurospeech 1997, Rhodes, Greece, pp. 1895–1898 (1997)
Brümmer, N.: Application-independent evaluation of speaker detection. In: Proc. Odyssey, Speaker and Language recognition workshop, ISCA 2004, pp. 33–40 (2004)
Brümmer, N., du Preez, J.: Application-independent evaluation of speaker detection. Computer Speech and Language 20, 230–275 (2006)
NIST: The NIST year 2006 Speaker Recognition Evaluation Plan (2006), http://www.nist.gov/speech/tests/spk/2006/index.htm
Campbell, W.M., Reynolds, D.A., Campbell, L.P., Brady, K.J.: Estimating and evaluating confidence for forensic speaker recognition. In: Proc. ICASSP, pp. 717–720 (2005)
Campbell, W.M., Brady, K.J., Campbell, J.P., Granvile, R., Reynolds, D.A.: Understanding scores in forensic speaker recognition. In: Proc. Odyssey 2006 Speaker and Language Recognition Workshop (2006)
Ramos-Castro, D., González-Rodríguez, J., Ortega-Garcia, J.: Likelihood ratio calibration in a transparent and testable forensic speaker recognition framework. In: Proc. Odyssey 2006 Speaker and Language Recognition Workshop (2006)
Brümmer, N., van Leeuwen, D.A.: On calibration of language recognition scores. In: Proc. Odyssey 2006 Speaker and Language recognition workshop (2006)
Auckenthaler, R., Carey, M., Lloyd-Thomas, H.: Score normalization for text-independetn speaker verification systems. Digital Signal Processing 10, 42–54 (2000)
Navrátil, J., Ramsawamy, G.N.: The awe and mistery of t-norm. In: Proc. Eurospeech, pp. 2009–2012 (2003)
Van Leeuwen, D.A., Martin, A.F., Przybocki, M.A., Bouten, J.S.: NIST and TNO-NFI evaluations of automatic speaker recognition. Computer Speech and Language 20, 128–158 (2006)
Swets, J.A.: Signal detection and recognition by human observers; contemporary readings. Wiley, New York (1964)
Green, D.M., Swets, J.A.: Signal Detection Theory and Psychophysics. Wiley, New York (1966)
Bernardo, J.M., Smith, A.F.M.: Bayesian Theory. Wiley, New York (1994)
DeGroot, M., Fienberg, S.: The comparison and evaluation of forecasters. The Statistician, 12–22 (1983)
Doddington, G.R., Przybocki, M.A., Martin, A.F., Reynolds, D.A.: The NIST speaker recognition evaluation—Overview, methodology, systems, results, perspective. Speech Communication 31, 225–254 (2000)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
van Leeuwen, D.A., Brümmer, N. (2007). An Introduction to Application-Independent Evaluation of Speaker Recognition Systems. In: Müller, C. (eds) Speaker Classification I. Lecture Notes in Computer Science(), vol 4343. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74200-5_19
Download citation
DOI: https://doi.org/10.1007/978-3-540-74200-5_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74186-2
Online ISBN: 978-3-540-74200-5
eBook Packages: Computer ScienceComputer Science (R0)