Abstract
Automatic evaluation metrics should correlate with human judgement. We collected sixteen ASR mediated dialogues using a map task scenario. The material was assessed extrinsically (i.e. in context) through measures like time to task completion and intrinsically (i.e. out of context) using the word error rate and several variants thereof, which are based on smaller units. Extrinsic and intrinsic results did not correlate, neither for word error rate nor for metrics based on characters, syllables or phonemes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Anderson, A., Brown, G., Shillcock, R., Yule, G.: Teaching Talk: Strategies for Production and Assessment. Cambridge University Press (1984)
Apache: ColognePhonetic (2013), http://commons.apache.org/codec/apidocs/org/apache/commons/codec/language/ColognePhonetic.html
Callison-Burch, C., Osborne, M., Koehn, P.: Re-evaluating the Role of BLEU in Machine Translation Research. In: Proceedings of EACL 2006 (2006)
Doddington, G.: Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics. In: Proceedings of HLT 2002 (2002)
Garofolo, J., Voorhees, E., Auzanne, C., Stanford, V., Lund, B.: 1998 TREC-7 Spoken Document Retrieval Track Overview and Results. In: Proceedings of TREC, vol. 7 (1998)
HyFo: HyFo library (2013), http://defoe.sourceforge.net/hyfo/hyfo.html (accessed January 2013)
Kawahara, T.: New perspectives on spoken language understanding: Does machine need to fully understand speech? In: ASRU 2009. IEEE Workshop on Automatic Speech Recognition & Understanding (2009)
McCowan, I., Moore, D., Dines, J., Gatica-Perez, D., Flynn, M., Wellner, P., Bourlard, H.: On the Use of Information Retrieval Measures for Speech Recognition Evaluation- Research Report 04-73. Tech. rep., IDIAP Research Institute (2005)
Möller, S.: Parameters for quantifying the interaction with spoken dialogue telephone services. In: Proc. of SIGDial 2005 (2005)
Möller, S., Ward, N.: A framework for model-based evaluation of spoken dialog systems. In: Proc. of Workshop on Discourse and Dialogue, SIGDial 2008 (2008)
Morris, A., Maier, V., Green, P.: From WER and RIL to MER and WIL: Improved Evaluation Measures for Connected Speech Recognition. In: Proceedings of International Conference on Spoken Language Processing (2004)
Nuance: German academic version of Dragon naturally speaking version 11 (2012)
Postel, H.: Die Kölner Phonetik. Ein Verfahren zur Identifizierung von Personennamen auf der Grundlage der Gestaltanalyse. IBM-Nachrichten 19, 925–931 (1969)
Rodgers, J., Nicewander, W.: Thirteen ways to look at the correlation coefficient. The American Statistician 42(1), 59–66 (1988)
Schneider, A., Luz, S.: Speaker Alignment in Synthesised, Machine Translated Communication. In: Proceedings of IWSLT 2011 (2011)
Skantze, G.: Exploring Human Error Recovery Strategies: Implications for Spoken Dialogue Systems. Speech Communication 45(3), 325–341 (2005)
Stanier, A.: How Accurate is SOUNDEX Matching? Computers in Genealogy 3(7) (1990)
Walker, M., Kamm, C., Litman, D.: Towards developing general models of usability with PARADISE. Natural Language Engineering 6(3-4), 363–377 (2000)
Wang, Y.Y., Acero, A., Chelba, C.: Is word error rate a good indicator for spoken language understanding accuracy. In: 2003 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2003, pp. 577–582 (November 2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Schneider, A.H., Hellrich, J., Luz, S. (2014). Word, Syllable and Phoneme Based Metrics Do Not Correlate with Human Performance in ASR-Mediated Tasks. In: Przepiórkowski, A., Ogrodniczuk, M. (eds) Advances in Natural Language Processing. NLP 2014. Lecture Notes in Computer Science(), vol 8686. Springer, Cham. https://doi.org/10.1007/978-3-319-10888-9_39
Download citation
DOI: https://doi.org/10.1007/978-3-319-10888-9_39
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-10887-2
Online ISBN: 978-3-319-10888-9
eBook Packages: Computer ScienceComputer Science (R0)