An Evaluation of Visual Speech Features for the Tasks of Speech and Speaker Recognition

  • Simon Lucey
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2688)


In this paper an evaluation of visual speech features is performed specifically for the tasks of speech and speaker recognition. Unlike acoustic speech processing, we demonstrate that the features employed for effective speech and speaker recognition are quite different to one another in the visual modality. Area based features (i.e. raw pixels) rather than contour features (i.e. an atomized parametric representation of the mouth, e.g. outer and inner labial contour, tongue, teeth, etc.) are investigated due to their robustness and stability. For the task of speech reading we demonstrate empirically that a large proportion of word unit class distinction stems from the temporal rather than static nature of the visual speech signal. Conversely, for the task of speaker recognition static representations suffice for effective performance although modelling the temporal nature of the signal does improve performance. Additionally, we hypothesize that traditional hidden Markov model (HMM) classifiers may, due to their assumption of intra-state observation independence and stationarity, not be the best paradigm to use for modelling visual speech for the purposes of speech recognition. Results and discussion are presented on the M2VTS database for the tasks of isolated digit, speech and text-dependent speaker recognition.


Hide Markov Model Speech Recognition Visual Feature Speaker Recognition Visual Speech 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    C. Bregler and Y. Konig. Eigenlips for robust speech recognition. In International Conference on Accoustics, Speech and Signal Processing, Adelaide, Australia, 1994.Google Scholar
  2. [2]
    S. Cox, I. Matthews, and J.A. Bangham. Combining noise compensation with visual Information in speech recognition. In Auditory-Visual Speech Processing, Rhodes, 1997.Google Scholar
  3. [3]
    L. Deng. A generalized hidden Markov model with state-conditioned trend functions of time for the speech signal. Signal Processing, 27:65–78, 1992.zbMATHCrossRefGoogle Scholar
  4. [4]
    M. S. Gray, J.R. Movellan, and T. J. Sejnowski. A comparison of local versus global image decompositions for visual speechreading. In 4th Joint Symposium on Neural Computation, pages 92–98, 1997.Google Scholar
  5. [5]
    F. Lavagetto. Converting speech into lip movements: A multimedia telephone for hard hearing people. IEEE Transactions on Rehabilitation Engineering, 3(1):90–102, March 1995.CrossRefGoogle Scholar
  6. [6]
    I. Matthews, T. Cootes, S. Cox, R. Harvey, and J.A. Bangham. Lipreading using shape, shading and scale. In Auditory-Visual Speech Processing, pages 73–78, Sydney, Australia, 1998.Google Scholar
  7. [7]
    S. Pigeon. The M2VTS database. Laboratoire de Telecommunications et Teledection, Place du Levant, 2-B-1348 Louvain-La-Neuve, Belgium, 1996.Google Scholar
  8. [8]
    G. Potamianos and H.P. Graf. Linear discriminant analysis for speechreading. In IEEE Second Workshop on Multimedia Signal Processing, pages 221–226, 1998.Google Scholar
  9. [9]
    G. Potamianos, H.P. Graf, and E. Cosatto. An image transform approach for HMM based automatic lipreading. In International Conference on Image Processing, volume 3, pages 173–177, 1998.Google Scholar
  10. [10]
    G. Potamianos, J. Luettin, and C. Neti. Hierarchical discriminant features for audio-visual LVCSR. In International Conference on Acoustics, Speech and Signal Processing, volume 1, pages 165–168, 2001.Google Scholar
  11. [11]
    S. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland. The HTK Book (for HTK version 2.2). Entropic Ltd., 1999.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Simon Lucey
    • 1
  1. 1.Advanced Multimedia Processing Laboratory Department of Electrical and Computer EngineeringCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations