Automatic Speechreading using dynamic contours

Abstract

Recent developments in contour tracking permit the outlines of moving, natural objects to be tracked live, at full video-rate. Such a capability can be used to turn parts of the body—for instance, the hands and lips—into input devices. The results presented here were obtained using a real-time lip tracker which utilises a novel Kalman filter based dynamic contour to track the outline of a speaker’s lips. The tracker incorporates predictive dynamics which can be learned from training sequences and automatically tuned to follow typical motions found in speech. The visual data from the tracker is incorporated into an acoustic automatic speech recogniser enabling robust recognition of speech in the presence of acoustic noise. Tests on a 40 word vocabulary using a dynamic time warping based audio-visual recogniser demonstrate that the lip outline is a rich source of information for speech recognition and establish dynamic contour tracking as a viable instrument for near real-time speechreading applications.