Relations of Audio and Visual Speech Signals in a Physical Feature Space: Implications for the Hearing-impaired

Abstract

Goal of this paper is to introduce appropriate motion models for those visual articulatory movements that are relevant for the process of speechreading, and with this, design a facial animation program with an open input text vocabulary for use as a training aid for speechreading.

Since the experimental work of Menzerath and de Lacerda (1931) it is known that the movements of the speech organs are structurally interrelated within a spoken context. The sound signals and the related visual articulatory movements are created in the course of a fully overlapping coarticulation.

The paper will illustrate the interrelation of audio and visual speech features. It can be shown that even in fluent speech an interactive determination of moments of optimum articulation is possible for most of the phonemes. The facial expressions and the lip contours as well as the set points within the phoneme boundaries depend on the context. Several approaches for the description of these dependencies will be explained and discussed.

A facial animation system will be described which uses a codebook of specific key-pictures corresponding to the moments of optimum articulatory positions in fluent speech. Face movements are generated by selecting video pictures out of thecodebook and subsequently calculating interim pictures with the help of interpolation algorithms Movements of the tongue are artificially introduced into the opening of the mouth.

In order to evaluate different motion models it is necessary to present the corresponding facial animation model to people who can speechread. This is due to the point that too many parameters of the process of human visual perception are still widely unknown. For this reason, the presented evaluation methods are based on visemes as the smallest perceptible visual units of the articulation process. The interaction of these units is qualitatively described. Results of this experimental research extend the knowledge about articulation and coarticulation and are being used for the improvement of the facial animation system.