Human Speechreading: Psychology and Cognition

Abstract

Prior to the session, some of the behavioral scientists had asked if the machine speech recognition researchers could elaborate on the kinds of problems that they were trying to solve. It seemed that if we could achieve a clearer understanding of the problem space of the machine researchers then the behavioral researchers would be better positioned to contribute relevant knowledge. To this end, David Stork began saying that what the machine speech recognition researchers really needed to know more about concerned the input or signal information that was perceived and attended to by humans during speechreading. At the present point in time, he said that they were less interested in how humans processed or integrated multiple sources of information. For example, he stated that to simplify the problem for the moment, we should not be concerned with the contribution of the acoustic signal or higher level knowledge. Oscar Garcia and Micheal Brooke noted that there was, nevertheless, an important relationship between the acoustic speech signal and mouth shape that influenced bimodal perception. David then opened the discussion by posing the question “what signal features should machines extract from faces?” Michael Brooke pointed out that it seemed difficult to use a simple inventory of features because any priorization of the features would require knowledge regarding how the features were integrated. Dominic Massaro suggested that it would be useful to use oral interpreters for the deaf as informants who might have insights into what facial features or gestures made speechreading easier.