Research on Audio-Visual Asynchronous Correlation for Speaker Identification Based on DBN

Abstract

In noisy or other adverse conditions, consistently high speaker identification accuracy is difficult to attain via speech signal, hence visual component which can complement audio information is of particular interest. In this paper, we capture the asynchronous correlation instead of tight synchrony between audio and visual modalities. Furthermore, the apparent asynchrony between the two modalities is effectively modeled based on Dynamic Bayesian Network (DBN) with asynchronous articulatory feature in three ways: (1) there are three hidden state variables, each representing one articulatory feature, (2) the degree of asynchrony among articulatory features is controlled by probability distribution, (3) the audio and video observations depend on all three hidden state variables. Then a multi-level hybrid fusion is explored to combine model-level and decision-level fusion. The experiment results for audio-visual bimodal corpus show that the effectiveness of the method.