ISCA Archive Interspeech 2004
ISCA Archive Interspeech 2004

Of the top of the head: audio-visual speech perception from the nose up

Chris Davis, Jeesun Kim

When someone speaks, linguistically relevant movements are produced. Previous work examining visual speech investigated regions around the oral aperture. Here we examined whether people could use visual information from the top part of the talker's face. In Experiment 1 observers saw two pairs of short silent video clips that showed the top part of a talker's head. The task was to judge the pair of videos in which the talker said the same sentence (different tokens were used). Observers were able to make this judgment at better than chance levels. In Experiment 2, participants were presented with auditory-visual pairs and had to select the pair in which the talker said the same sentence. Matching performance was again above chance. The results indicate that participants are sensitive to the speech relatedness of movements from the upper part of the head and face. Implications for speech processing and for the development of virtual characters are considered.