AUDIOVISUAL COMMUNICATION SYSTEMS

Audiovisual Communication Systems

Audiovisual Communication Systems

Section A:

Question 1: a

In our interaction with others, we make use of all of our sensory modalities as we communicate and exchange information. Our senses are exceptionally well-adapted for interaction, and our neurophysiology enables us to effortlessly integrate information from different modalities fusing data to optimally meet the current previous termcommunicationnext term needs. A full account of the speech previous termcommunicationnext term process must therefore include multiple modalities. While the auditory modality often provides the phonetic information necessary to convey a linguistic message, the visual modality can qualify the auditory information providing segmental cues for place of articulation, prosodic information concerning prominence and phrasing and extralinguistic information such as signals for turn-taking, emotions and attitudes. Although these observations are not novel, prosody research has traditionally concentrated on describing the acoustics of prominence and phrasing in restricted speaking styles. While different speaking styles including expressive and emotional speech have received more attention during recent years, we still lack basic knowledge concerning how auditory and visual signals interact to signal communicative functions in expressive speech. One reason for this is the primary status of auditory speech. Another reason is the relatively more complicated analysis and synthesis of visual speech. Most of the work that has been done in multimodal speech perception has concentrated on segmental cues in the visual modality.

Question 1: c

The visible articulatory movements are mainly those of the lips, jaw and tongue. However, these are not the only visual information carriers in the face during speech. Much information related to e.g. phrasing, stress, intonation and emotion are expressed by for example nodding of the head, raising and shaping of the eyebrows, eye movements and blinks. These kinds of facial actions should also be taken into account in a visual speech synthesis previous termsystem,next term not only because they may transmit important non-verbal information, but also because they make the face look alive. These movements are more difficult to model in a general way than the articulatory movements, since they are optional and highly dependent on the speaker's personality, mood, purpose of the utterance, etc.

As we attempt to take advantage of the effective previous termcommunicationnext term potential of human conversation in spoken dialogue previous termsystems,next term we see an increasing need to embody the conversational partner using previous termaudiovisualnext term verbal and non-verbal previous termcommunicationnext term implying the use and integration of both audio and visual modalities (Massaro, 1998 D.W. Massaro, Perceiving Talking Faces: From Speech Perception to a Behavioural Principle, The MIT Press, Cambridge, MA (1998).Massaro, 1998). Effective interaction in dialogue previous termsystemsnext term involves both the presentation of information and the flow of interactive dialogue. A talking animated agent can provide the user with an interactive partner whose goal is to take the role of the human agent. An effective agent is one who is capable of supplying the user with relevant information, can fluently answer questions concerning complex information and can ultimately assist the user in a decision making process ...