Introduction
Hello, and welcome to our next lesson.
In our previous session, we explored the concept of embodied cognition, establishing that our perception of sound is not a detached, abstract process but one deeply rooted in our physical, bodily experience. We saw how filmmakers use sound to activate embodied image schemas and conceptual metaphors, allowing us to feel narrative tension and character emotions on a pre-reflective, physical level.
This lesson builds directly on that foundation. We've established the cognitive and embodied aspects of perception; now, we will explicitly integrate the emotional dimension. Our learning outcome is to discuss how cognitive and affective processes interact in the perception of film soundscapes.
We will move beyond seeing 'thinking' and 'feeling' as separate activities. Instead, we'll examine them as a dynamic, interactive loop where our cognitive appraisal of a sound shapes our emotional response, and our affective state, in turn, influences what we notice and how we interpret the soundscape. We will ground this discussion in established psychological theories and analyze how cinematic techniques exploit this very interaction.
1. Appraisal Theory: The Cognitive Root of Emotion
A foundational concept for understanding the link between cognition and affect is appraisal theory. In short, this theory proposes that emotions are not direct, raw reactions to stimuli. Instead, they arise from our cognitive evaluation—or appraisal—of a stimulus or situation in relation to our own well-being, goals, and expectations. This appraisal can be instantaneous and unconscious, or more slow and deliberate.
A comprehensive academic article on soundscape perception provides an excellent overview of the relevant emotion theories.
Assessments of Acoustic Environments by Emotions - PMC - NIH
The paper 'Assessments of Acoustic Environments by Emotions' offers a strong theoretical basis for our discussion. It outlines the core tenets of emotion theory and its application to soundscape research, which we can adapt for our analysis of film.
Please read the Introduction and the sub-section Emotion Theory and Research (found under the Introduction). Focus on these key points: The definition of emotion as a relational, purposeful, and short-lived phenomenon. The concept of appraisal: the idea that events elicit emotions through their significance and meaning to the individual. The distinction between conscious emotions and the often unconscious appraisals that lead to them.
The key takeaway is that an emotional response to a sound in a film is not triggered by the sound wave itself, but by the meaning we assign to it. The process looks like this:
Sound Stimulus → Cognitive Appraisal ("What is it? Is it safe? What does it mean?") → Affective Response (Emotion)
The article you just read proposes a conceptual framework (in its Figure 3) that visualizes this as a feedback loop. An initial appraisal leads to an emotional response and behavior. Over time, these repeated experiences can solidify into a more stable mood or attitude, which then influences future appraisals. In a film context, a director can "train" an audience to associate a specific sound with danger, so that each subsequent time the sound is heard, the appraisal is faster and the affective response of fear is stronger.
2. The Cognitive-Affective Power of Sound Objects
How, then, do film sounds acquire the meaning that we appraise? Building on our last lesson, this happens through metaphor, symbolism, and context. The following reading offers a detailed breakdown of how auditory objects are loaded with semantic and emotional potential.
[PDF] The Emotional Impact of Sound: A Short Theory of Film Sound Design
Dr. Thomas Görne's paper, 'The Emotional Impact of Sound,' provides a vocabulary for dissecting how specific types of sounds trigger cognitive-affective responses. It connects the dots between perception, meaning-making, and emotion.
Please read the following three short sections: 1.1 Crossmodal Metaphors, Linguistic Metaphors: This section expands on our previous lesson, showing how metaphors (e.g., a low-pitched sound is 'large,' 'dark,' 'deep') create an initial layer of meaning and emotional connotation. 1.2 Sound Symbols: Focus on how certain sounds (wind, thunder, a bell) come pre-loaded with archetypal, symbolic meanings that carry inherent emotional weight. 1.4 The Ambiguous Object: This is a crucial concept. Pay attention to how a sound that defies cognitive categorization can provoke a strong affective response like fear or unease.
This paper provides us with a clear model of the cognitive-affective interaction:
- Cognitive Process: The listener's brain performs causal listening, attempting to identify the sound source and categorize it. It uses tools like crossmodal metaphors (
high pitch = sharp) and learned symbols (howling wind = lonely/dangerous). - Affective Result: The outcome of this cognitive process directly informs the emotional response. A sound cognitively appraised as 'sharp' produces an affective sense of aggression or pain. A sound appraised as 'ambiguous' or unidentifiable creates an affective state of anxiety because it represents a potential, unknown threat. The cognitive failure to safely categorize the sound is the source of the emotional unease.
3. Case Study 1: The Auditory Kuleshov Effect and Cognitive Dissonance
One of the most powerful demonstrations of the interplay between cognition and affect is when sound is used to re-frame our understanding of an image.
First, let's watch a video that explains the famous Kuleshov effect and proposes its auditory equivalent.
How movie music manipulates your emotions
The video 'How movie music manipulates your emotions' provides a clear, practical demonstration of how auditory information can fundamentally alter our emotional interpretation of ambiguous visual information.
Please watch the segment from 09:27 to 14:53. This section introduces the Kuleshov effect and then extends the concept to sound, showing how different musical scores can make the same visual scene feel pensive, romantic, or sinister.
The video demonstrates what Dr. Görne's paper calls polarization: an emotionally neutral image is loaded with meaning by an affective sound. This is a cognitive act—the brain synthesizes the two streams of information into a single meaning—that results in a powerful affective shift.
Now, let's look at a more complex version of this phenomenon: incongruence. This occurs when the cognitive appraisal of the sound contradicts the cognitive appraisal of the image.
[PDF] The Emotional Impact of Sound: A Short Theory of Film Sound Design
Let's return to Görne's paper to analyze this more advanced technique.
Please read section 3.2 Image / Sound Relations. Focus specifically on the discussion of Incongruence and how it causes cognitive dissonance in the audience.
When a filmmaker pairs a pleasant image (e.g., children playing in a sunny park) with a menacing, low-frequency drone, they create cognitive dissonance. The brain's struggle to reconcile these contradictory signals generates a powerful affective response: tension, dread, and a sense of foreboding. The emotion arises directly from the cognitive clash.
4. Case Study 2: Subjective Point of Audition
Another potent technique for merging cognitive and affective experience is manipulating the point of audition. By aligning the soundscape with a specific character's perception, the filmmaker invites us not just to see what they see, but to feel what they feel.
Sound and Silence in Gravity: Fidelity vs Intelligibility
The video analysis of the film Gravity offers a perfect example of how a cognitive framework—understanding whose perspective we are hearing from—drives an emotional experience.
Please watch the segment from 07:19 to 09:44. This part introduces Michel Chion's concept of 'point of audition' and analyzes how Gravity uses its subjective sense to anchor us in the character's physical and emotional reality.
Here's the cognitive-affective breakdown:
- Cognitive Frame: The film teaches us that what we hear is filtered through the character's body and experience. We cognitively accept the rule: "I am hearing what Dr. Stone is hearing."
- Affective Experience: Because of this cognitive frame, the sound design becomes a direct conduit for emotion. The muffled vibrations of impacts felt through her suit, the high-pitched ringing of tinnitus after an explosion, the return of sound as she regains consciousness—these are not just informational cues. They generate a visceral, affective experience of claustrophobia, panic, and physical shock in the viewer. Our cognitive understanding of the sonic perspective is the key that unlocks this direct affective transmission.
5. Affect from Unconscious Perception
Finally, the interaction between cognition and affect is not always a conscious process. Sound can influence our emotional state even when it's not the focus of our attention—or even when we don't consciously notice it at all.
[PDF] The Emotional Impact of Sound: A Short Theory of Film Sound Design
For this final point, let's revisit Görne's paper one last time to look at the fascinating phenomenon of 'inattentional deafness.'
Please read section 2.3 Gorillas at the Cocktail Party. Focus on the concepts of inattentional deafness and the 'inaudible gorilla'—a meaningful sound outside the focus of attention that can still emotionally intensify a scene.
This concept should be very familiar from your professional experience as a mixer. It refers to the subtle layers of sound—a faint, ominous drone beneath dialogue; the almost imperceptible sound of an animal growl mixed into a machine's hum—that an audience may not consciously register.
In this case, the cognitive process of selective attention filters these sounds out of conscious awareness. However, research suggests our affective systems can still process them, leading to a heightened sense of tension or unease without the viewer knowing exactly why. This is a powerful example of the cognitive and affective systems working in parallel, with the former managing conscious focus while the latter responds to the full richness of the auditory scene.
Conclusion
Today, we have bridged the gap between thinking and feeling in the perception of film soundscapes. We've established that these are not two separate tracks but a deeply integrated and interactive system.
Key Takeaways:
- The perception of sound is a dynamic loop of cognition and affect. We feel things because of how we cognitively appraise what we hear, and our feelings, in turn, color our perception.
- Appraisal theory provides a core model: emotions are generated by our evaluation of a sound's meaning and significance to us.
- Filmmakers manipulate this interaction through specific techniques. Incongruence creates cognitive dissonance that we experience as emotional tension. Subjective point of audition uses a cognitive frame to enable direct affective empathy.
- This interaction occurs on both conscious and unconscious levels. We can be emotionally influenced by sounds that our conscious, attentive mind has filtered out (the 'inaudible gorilla').
Preview of the Next Lesson:
Having discussed the general interaction between cognition and affect, our next lesson will zoom in on a specific cognitive faculty: attention. The learning outcome will be to analyze how filmmakers exploit perceptual tendencies to direct audience attention through sound. We will explore how sound design serves as a primary tool for guiding the viewer's focus, managing information, and building narrative structure, building on concepts like 'perceptual realism' and the 'orienting reflex' that were introduced in today's readings.