Materialisation, Emotion, & Attention: Tracking Sound’s Perceptual Effects in Film

Curator's Note

Musicians, film-goers, and cinema theorists alike understand the emotional power of sound in film. Whilst the effects of the soundtrack may often be elicited subliminally, the strength of audio to add value to image and narrative is well accepted, if not fully understood. Part I: “Sound, Ears & Emotion” begins by outlining the functional mechanics of score and sound design. It draws on inherent elements of evolutionary biology and principles of systematic musicology to better understand how sound can make the audience feel. Importantly, these academic principles support an active practitioner’s perspective, and the essay explores through its form (music choices, vocal delivery, sound design, and visual choices) the very theories it is presenting.

Having established the means by which sound charges and changes the emotional experience of image and narrative, the essay then takes advantage of the opportunities afforded by eye tracking technology to consider more deeply the effects of sound beyond affect. Part II: “Sound, Eyes & Attention” draws on results from an ETMI Research Group study. It examines data collected from six participants watching Monsters Inc. under two conditions – sound on and sound off. By so doing, it considers how audio shifts the traditional balance between volitional (top down) and reactive (bottom up) attention. Through the exploration of the data collected, it demonstrates how contemporary sound film under silent conditions draws the eye to low-level, salient stimuli when a scene is assessed. These findings speak to sound’s capacity to immerse the audience in the world of the film, rather than being distracted in its absence by movement, light, and colour. Through audiovision, decoding the image is no longer a series of knee-jerk reactions. Viewers “view” differently.  Rather than the established understanding, where sound design is considered an instrument for generating the perception of image’s authenticity or music is used to synchronize group emotions, this research demonstrates sound’s capacity to actively shift our focus and change the methods by which we engage with what’s on screen.

For the first section, the video essay format provides an ideal opportunity to present the cinematic audiovisual relationships under discussion. Beyond mere outlines, these principles can be actively demonstrated. Examples are played with and without sound, radical recontextualisations and juxtapositions are constructed, and scenes from very different films are sutured together with audio as ideas are being explored. In the second section, examples of the footage from the eye tracking tests exploit a different advantage of the format, and data is presented as a temporal experience. In so doing, it allows for more dynamic and, at times, playful graphic analysis as a means to present what could otherwise be a series of dry stills.  What interests me most is the extent to which this format allows for a shift in tone from a standard journal article. Sadly, in print, there is limited capacity to augment academic writing with violence and stupidity. The possibility for academic rigor to coexist with devices from popular entertainment, to inform and engage beyond the limits of an academic cohort, is possibly the most exiting potential of the form.

Response to Reviewers' Comments:

This essay was my first attempt to explore the affordances and limitations of a new medium for disseminating ideas. I think both authors have identified the compromises at the core of trying to balance tone - a sense of playful provocation delivered at an inherently fast pace - with the traditional clarifications, cautions and caveats expected of academic writing. Even though I would argue that some of the listed criticisms have misunderstood what I was saying, the fact that such misunderstanding occurred speaks to the options for careful explanation inherent in the printed journal format when compared to a video essay explicitly aiming to bring these ideas to a broader audience at speed. This is in part to do with space and traditions of the written word – but interestingly, I think it also identifies the ease in print of rereading until understood, when compared to the one-pass approach we expect from temporal audiovisual media.

With more time – and if provocation and pace were to be sacrificed - there are a number of issues I could certainly have made clearer:

For example, the essay isn’t suggesting that profound monosensory experiences aren’t possible (indeed, that they are is at the core of my more recent 4D research into the flavours of aesthetic experience). Rather, this essay’s focus was what happens to sound film when its audio is removed. Similarly, it’s not suggesting that misdirection isn’t direction – in fact, I agree that it’s quite the opposite. “Reality”? That cornstarch sounds more like snow than recordings of snow is an example of how visceral exaggeration can make the audience intuitively feel that something is “right” even when the sound in a film is as patently false as the colour. For misattribution, Brown and I are arguing about agreeing – as the point that I was making was that the salient stimulus, to which the feelings generated by sound are ‘misattributed’, is the vision. It’s simply an extension, via cognitive musicology, of Chion’s uncontroversial ‘added value’ principle.

For Smith, whilst I certainly agree that there is usually an emotional audiovisual alignment in film, hence “misattribution” as a term is contestable, I would argue that, in other instances, our emotional state, perceived as a “legitimate response to the movie” is still being “misattributed” to the narrative experience when it can often be being generated by the music. The misattribution then refers to where we might mistakenly perceive the source of the emotion. This is completely independent of empathetic or unempathetic alignments. Whether it’s the Strauss waltz or the Jaws theme as we look up at the swimmer’s legs (see Higson’s Jaws Opening Scene on Youtube), our emotion is still misattributed onto the situation we’re watching – eg. this is a beautiful image/scene vs this is creepy/anxiety-inducing scene – all from the black ops of the sound.

These clarifications simply highlight the importance of time and space to ensure a point is understood – and also the care with which arguments should be made and qualified in academic discourse.  In a way, my first attempt at a video essay was an opportunity to play with the violence and stupidity to explore how it might sit with academic rigour. Ultimately, this was done in order to find a broader audience to contemplate these ideas. This piece should be viewed then as a playful starting lob for more rigorous sound discussions to come. That it might be better suited to youtube than an academic journal however is certainly something I am open to.

P.S. Hilariously, I actually thought that my selection of Citizen Kane for a “good” film and Batman vs Superman for a “bad” would be utterly uncontroversial!! I have much to learn about cinema theorists….