Commonly, in art classes, students are taught to analyze a work of art based upon the elements that comprise it. This includes line and emphasis in painting, or melody and harmony in an orchestral piece, or usage of space and timing in dance. A big feature of newer artistic media- television, film, video games, and so forth- is that they are inherently multimodal. Unlike a painting, to discuss a music video for a new hit song, one must address the visuals, the choreography, and the lyrics to really analyze the piece. This analysis requires multi-faceted perspectives that approach different technologies or art styles in an effort to understand what a given text or art form is trying to be. This task can be trying as new ways of creating continue to spring forth, but as electronic texts and electrical art forms progress, one will need to become more comfortable at breaking down all of a piece’s components.
It is here that Heidi McKee’s “Sound matters: Notes toward the analysis and design of sound in multimodal webtexts”, enters the stage. McKee’s text seeks to offer readers a means of interpreting aural aspects of webtexts with respect to how these aspects interact with other elements of the text in order to bring its form, in totality, forth. Mckee identifies four elements of sound: vocals, music, sound effects, and silence.
Mckee states when discussing the meaning behind the integration of sound to webtexts, “Yet when working with computerized writing technologies, despite the integration of modes in the final product—for example, animated images and music in a Flash movie—each of those elements still exists separately in the computer”. McKee predominantly referenced the separated nature of the components of a webtext in order to invoke Lev Manovich’s discussion of modularity in new media. She uses this to bring forth the point that, while writers and audiences may be able to see the unified structure of a webtext, all parties must be aware of the limitations of the technology that keep its multimodal elements separate.
Mckee’s idea is interesting because this element of separation, I would argue, is also a facet of a writer’s creative process and an audience’s reception. There is an on-going, memetic phenomena amongst segments of internet subculture that encourage the production of multimodal webtexts. The production of these webtexts is inherently reliant on the potential for the elements of the text to remain separate. As an example, here is the following:
Presumably created by Youtube user “Zarnith”, the above video is an excerpt of Youtube content creator Ryan “Northernlion” Letourneau’s commentator during a video of his gameplay of The Binding of Isaac: Afterbirth+ given a musical backing. The creator of “Northern Lion Ultra Hard Challenge” worked around the vocal, and to an extent visual, quality of the text they sourced from. Invoking the spirit of Jeff Rice, the text is essentially a remix of Letourneau’s soliloquy. The important thing to consider here is that, purely through the aural elements of the text, the creator repurposed Letourneau’s commentary to create something new.
Using Mckee’s terms, the video predominantly utilizes the elements of voice, music and sound effects.
The vocal element of the video is undoubtedly the most important. Letourneau’s enraged diatribe is given in a very steady and measured tone. The calmness the speaker possesses does much to add to the severity of the statements he makes as “the fire in his belly imbued by his ancestors hunting lions on the African planes” is lit. There is even tension, roughness, and pitch, and the speech is given in a relatively quiet voice, when compared to the music. The music is what brings the text into its own. The low bass and repetitive electronic beats punctuate the “lyrics”, so to speak. It is through the music that aural elements are given value to listeners who approach from the sensuous and expressive planes. Aside from a high-pitched ring that introduces the audience to the video, the only sound effects present in text are the barely audible sound effects from the game Letourneau was playing when the speech was originally given, a hint at the hidden context surrounding the text as a whole.
The important thing to consider here that Letourneau’s speech is a component of a much greater text in its own right, being part of his overall commentary given through episode 20 of his Let’s Play video. With the incorporation of music alone, the text is altered. The steady beats invoke the idea of a rap track, and such an intent is apparent as the author appends “Rap God” next to the picture of Letourneau’s face. Especially for any who are unfamiliar with Letourneau, the “Ultra Hard” challenge within the game, or the game itself, the is a tool by which the vocals gain meaning. His desire for Little Haunt to “destroy the enemies as you yourself have been destroyed” is given a metaphorical quality that makes the text useful to someone who lacks the full context behind the text’s creation. Though we can’t see the bitter enemies the speaker faces, we can feel the confidence, anger and hatred that the speaker feels.
Though a peculiar example, Zarnith’s video serves as an almost eeriely (especially considering how I only recently found the video while searching for good examples to discuss here) apt example for understanding how auditory elements can be picked out and analyzed individually, but also how they can serve alongside textual or visual elements. In an increasingly multimodal world, rhetorical elements will draw from all components of a text, and as such, audiences must follow in their analyses of these texts.