Introduction

SOUND, FILM THEORY AND THE CINEMATIC EXPERIENCE

The human skin of things, the epidermis of reality: this is the primary raw material of cinema. Cinema exalts matter and reveals it to us in its profound spirituality.

—Antonin Artaud¹

The increasingly important role played by sound in contemporary audiovisual mediascape has been widely acknowledged. Consequently sound has become one of the most fruitful fields of cinema scholarship since the 1990s. While the many significant contributions produced in the last two decades can no longer justify the clichéd lamentation of being neglected, the field of film sound is still sometimes perceived as "always emerging, never emerged."² This is partly because, in my opinion, while the practices of cinema, television, video games and many other audiovisual media have all made intriguing claims for sound, sound’s role in a theory of the audiovisual experience remains ambiguous. To pose the question in a more specific context, what exactly is the nature of sound in a hybrid art that is known as cinema? Is sound an aid (parallel) or a counterpoint to the visual? Or is it a secrete passage to emotion? Ultimately, to what does sound make its contributions?

Although sound has become a crucial aspect of the cinematic experience as such, its role in film theory remains to be reassessed. A burgeoning ground for the last few decades, the field of film sound now calls for a theory of cinematic experience where sound plays a constituent part. In reaction to the contemporary film theory’s lack of grip on sound, this dissertation proposes a worlding theory of the cinematic experience where the sound’s role is nothing less central. What I offer in the pages that follow is foremost a theory of film sound substantiated in the history of sound practices. Yet this sound theory also implies a radical reformation of film theory that is grounded on the audiovisual experience of cinema. The notion of the world foregrounds the audiovisual gestalt by which we engage with a phenomenological rendering of the world that the film gives us. In conceiving a sound theory around the notion of the world, I aim to raise the stake for sound by finding a way to articulate not only what connects the classical film theory and the contemporary film theory, but also what could make the two working together, for a worlding theory of cinematic experience is concerned with the essence of cinema (as preoccupied by the classical film theory) but also rooted in the experiential dimension of the film spectatorship (a central issue in the contemporary film theory).

What is a worlding theory of the cinematic experience? To answer it is helpful to recall the first Lumière film program, held in 1895 at Salon Indienne. Although technically not the first time moving images were made or exhibited in a public venue, this program signals the beginning of a worlding mode of cinema for its unification of several experiential characteristics that constitute the cinema as we know it: the projection of light on a white screen to create life-size images, collective and seated viewing in a dark room, a certain length of stay that immerses the audience in another world of flickering images and ephemeral sounds. I identify this moment as the beginning of a world mode of cinema, for it is the first time a threshold is being crossed where a cinematic world is conjured up in front of its audience. In spite of their rational beliefs, the audience is impressed and seduced by this world for its uncanny verisimilitude. Fiction or non-fiction, photographic or non-photographic, cinema helps us to encounter an alternative world that not only builds on the so-called impression of reality but complicates our very relation to this reality: it engages our sense with what we take as real with considerable liberty and playfulness; it acknowledges our knowledge and perceptual schema of the world and uses them to put us under its spell. What the cinema does is not only to remind us of the world out there; it offers us its own cinematic world.

Once we identify this world mode of cinema, it becomes clear that it entails a mode of perception quite distinct (but not entirely separate) from cinema’s other modes (think abstract films or other film practices that focus on non-figurative images): the constituents of the aesthetic experience are quite different. The world mode of cinema caters to the desire for perceptual immediacy, characterized by full-fledged mobilization of sensorial faculties and embodied perception. It overwhelms the audience, albeit in a pleasing fashion, with excessive (nevertheless orderly) audiovisual details. It is a simulacrum of our waking experience, but much more heightened with numerous phenomenological intricacies offered by the audiovisual medium.

While the identity of this fluid phenomenon that we call cinema has been constantly changing throughout its more than a century of history, cinema has always maintained its aspiration to a world. To characterize cinema as a worlding experience therefore means to critically discern a mode of audiovisual perception that negotiates with but also exists independently of the issues of medium, economics of production/circulation, aesthetic agenda and socio-political implications. A world mode of cinema can exist across different technological supports (such as the physical medium of film strip), production practices (such as the Hollywood studio), and film genres (such as narrative fiction). Neither the newsreel footage nor the animation film should be excluded as long as it gives its audience a world to hold onto.

Taken from this “worldview,” the so-called new media are a natural expansion of what cinema has been throughout its history. Indeed, a great variety of display formats (ranging from TV, computer screen, handheld device to virtual reality headset) and means of storage and distribution (magnetic tape, laser disc, internet) have emerged since the latter half of the twentieth century. And the many significant reconfigurations of the aesthetic and social meaning of audiovisual media have expanded the platforms on which the cinematic world can be experienced. But this proliferation of media—which we perceive as “new” in a typically amnesiac fashion—has not diminished the relevance of this vital connection to the world. The decline (or even death) of cinema has been pronounced many times in its history. That might be true if we take cinema as the equivalent of a certain technological support or mode of image-making practice. If we expand the notion of cinema through its worldly ambition, however, cinema will exist as long as there is a world to inhabit. In fact, the world mode of cinema is being constantly rejuvenated through what it has learnt competing with other media practices. The rapid expansion of IMAX theaters in the last two decades, as well as the range of narrative fiction films that are now exhibited in this venue, point to one of the many ways in which movie exhibition can be creatively retooled to survive and evolve within the always intensely competitive media ecosystem.³

Looking from a 21^st century vantage point a theory of cinema should take into account its diverse practices, its medium promiscuity and its enormous range of cultural implications. But what also needs to be investigated is the persistence of a mode of cinema that engages the world. Conceiving the nature of the cinematic experience as an audiovisual world we can playfully inhabit, this dissertation focuses on the ways in which cinema brings us this world in all its liveliness: it theorizes how sound reinvents the diegetic world, which I call the audiovisual diegesis; it investigates the complex negotiations between sound and space; it tells the story of how cinema learns to speak and how the human voice is perceived in the history of the talking cinema. If, following an imagistic film theory, cinema cultivates an impression of reality, then sound can transform this impression into a sense of immersion within the filmic world; and it strives to make us believe that it is an authentic world to behold.

The Status of Sound in Film Theory

A worlding theory of the cinematic experience is already a non-trivial proposal; but to propose that the sound should play a key role in such a theory is nothing short of scandalous. The changing status of sound in film theory deserves its own history that I can only briefly outline here. A cursory look at how sound is portrayed in the history of film theory gives the impression of a melodrama: sound is either too noisy (an intruder!) or too dutiful (a servant!) in the house of images. Although diverse practices of sound existed at the outset of the cinema—one might even propose that since the recorded sound preceded the recorded moving images,⁴ cinema began with sound, rather than with images—sound never became a worthy subject in film theory until the so-called “advent of sound” in the late 1920s. With rare exceptions early film theorists shun away from engaging sound and regard cinema as essentially an art of the moving images.

In the wake of The Jazz Singer (1927) and the film industry’s rapid transition to sound, some film theorists reacted to sound with open hostility. Largely supporting filmmakers such as Sergei Eisenstein, Charlie Chaplin and René Clair’s warning against the talkie’s trivial realism, Rudolf Arnheim characterizes cinema’s medium specificity as that which cinema differs from reality. Sound, following this logic, needs to remain absent (Arnheim’s partial illusion thesis) or to be creatively juxtaposed to the images (Eisenstein’s counterpoint thesis). In both cases, only the absence of sound or its non-resemblance to how sound works in the real world can endorse cinema’s identity as a medium of artistic expressivity, to prevent its regression into the theatrical mode.

While sound certainly made a big noise in its first entrance into the temple of film theory, the contemporary turn of film theory again relegates sound to a corner of oblivion. The so-called contemporary film theory—a loosely defined body of critical works mostly dated from 1968 onwards—constitutes a radical break from classical film theory’s aesthetic fixation; but it throws out the baby with the bath water by disregarding the former’s productive debate on sound. Drawing discursive constructs from other disciplines to illustrate the nature of cinema, contemporary film theory throws fresh light on the question of what cinema is. Yet by stepping outside cinema’s own textual, historical and technological specificities it has no recourse but to champion the method of analogy⁵ and to aim at producing provocative yet vague claims. Most importantly, with rare exceptions this line of thought is only interested in sound in an extremely intermittent manner. Either sound is entirely ignored, taken out of the picture, or is it assigned subordinate positions that fail to acknowledge the entire century of its diverse practices. This is not the place to conduct a thorough critique of what I call the epistemological deafness of contemporary film theory; suffice to use one exemplary case to demonstrate how sound is muted in contemporary film theory’s bold maneuvers.

A case in point is Jean-Louis Baudry’s apparatus theory. I want first to give credit to this theory, a belated attempt to theorize a cinema that is already on the verge of morphing, for its seminal emphasis on the effects of cinema on its spectator. The evocative analogy Baudry made between Plato’s cave and the movie theater accentuates the mediating role of the so-called cinematic apparatus between the perceiving subject and what is to be perceived. Instead of the classical film theory’s focus on what makes the film (an art), here the concern shifts to what makes the cinematic experience. It is perhaps not the first time that a theory of cinema downplays the question of the medium and artistic sensibility and explores instead the connections between the technological conditions of cinema and its psychological effects. But it is certainly the first time the idea of cinema as a mode of experience is encapsulated in such a powerful analogy.

Baudry’s formulation of cinematic experience, however, is visibly constrained by the very analogy that makes it so provocative. Most crucially for me, this analogy misconstrues the nature of the cinematic experience in relation to the world. What serves as a critical but in fact largely rhetorical move in the apparatus theory’s overdetermined ideological agenda is the unjustified conflation between the chained prisoners in Plato’s allegory and the (in)credulous spectator⁶ already well seasoned in the world. Instead of spending her life in the cave, the movie spectator voluntarily enters and leaves the dark theater. The cave allegory in Baudry’s formulation conceives cinematic experience as fundamentally deceitful: however convincing and impressive this art of the shadows and echoes might be to its audience, it bears no relation to the world that is outside the cave. It grants them neither access to reality nor worldhood. This dissertation seeks to demonstrate the contrary. It argues that cinema provides its audience with a unique kind of access to world that is neither the evildoing of ideological manipulation nor the psychic fantasy of infantile regression. It is rather an experience that enriches us perceptually, emotionally and intellectually by exposing us to what is beyond our immediate surroundings. It is truly an experience of the world.

Baudry had little to say about sound—even Plato mentions briefly the faint echoes bouncing off the cave wall—a baffling omission considering the fact that it is a theory of cinema formed long after sound has firmly taken root. Although Baudry’s theory of cinema seems to have run its course, its repercussions can still be felt in what may be referred to as the Althusserian-Lacanian film theory, where sound is characterized in deeply problematic ways. It predictably regurgitates Baudry’s point of view by claiming that either sound is another form of deceit that embodies bourgeois ideology⁷ or, instead of Plato’s cave, the movie theater is now analogized to the uterus⁸—the deaf spectator in the apparatus theory is replaced by a blind subject that listens from inside a bag of liquid. Most importantly, Althusserian-Lacanian sound theory (if there is such a thing) often uses selective evidence (only the voice is conveniently considered) in ahistorical ways, far removed from the technological basis of cinema.

The World Heard: Towards a Worlding Theory of Film Sound

How is it that sound, a most exciting part of the cinema’s first century of history, can be so easily taken out of the plot? Is it the case that sound is truly secondary⁹ (and therefore deserves this denigration?) Or is it simply a result of our lack of attention (and insights) to sound matters? Considering the fact that the auditory faculty is a natural gift that most human beings are endowed with, it is rather curious why sound has never reached a natural¹⁰ position in the bulk of theories concerning a supposedly audiovisual medium.

This dissertation calls for a serious engagement with sound in the realm of film theory. In accordance to the recent proliferation of film sound studies that has produced numerous insights on the history of sound technology, individual sonic styles and cinematic genres heavily invested in sound strategies, it is perhaps time to infuse sound into the core of cinema. For the early film theorists, sound may have not earned its position in the cinematic art, hence its remiss in film theory; but today, carrying on with the same assumption by not giving sound its duly deserved attention would be straightforward discrimination. Ultimately, what needs to be addressed is not only a lacuna of sound in film theory, but how sound theory and film theory can work together so as to forge to a contemporary theory of cinema with a sound orientation.

Such a theory needs not to be drafted from scratch. Instead, my project draws from various post-WWII film theorists such as Andre Bazin, Siegfried Kracauer and Jean Mitry. Nourished by masterpieces of sound cinema these theorists follow the path paved earlier by Béla Balázs and Jean Epstein¹¹ and embrace sound as a legitimate element in film aesthetics. I am especially indebted to Andre Bazin’s cinematic realism, which to my belief offers the seed of a worlding theory of cinema. Although Bazin’s admittedly few discussions on sound cannot be weighted against his writings on the images, they suggest a theory of cinema where sound and image are equal partners contributing to one same goal. Thus the Bazinian cinematic realism points to, I believe, a way of leading sound back to the center of film theory, a way to incorporate a thorough investigation of film sound’s historical trajectory under the auspices of its worldly orientations.

Bazin’s attitude towards sound among film theorists is quite exceptional. Approaching cinema from a particular vantage point, Bazin didn’t have to struggle with a quasi-emotional attachment to silent cinema and to justify that sentiment with an essentialist view of the medium specificity of cinema. Bazin might not be the first film theorist to embrace sound (he was a little late to the party), but he is the first to propose a film theory that has a built-in place for sound, instead of an afterthought.¹² Unlike Kracauer¹³ and the French cine-semiology school that came after Christian Metz,¹⁴ Bazin did not offer a prescriptive classification of sound’s aesthetic functions; nor does he have to imagine, as Balázs had to, what sound should do to accomplish a truly humanist cinema. Bazin’s discussion on sound has on the one hand a pragmatic dimension in that he seldom talks about the issue of sound in abstract but instead roots his ideas in individual film’s use of sound. On the other hand, compared to virtually all other film theorists, classic and contemporary, Bazin assigns sound a crucial place in the evolution of the language of cinema. In the eponymous essay Bazin famously divides filmmakers into two categories: “those who put their faith in the image and those who put their faith in reality.”¹⁵ In what sense does a “faith in reality” constitute the (almost) opposite of a faith in the image? What does “faith” entail and where is the “reality” located? I am certainly not the first one to raise these questions. Yet I feel that a crucial aspect of Bazin’s reasoning, namely, how does sound have an effect on the “faith” or “reality,” deserves more of our attention. Indeed, Bazin expects sound to play a central role against this “faith in the image,” namely, plasticity and montage. In an oft-quoted (but seldom engaged) passage Bazin writes,

…the sound image, far less flexible than the visual image, would carry montage in the direction of realism, increasingly eliminating both plastic expressionism and the symbolic relation between images.¹⁶

Instead of taking sound as an additional element of montage, and therefore perpetuating an aesthetic agenda already fully developed in the silent film, Bazin proposes that the sound cinema constitutes a radical break from its past and that through certain uses of sound, cinema has forged a new identity for itself. Most importantly, Bazin recognizes that this distinction of faith lies not so much in the presence of sound but rather in the transforming role sound plays in contributing to what he calls the “faith in reality.” Although a deterministic reading of Bazin is by no means my intention, I find what Bazin calls “faith in reality” an apt description of what a worldly theory of cinematic experience entails. This theory sounds elusive precisely because it seeks to characterize the elusive nature of the cinematic experience itself. At different points of Bazin’s career he attributes this faith in reality to the ontology of the photographic image, long take, deep staging, or certain uses of sound. What lies beneath Bazin’s perceptive explications on these individual cases of “boosted” realism is a phenomenological ideal that manifests through stylistic choices. But it would be a mistake to define the Bazinian cinematic realism in terms of—and unfortunately often exclusively with—these stylistic choices. Instead the Bazinian cinematic realism is predicated upon a sense of perceptual presence that the film offers. Christian Metz once observes, “one is almost never totally bored by a movie.”¹⁷ This is perhaps because even the worst film is not entirely deprived of this connection to the world. “Occasionally films are, " Metz continues, “even in the absolute, very convincing. They speak to us with the accents of true evidence, using the argument that ‘It is so.’”¹⁸ While Metz has not addressed what distinguishes these two kinds of experience (almost boring vs. very convincing), this is clearly where Bazin’s fascination lies: when the cinema gives us its best, it offers us a worldhood.

As a film critic, Bazin’s acute awareness of cinema’s capacity to offer us a world shows up in a range of topics: editing, camera movement, acting, décor, and last but not the least, sound. His trademark perceptiveness is exhibited in his ability to grasp anything that stands out in the soundtrack for its worldly dimension. But most importantly what Bazin has to say about sound—few in quantity as they are—seems to align perfectly to his theory of cinematic realism. Instead of fragmentary discussions of the artistic use of this and that sound—which these days serve often as a trampoline for cultural interpretation—Bazin’s sound ideas form an integral part of his theory of cinema. In other words, Bazin demonstrates, admittedly in a latent form, the possibility of integrating a theory of film sound as part a theory of cinema**.**

It is regrettable that Bazin’s theoretical contribution to cinema has to end by the 1950s, since many of the technological experimentations in sound that have a profound impact on cinema auditorship took place in the latter half of the 20^th century. Although Bazin has experienced and written about the experiments on widescreen (cinemascope, Cinerama) and 3D, like his contemporaries he could not have guessed the leading role sound has played in cinema’s recent technological evolution. Yet in delving into this history, namely, film sound’s coming of age, I found Bazin’s response to the perpetual effects of technological innovations of cinema nothing short of exemplary. On the one hand he displays an in-depth knowledge of the technical nature of these innovations, which in turn enables him to see the true potential of new forms of cinematic worldhood in its very process of formation. On the other hand Bazin maintains a critical distance from the industry’s discourse on technological progression towards sensationalism. He holds his ground firmly thanks to his theory of cinematic realism, a theory forever in the developing, enriched by every new direction of the technological means and their artistic adoption and sublimation.

That the cogency of the Bazinian film theory has much to do with sound can be hardly contested, yet the exact role of sound in Bazin’s own writings remains underarticulated. In a recent essay Tom Gunning has elaborated on the cogency of the Bazinian cinematic realism:

…we must now raise again the question that Bazin asked so passionately and subtly (even if he never answered definitively): what is cinema? What are cinema’s effects and what range of aspects relates to its oft-cited (and just as variously defined) realistic nature? Given the historically specific nature of Bazin’s arguments for cinematic realism as an aesthetic value (responding as he did to technical innovations such as deep focus cinematography and to new visual and narrative styles such as Italian Neorealism), it makes sense for a contemporary theory of cinematic realism to push beyond those aspects of cinematic realism highlighted by Bazin. Specifically, we need to ask in a contemporary technical and stylistic context: what are the bounds that cinema forges with the world it portrays?¹⁹

It could be said that my project responds to the above passionate call for action. Building on Bazin’s case, this dissertation seeks to demonstrate how exactly sound can contribute to a contemporary theory of cinematic realism by proposing sound as a critical component in cinema’s capacity of world building. By questioning the inactive status of sound in contemporary film theory, I aim to place sound at the center of the unique kind of worldhood that cinema offers and call for a theory of the cinematic worldhood with sound as a founding member.

Chapter Layout

I divide the dissertation into three parts where each part corresponds to what I believe an important aspect of the cinematic worldhood offered through sound. One might say that these aspects form three largely independent sonic paths that lead to the filmic world. They are not exhaustive accounts of what the filmic world might entail in terms of sound, but rather, what I see as conspicuous trajectories of sound in a worlding theory of cinema. Each of these trajectories juxtaposes a theoretical exposition with historical survey where the theoretical proposals are substantiated in the history of cinema and tested against concrete instances of filmmaking.

The first part, “Hearing the World of Film,” engages with a central term that we use to characterize the perception of the filmic world, namely, the notion of diegesis. Examining the epistemology of the term, I call for a return to its original conception (Souriau): the world of film. Instead of taking for granted a fixed meaning (and debating on its (non)applicability in contemporary films), I argue that we need to adopt an evolutionary sense of the term by investigating the historical moments where the term’s meaning changes radically. The advent of sound, therefore, constitutes a landmark moment where the image-based diegesis is transformed into what I call the audiovisual diegesis. The notion of audiovisual diegesis is not only a theoretical solution to the aporia of diegetic/nondiegetic sound; it offers a model through which a historical account of the use of sound effects in early and silent cinema can be traced. This historiography is crucial in establishing the many different conceptions of how sound functions in relation to the diegesis (sonic attraction, punctuation, acoustic signifier, etc.) that lead up to the sound cinema where they continue to exist to various extents in the transition years and beyond. The emergence of audiovisual diegesis nevertheless constitutes a truly different conception of sound in the history of cinema for it forges and consequently demands new sound-image relations that are germane to this perception. The formation of audiovisual diegesis not only changes our relation to the sounds, it also changes our relation to the images—both now become an integral part of a larger perceptual schema that is the world. Finally, through a comparative analysis between two canonical films—M and Kameradschaft—I give a concrete example of how the terminology and historical context developed in this chapter can inform our understanding of the stylistic choices of individual films. Equipped with the notion of audiovisual diegesis, we might be in a better position to understand not only how exactly films in the transition era sound differently but also what are the paradigms implied in these differences.

The second part, “Immersed in Sound,” focuses on how the cinematic world immerses its audience with sound. Clearly, human vision and audition already process space in very different terms. Cinema further complicates the issue by adding layers of technological mediation between the world and our perceptual faculties. How does sound convey a sense of space in cinema? Why does the industry often conceive its ultimate goal in the technological evolution of cinema as a sense of immersion? The point of this part is not to come up with an ahistorical notion of immersion but to see it as a more reflexive framework capable of generating new lines of inquiry and revising itself in view of the empirical formations it explores. What does immersion really mean in terms of sound? If it refers to the palpable spatial aspects of the world heard in films, what are the models of spatial perception proposed and implemented in the history of cinema sound? To answer these questions, the second chapter surveys the lineage of sound technologies aimed at curating a spatial experience at the movies. It proposes a particular narrative that makes salient a contrast or a paradigm shift from the embedded space to the embodying space.

After the theoretical deliberations, the third chapter presents a case study of one of the most technically immersive films of the current decade, Gravity (2013). Not only does the film rewrite the rules of camera movement thanks to its combination of robotic motion control camera and LED lighting panels, but its immersive sound finds perfect use of the newly emerged Dolby Atmos platform. My study offers a detailed account of the film’s sonic achievement and contextualizes its practice of sound space in the historical development of immersive sound. Although Gravity is a film set in the outer space that carries no sound, the spatial dimension of the sound proves to be not only relevant but also exemplary. Indeed, the film offers a perfect example, albeit an extreme one, of issues related to cinema’s exploration of sound space such as voice panning, vocal proxemics and radio mike.

The third and final part, “Cinema Learns to speak,” deals with an essential aspect of sound cinema, namely, its adoption of the human voice. Needless to say, the presence of voice constitutes a true game changer for cinema in its worldly offerings. The so-called advent of sound in the late 1920s is in fact largely the advent of voice. Yet how did the voice exist before and after this event? In the fourth chapter I propose the term “perceived authenticity” to characterize our perception of the human voice in cinema. I chart the trajectory of how this perception evolves and how it shapes cinema’s century long attempts to tame the human voice. This chapter conceives the process through which “cinema learns to speak” as an ongoing process: instead of remaining stable, our perception of the voice in cinema constantly fluctuates in response to technological changes, film styles, distribution strategies and so on, and this fluctuation constitutes a vital aspect of the cinema auditorship.

The fifth and final chapter gives a sense of closure to the dissertation by analyzing a film serendipitously titled The World (2004). I start my investigation on a detailed account of the use of dialects in this particular film. Then I move onto a broader context of the recent wave of dialect films in Chinese cinema where Jia Zhangke is the foremost proponent. The emergence of a cinema of dialect—a phenomenon that can be founded in many national cinemas—calls for a theoretical framework in which the aesthetic qualities offered by a heteroglossic representation of languages can be productively examined. Ultimately, the dialectal voice corroborates the previous chapter’s theorization of the phenomenal authenticity of the voice in that it points to a series of “new” developments beyond the transition years that exploit this very “old” quality of voice.