For a Synesthetic Sound Studies

whatever forms they take, artifacts are full of information, of ephemeral moments in the past that packed a punch then, and, interpretively, still do now, in the present. a more imaginative phenomenology of how we perceive them can feed better ways that we preserve them—and vice-versa.

A short video talk for Sound Submissions 4: Developing Infrastructure to Preserve Sound Panel at the Radio Preservation Task Force Conference 2023: A Century of Broadcasting—Preservation and Renewal.

Sound recordings are worth preserving in of themselves, of course, as artifacts that help us access and study the ephemeral past, but they are also important because they offer the opportunity to think about how we analyze all types of materials. Listening better can help us look better, perceive better, preserve better, and analyze better. Particularly as so many artifacts converge in the more ductile and transposable form of digital code, shifting us to the sonic register accentuates surprising new paths to understanding what our artifacts have to say about the past.

I certainly found this to be the case in my current research on the Berkeley Folk Music Festival. A 30,000-plus artifact archive of a folk music festival that took place between 1958 and 1970 on the campus of the University of California offered enormous possibilities for documenting and studying an important, but underappreciated, event in the history of the 1960s folk music revival. As a music festival, here was a sonic event that was temporary but immersive, casual but also in its festive way very communal—an event that was full of the kinds of sensory, ephemeral experiences that are so often the stuff of living yet can so oddly fade from the historical record. Yet despite the fact that the Berkeley Festival was broadcast on the radio a number of times, and that its founder and director started out by hosting a radio show, there was, sadly, very little audio preserved from the event. What did exist in the archive, however, was a robust written and visual record, particularly over 10,000 photographs of the event.

As we digitized the collection and began to consider how to interpret it, the lack of preserved audio, of saved sound, got me to thinking about a strange question: does a sound studies approach have something to offer us even when we have few sound artifacts? To be sure, we can look to the many clues to sonic experience and its historical meanings embedded in visual artifacts: who is performing, what instrument are they using, what size and acoustic qualities did the spaces have, how is the audience reacting, what amplification systems are documented in photographs? And much more.

Yet I found this not to be enough. Why do we only think about sound as sound and image as image, particularly as artifacts all become digitized and in fact exist now, at the material level, in the same form: as computerized code, and really at the most basic level as on-and-off electronic pulsations? As scholars from Jonathan Sterne to Viet Erlmann to Emily Thompson have taught us, the senses were not always understood to be so discrete from each other anyway. Now, even as we think of our sensoria as separated into five completely different realms, the things we sense to, as it were, make sense of the past—which is to say the artifacts—in fact themselves grow more integrated as digital material. What if we started to preserve and analyze them in a more synesthetic spirit?

I began to wonder as I looked over the marvelous photographs of the Berkeley Folk Music Festival in digitized form, what might a photograph sound like? What if we insisted on running our silent artifacts through sonic filters to try to amplify their details, their qualities, what they have to tell us out loud about their silent preservations of the ephemeral past? What if we adopted sound not just as a mode of preservation, but also as a mode of analysis itself? What if we not only used our artifacts to “glimpse” the past, but also to hear it?

In the last few years, informed by the question of what digital humanities approaches can do that other modes of preservation, access, and analysis cannot do, I have been starting to try to map out this strange notion of listening to images. Might the convergence of artifacts into digital code allow us to transit among different sensorial outputs of their data? How might computational means of more inventive interpretation result, for example, from asking what a visual artifact’s data sound like. Pixel brightness, shapes and morphologies, measurements of bodily or spatial relationships: a photograph captures many aspects of a past moment. We can see these, but sometimes we look so much, so often, that we don’t see what is right before our eyes, or even what be hidden from our eyes but caught by the camera lens, like the infamous dead body in the photograph from the film Blowup. Re-representing visual forms sonically might offer better unveilings of evidence by asking us to move between what we hear in visual data and what we see.

Image sonification adds to our analytic repertoire in two ways. First, it alerts us to the representational quality of the artefactual record. After all, photographs are already bizarre technological feats worthy of close philosophical attention, as anyone who has spent time with, say, Barthes’ Camera Lucida, learns. They are at once a moment from the past and a fictive indexing of that moment. Instead of bemoan this fact—that our empirical visual record is in fact constituted by representations—let’s make the most of it. Not just: you have to see it to believe it, but also, maybe, you have to hear it to analyze it fully. Making sense of something means using al the senses. A synesthetic approach.

My early, rudimentary explorations of image sonification have made for very strange sonic experiences. Sometimes even the feeling of: what’s that? Or, that’s it? But, I have also found also that these experiments, by disorienting the senses, also perhaps reorient. For instance, a sonification of Mance Lipscomb performing at the 1963 Berkeley Folk Music Festival, the second time he appeared at the event, alerted me to how one might more robustly interpret the image’s emphasis on his silhouette in front of an audience listening intently to him. When I sonified the image in a very simple way, using Michel Rouzic’s Photosounder app to produce pink noise from the pixel density, scanned left to right across the photograph, the staticky noise when quieter when it reached Mance Lipscomb’s silhouetted body. Here, I realized, quiet had something to say.

Lipscomb, who arrived at Berkeley first in 1961 and returned two years later, was making some noise. He was singing center stage. This African-American sharecropper from rural Navasota, Texas, leapt across social classes and locations, during the height of the folk revival and also of the modern African-American civil rights movement. And yet he was something of an enigma, a blank space onto which the more middle-class, primarily white audiences of the folk revival could project their own feelings and fantasies. The photograph captured this negotiation of space, place, power, and beauty—of the fraught but potent efforts to remake social connections in an American society wracked by racism and other ills—but as for me, I didn’t realize or really see this until I heard it. Hearing sonic static helped me see better.

This very early example of image sonification experimentation does not blow me away, mind you. It goes by quick, it’s suggestive not decisive. But, so what? It was more in the process of making it and listening to it, experimenting and exploring, and in preserving various new iterations of the image sonically that my analysis sharpened. Here is a look…and a listen [image sonification].

Maybe a little too weird for many tastes, I know, but for me, seeing and hearing, silence and noise, stillness and motion, together can spark new insights, and, we might say, insounds. Our artifacts, whatever forms they take, are full of information, of ephemeral moments in the past that packed a punch then, and, interpretively, still do now, in the present—if we remain alert to their forms and to a more imaginative phenomenology of how we might perceive them. Image sonification is just one of a number of approaches we might investigate to develop a more synesthetic sound studies: one could fuse visual and sonic data from various sources to create mini-filmic documentaries and pay attention to how images and sounds converge; one could even glitch images and sounds together to let computational serendipity generate new perceptions and interpretations. As surrealism has taught us: from distortions can come clarifications!

Finally, new ways of seeing visual data of the past by hearing it also presents some questions about the relationship of analysis to preservation in the digital era. What is it we should preserve or save now? The original artifacts, to be sure, but also, as media archeologists teach us, perhaps the layers of representation and subsequent exploration as well. In the Mance Lipscomb case, we might develop mechanisms for preserving the original photo and its technological and social context, but we might also find ways of preserving the interpretive work that arises out of the digitization of the photo. The code of the digital image, yes, but also, perhaps, subsequent sonifications of it: the “original” artifact, but also the ways this or any any original artifact is always also the choices made about how to re-represent it. There is no one true representation here to discover and preserve, only an endlessly generative array of re-representational possibilities. That is not a truth about the empirical record to fear, necessarily, but rather one we can embrace and cultivate, I would argue. Analysis and preservation go together.

The ephemerality of sound alerts us to this need for creative and effective preservation and analysis as connected endeavors. Sound fades away, but sound is also profoundly potent and immersive. Sound, like history itself, is time vibrating perceptually. Sound is in the very ether, on the air, waving to us as it recedes into the universe, into the past. Sound never ends even as it disappears into memories. Our task, if we listen to the call close enough, is to hear sound soundly, every which way it transits and moves, even when it is to be found, synesthetically, in material and artifacts such as images.

Thanks. Questions? Comments?

Leave a Reply

Your email address will not be published. Required fields are marked *