A Foreign Sound To Your Ear: Digital Image Sonification for Historical Interpretation

what does an image sound like? for a synesthetic digital humanities.

Essay now published in Digital Sound Studies (October, 2018).

My essay, “‘A Foreign Sound to Your Ear’: Image Sonification For Historical Interpretation,” is planned for inclusion in the upcoming edited collection, Digital Sound Studies, due out from Duke University Press in 2018. Here is an except. This is a draft, so please do not republish without permission of the author (you may link to this version). Comments welcome as this moves toward the final version.

“A Foreign Sound to Your Ear”: Image Sonification For Historical Interpretation

Michael J. Kramer

So don’t fear if you hear / A foreign sound to your ear— Bob Dylan[1]

Introduction: Mance Lipscomb’s Silhouette

Photographs are visible, but photography is not only a “visual” practice.—   Margaret Olin, Touching Photographs[2]

Mance Lipscomb, African-American songster and sharecropper from Navasota, Texas, sits in a chair, center stage at the Greek Amphitheater. It is a beautiful summer day in 1961. He occupies the middle of a black and white photograph. Guitar across his lap, microphone stand rising up in front of him, he performs to what looks like a full house at a facility built in 1903 in the hills above the University of California in Berkeley. Modeled after the theater at Epidaurus and funded by California newspaper magnate William Randolph Hearst, the Greek Amphitheater was a symbol of the aspirations of Cal to become the “Athens of the West” and link itself as a twentieth-century American research university to the democratic traditions of classical antiquity.[3] There, before the crowd in 1961, sat the unlikely figure of Lipscomb, a working-class black man from the rural South transplanted to a hotbed of the primarily white, middle class, 1960s folk music revival, the Berkeley Folk Music Festival, which took place annually between 1958 and 1970 on the University of California’s flagship campus.

…

Only, we are not really “witnessing” history here in any direct or unmediated way. To consider Lipscomb at Berkeley in 1961 from our contemporary digital vantage point means paying close attention to what a digital image is as compared to a photographic one. Rather than bemoan what at first might seem like a further step away from the past, rather than feel dismay at the receding historical moment, we can embrace the new possibilities for interpretation and analysis raised by the passage of Lipscomb’s image into the digital medium. The question becomes: What do we do with a photograph when it goes digital, when it becomes data? Much has been written, of course, about photography itself as an odd medium.[4] The nature of all archival sources as partial representations—palimpsests—of the past is well known too.[5] Furthermore, much ink has been spilled (or more accurately these days, many characters have splashed across computer screens) about the folk revival itself as a mediated cultural setting.[6] These literatures share an interest in in questions of authenticity and the vexed project of capturing and conveying the real within the shifting contexts of “remediations” both material and social. They all pay close attention to how we access the “reality” of the past. When our source materials, our documentary pathways into history, undergo transmogrifications such as the move into digital form, how should we respond in our pursuit of understanding what has come before us? In other words, how, exactly, do we look at Mance Lipscomb’s digital silhouette?

This essay argues that one surprising way to look at Mance Lipscomb’s silhouette when it goes digital might be to listen to it. When it comes to images from the folk revival, the transformation into digital data is not a loss of direct access, but rather an exciting opportunity to think more sensitively, sensuously, adventurously, creatively, insightfully, and even more “insoundfully” about the past. Most of all, the digital medium opens up ways of recalibrating the privileging of the optic over the aural in historical investigation.[7] If we let go of two fantasies—the fetishizing of sound as some kind of pure, unmediated, and direct form of experience and the privileging of sight as a more convincing mode of abstract reasoning and connection-making—digital technologies become a means for deepening comprehension of both the visual past and what Jonathan Sterne calls “the audible past.”[8]

The peculiar quality of the digital is that at root it encodes the world, including its archival materials, into an organization of on-and-off electronic pulsations. We metaphorically refer to these as ones and zeroes or bits and bytes (even calling them on-and-off pulsations is, technically, an abstract representation of physical processes). The act of digitization transforms all media and information into the same modular code so that it becomes more effectively machine-readable. In doing so, it makes new interchanges between visual and aural mediations of encoded information possible. By thinking carefully about the programs, software, and tactics we use to transform encoded machine-readable digital information into human-readable forms, we have the opportunity not so much to access the past in some more magically more “objective” manner, but rather to remediate it yet again through digital processes that expand the range of ways by which we interpret evidence. These extend historical accuracy precisely by allowing us to register the multidimensional fullness of the past as represented through its evidentiary artifacts.[9]

The seemingly fateful move away from the privileging of the so-called original artifact by digitizing it turns out, paradoxically, to offer opportunities for getting closer to the past. Not because digital data—or any numerical data for that matter—is somehow more true, but rather because code and software allow historians to pivot between representations of the past in enhanced synesthetic ways: in the case of this article and my research, visual information can now transit through creative reprogramming of the data into correlated sound compositions; these sound compositions are aural versions of the visual code; they allow historians to lend an ear to the images as well as staring them down. At a variety of scales, from one photograph of Mance Lipscomb at the Greek Theater in 1961 to the more than 10,000 photographs in the Berkeley Folk Music Festival Collection, historians can now listen to image as well as look at them, thus broadening perceptual approaches to the evidentiary record.

While much work has been done in terms of “visualization,” or the rendering of data, including sound, in digital graphs, charts, trees, tables, maps, animations, and other visual representations, far less has been done the other way: turning visual data into sonic form.[10] Once different media become code, what we might call “sonification,” or the transliteration of data from other inputs, such as visual ones, into sonic outputs, becomes a potential avenue for analyzing how source materials convey information about and interpretation of the past. This is so not because sound is somehow recoverable in some pure, unmediated form from images (or audio recordings for that matter). Digital processes cannot recover the precise sounds being made in a photograph (at least not yet!). Rather, moving data drawn from visual sources into new sonic forms requires historians to draw upon more than just what they see to make sense of sources and the historical meanings they represent; data sonified allows us to widen our historical ears to listen for patterns and the truths they suggest as well. Particularly when it comes to a historical topic such as the folk revival, which mattered to so many people as a musical and sonic phenomenon, new perspectives on history can emerge through heightened synesthetic awareness arising from the movement of data between the optic and the aural.

The use of digital remediation to expand representations of artifacts shifts historical study from a focus (short-sighted, if you will) solely on the epistemological issue of how we know to the ontological level of what a digitized artifact such as a photograph from 1961 is and the phenomenological question of how we perceive it. Ontologically, the digital image file is not the photograph is not the negative is not the moment in 1961 on stage at the Berkeley Folk Music Festival is not Lipscomb on his front porch in Navasota. Or more accurately, it is all those things, but they are remediated through binary code and software. This remediation, however, with its particular qualities of being able to pivot between correlations of visual and sonic media, makes the phenomenological perception of evidence a moment for sparking new comprehension. From as simple a technique as accompanying a photograph with a song to the amassing of visual and sonic information together in one virtual space to the transformation of visual inputs into sonic outputs, sonification can intensify our awareness of the compositional, representational, and affective dimensions of source materials. Even what at first might seem like computational distortions, or what digital literary scholars sometimes call “deformances,” potentially lead to greater accuracy in our comprehension of the past if computation is deployed and analyzed in inventive and thoughtful ways.[11]

The pivoting between vision and sound has a long history, particularly in terms of visualization tactics in music and especially folk music studies. Scholars have long drawn upon technological means to convey and analyze sound more robustly. These have included spectrographs and other approaches to creating non-standard musical notation. No less a figure than one of the founders of American ethnomusicology, Charles Seeger, father of folk revival mainstays Pete and Mike Seeger and himself a frequent speaker at the Berkeley Folk Music Festival, developed a comprehensive theory of notation and, in 1956, supported the building of the first electronic music writer in the United States, the melograph.[12] These endeavors, while fascinating, rely on the presence of sound in aural form then rendered visually. What do we do in the case of Mance Lipscomb on stage at the Greek Theater in 1961, in which there are no known audio recordings that one might pair with the image to create a kind of dialectic between captured image and sound? What follows are three modes of sonification that draw upon the specific peculiarities of the digital to extend analysis synesthetically, in service of sparking new perspectives on the past through creative recompositions of source materials.

First, digital sound design draws upon practices in theatrical and cinematic sound design to pair up related images and sounds to create a new compositional representation of the past that can illuminate—and amplify—historical meaning: we may only have images of certain performers at the Berkeley Folk Music Festival and no sound, but we can, for analytic purposes, bring photographs together with related audio recordings just as a documentary filmmaker might. Second, a certain kind of digital sound design we might refer to, using Lev Manovich’s term, as data fusion concentrates on the combining data in one mediated space to produce historical knowledge that neither mode of data could accomplish on its own: for instance, we can fusing musical recordings with a geographic map that was, in original form, inspired by sound. Finally, data sonification allows not for sound to be conjoined to images, but rather for it to emerge from the data of the visual medium itself so that we can discern visual patterns and their affective and semantic significances by, both literally and figuratively, sounding them out. These three activities—digital sound design, data fusion, and direct sound sonification—suggest how creative tactics borrowed from the arts can help historians investigate their source materials more closely. They remind us that while the digital may seem a further step away from the sensorial with its so-called virtual world, it also enhances sensorial approaches to the past. We can perceive the many dimensions of Mance Lipscomb’s silhouette not only by gazing at it closely, but also by listening to it more attentively.

---

---

[1] Bob Dylan, “It’s Alright Ma (I’m Only Bleeding),” Bringing It All Back Home (Columbia Records, 1965).
[2] Margaret Olin, Touching Photographs (Chicago, IL: University of Chicago Press, 2012), 1.
[3] One scholar cites the crowd that day as 41,000, which may have been the total attendance at the Berkeley Folk Music Festival in 1961, but the Hearst Greek Theater has a capacity of 8,500. See Glen Alyn, “Introduction,” in I Say Me for a Parable: The Oral Autobiography of Mance Lipscomb, Texas Bluesman, As Told To and Compiled by Glen Alyn (New York: W.W. Norton, 1993), 26. On the history of the Greek Amphitheater, see Carol Hyman, “UC Berkeley’s Greek Theatre turns 100 years old this month,” 11 September 2003, http://www.berkeley.edu/news/media/releases/2003/09/11_greek.shtml.
[4] The strange qualities of photographic representation have been noted since the creation of the technology itself. See, for instance, Roland Barthes, Camera Lucida: Reflections on Photography, trans. Richard Howard (1980; New York: Farrar, Straus, and Giroux, 1981); Susan Sontag, On Photography (New York: Farrar, Straus and Giroux, 1977); and Olin, Touching Photographs.
[5] This theme has been noted in terms of archival research in general, of course. Among many other studies, Arlette Farge, The Allure of the Archives (1989; reprint: New Haven, CT: Yale University Press, 2013) and Carolyn Steedman, Dust: The Archive and Cultural History (New Brunswick: Rutgers University Press, 2002). For an article on the archive in relation to digital history, see Robert Darnton, “The Good Way to Do History,” New York Review of Books, 9 January 2014, http://www.nybooks.com/articles/archives/2014/jan/09/good-way-history/.
[6] See, for instance, Benjamin Filene, Romancing the Folk: Public Memory and American Roots Music (Chapel Hill, University of North Carolina Press, 2000).
[7] The field of sound studies has begun this project of recalibration. See books such as Alain Corbin, Village Bells: Sound and Meaning in the Nineteenth-Century French Countryside, trans. Martin Thom (New York: Columbia University Press, 1998) and Mark M. Smith, Listening to Nineteenth-Century America (Chapel Hill: University of North Carolina Press, 2001). For overviews, see Jonathan Sterne, ed., The Sound Studies Reader (New York: Routledge, 2012) and Michael Bull and Les Black, eds., The Auditory Cultures Reader (New York: Bloomsbury, 2004).
[8] Jonathan Sterne, The Audible Past: Cultural Origins of Sound Studies (Durham, NC: Duke University Press, 2003).
[9] See, among other studies, Lev Manovich, The Language of New Media (Cambridge, MA: MIT Press, 2001) and Software Takes Command (New York: Bloomsbury, 2013).
[10] See, for instance, Franco Moretti, Graphs, Maps, Trees: Abstract Models for Literary History (New York: Verso, 2007), Stanford University’s Spatial History Project, http://web.stanford.edu/group/spatialhistory/cgi-bin/site/index.php, and David J. Staley, Computers, Visualization, and History: How New Technology Will Transform Our Understanding of the Past (Armonk, NY: M. E. Sharpe, 2013).
[11] On “deformance,” see Jerome McGann and Lisa Samuels, “Deformance and Interpretation,” New Literary History 30 (Winter 1999): 25-56; see also Mark Sample, “Notes Toward a Deformed Humanities,” 2 May 2012, http://www.samplereality.com/2012/05/02/notes-towards-a-deformed-humanities/; and Michael J. Kramer, “Navigating the “Screwmeneutic” Circle,” 2 May 2012, https://www.michaeljkramer.net/?p=657.
[12] See Charles Seeger, “Prescriptive and Descriptive Music-Writing,” The Musical Quarterly 44, 2 (April 1958): 148-195. On the melograph, see Ann M. Pescatello, Charles Seeger: A Life in American Music (University of Pittsburgh Press, 1992), 212. The effort to notate sound is deeply connected to interest in “folk” and vernacular music. See, for instance, Bruno Nettl, “I Can’t Say a Thing until I’ve Seen the Score: Transcription,” in The Study of Ethnomusicology: Thirty-one Issues and Concepts (1983; revised, Urbana: University of Illinois, 2005), 74-91. Thanks to Mary Caton Lingold for reminding me of this point and Charles Seeger’s important role in expanding musical visualizations beyond standard Western notation.