This Is What Classic Novels Sound Like When a Computer Turns Them Into Piano Music

Music and literature are both masters at evoking emotion. The ingenuous beats of Pharrell’s “Happy” can put a smile on the face just like the cathartic life-mashing of Mr. Darcy and Elizabeth Bennet. But two researchers wanted to find out what would happen if they created a direct link between the two art forms, building a program that would automatically translate the emotions of a written text into a unique musical score. The result, presented this month in Canada and Sweden by artist Hannah Davis and computer scientist Saif Mohammad, is both impressive and just the beginning.

Here, for instance, is what happens when their system, which they delightfully named TransProse, eats up playful Peter Pan and spits out piano music:

And here is what happens when that same system translates Cormac McCarthy’s blight-filled masterpiece The Road:

At the heart of their work is the budding field of sentiment analysis, or attempts to teach computers to detect emotions in text, often by using coded keywords. At its most basic level, sentiment analysis tries to categorize text — like tweets — as positive, negative or neutral, which has become information of serious value to marketers, politicians or anyone trying to sell a product. While even that level of analysis is still a flawed practice (programs have a hard time accounting for sarcasm, for instance), some researchers have been much more ambitious.

Davis and Mohammad, who works at Canada’s National Research Council (NRC), used a collection of 14,000 common words Mohammad helped code, known as the NRC Emotion Lexicon. Thousands of people were recruited online and asked which of eight basic emotions — anticipation, anger, joy, fear, disgust, sadness, surprise and trust — they associated with each word, and the results, released in 2013, offer an imperfect but robust foundation for the kind of work TransProse does.

The word aching, for instance, is classified as negative; a computer using Mohammad’s lexicon will also detect sadness. Countdown is associated with anticipation; criminal with anger, disgust and fear; respect with anticipation, joy and trust. Some words, like maternal, are coded as both positive and negative, while other words, like screen, came out of the annotating process without significant links to any of the emotions.

When it came to producing music, the researchers built a program that would read the text and produce an emotion profile, based on which emotion-packed words were used by the author and how often. They then used music theory to build a system that would assign values for features like key, octave and notes. Texts that had more positive words overall were translated in C Major, for instance, because major keys are associated with happiness. The octave was determined by the difference in how many words related to sadness or joy appeared; hence The Road’s deep rumbling score above. More emotion words of any kind meant more notes, while fewer yielded a more sparse piece of music. Consonant notes reflect joy and calm, while discordant notes embody exciting, emotion-packed parts of a novel.

“The basic, basic emotions are definitely coming through,” says Davis, who has been working in data visualization since graduating from New York University with her master’s last year. “It’s a first iteration.” She not only wants to fine tune the current system, limited to three melodies played on a piano, but add more instruments, generate longer pieces and try analysis of different texts — eventually creating a tool that could turn a State of the Union speech into a symphony in a split-second or transform Hamlet, with strings of notes representing dialogue of various characters, into a cohesive song.

“We are connecting literature with music and we chose emotions as the medium to connect them,” says Mohammad, “but there are other things we can capture as well and it works for any piece of text. We are limited only by our imaginations.”

Still, their initial results do a decent job of embodying the spirit of a storyline. And their work serves as a reminder that we should all be grateful not to be stuck on a Belgian riverboat in the Congo:

To hear the scores their system produced for nine different novels, visit the TransProse website here.

This is an edition of Wednesday Words, a weekly feature on language. For the previous post, click here