Voice Analytics: You Sound Emotional

Imagine you ask Siri to find a restaurant nearby. She replies, "You sound kind of down, so I'd recommend a warm and quiet spot a few blocks from here."

How did Siri know how you were feeling? Welcome to the world of intonation analytics.

Humans read emotion in verbal communication all the time: I don't like your tone! -- An edge crept into his voice. -- She spoke soothingly. But the science of sentiment analysis is still relatively new and focused primarily on text.

As explained by data scientist Brian Kolo on the Opera Solutions blog, algorithms have a hard enough time differentiating "that's hot" (as in, good) from "that's hot!" (as in, you'll burn yourself). It seems a stretch to suggest they can understand how we feel based on our intonations.

Or does it? At yesterday's Sentiment Analysis Symposium in New York, executives from Beyond Verbal demonstrated software that seems to pick up on these cues, too.

"It's not what you say, but how you say it," explained Dan Emodi, vice president of marketing at Beyond Verbal. "Vocal intonations convey your mood, attitude, even personality."

The research began 18 years ago, when a physicist and neuropsychologist began exploring the way that babies understand language even before they understand words, Emodi said. The team discovered that intonation transcends language and culture.

"Patterns of happiness, sadness, aggression, introversion, extroversion, apathy, and action-orientation, they're all the same across languages," he said.

On Beyond Verbal's site, vocal analytics are applied to the speech of President Barack Obama, Steve Jobs, and Princess Diana. The company also analyzed Edward Snowden's June interview with The Guardian, finding that his voice betrayed some egocentricity and pride in orchestrating his NSA document leak.

You can listen to me, but do you hear me?
Like most sentiment analysis projects, Beyond Verbal started with a set of samples -- 16,000 audio clips, in this case -- and trained its algorithms using crowdsourced human evaluators. Temper and anger were easiest to detect accurately, Emodi said. "Call centers have plenty of aggressive tones." Over the years, the training sample grew to 32,000 clips.

The potential applications are limited only by imagination, he added. Contact centers, market research firms, dating sites, search services, and many other businesses could benefit from this kind of technology.

I asked about law enforcement, and Emodi said the software has caught the attention of Homeland Security officials. Another audience member asked about healthcare applications, where the software could detect confusion in patients suffering from early stages of dementia, for example.

"We've been contacted by healthcare companies," Emodi said. "When we started this, we never even thought about healthcare."

Beyond Verbal's analytics happen in the cloud, so individuals can try it on the website or via the Moodies iOS app.

I gave it a shot this morning, and as you can see in the screenshot, it seems to have me pegged. "It helps us get a better understanding of our own selves," Emodi said.

What do you think, members? Do you see intonation analytics changing the way we interact with machines? What other applications might benefit from intonation analytics? Make your voice heard (but watch your tone) below.

— Michael Steinhart, Circle me on Google+ Follow me on TwitterVisit my LinkedIn pageFriend me on Facebook, Executive Editor, AllAnalytics.com

Related posts:

Michael Steinhart, Contributing Editor

Michael Steinhart has been covering IT and business computing for 15 years, tracking the rising popularity of virtualization, unified fabric, high-performance computing, and cloud infrastructures. He is editor of The Enterprise Cloud Site, which won the Least Imaginative Site Name award in 2012, and he managed TheITPro.com, a community of IT professionals taking their first steps into cloud computing. From 2006 to 2012, Steinhart worked as an executive editor at Ziff Davis Enterprise, writing and managing research reports, whitepapers, case studies, magazine features, e-newsletters, blog posts, online videos, and podcasts. He also moderated and presented in dozens of webinars and virtual tradeshows. He got his start in IT journalism at CMP Media back in 1998, then moved to PC Magazine, managing the popular Solutions section and then covering business technology and consumer software. He holds a Bachelor of Arts degree in communications/journalism from Ramapo College of New Jersey.

Biosensors, Robots Replacing Human Guards

In prisons and other security settings, electronic devices are predicting and detecting anomalous behavior with high accuracy and reducing the need for human personnel.

Questioning the Ethics of Learning Analytics

Schools around the country are weighing the benefits and ethical questions of learning analytics.

Re: Pretty accurate!
  • 3/10/2014 1:43:37 PM


Michael writes

I'd encourage everyone to give this a try and see how it reflects your emotions and mood


Well, darn. The software could not analyze my voice. I used the same microphone that provides over 99% accuracy with Dragon, so I dunno what the problem was. I'd like to think I'm inscrutable, but it was probably something else.

But actually, I'm relieved. I'd hope the spooks, cops, or any surreptitious adversaries would have the same problem.

Guess I may need some anti-paranoia meds, but the privacy-threatening implications of technology like this have me looking over my shoulder.


Re: Pretty accurate!
  • 3/10/2014 12:45:00 PM

Best if you'd be able to capture the voice surreptitiously, though. With a 12 year old at home, I lately find myself too often, "Watch your attitude." And his reply is typically along the lines of, "What? I didn't say anything!" Now, if I could capture is intonation on the sly, I'd have evidence of his sulleness!

Re: Pretty accurate!
  • 3/10/2014 9:15:07 AM

I would be interested in the results. I don't think kids know how they come across- much of the time.

Re: Pretty accurate!
  • 3/7/2014 5:04:19 PM

I'm definitely having my kids try it!

Re: Pretty accurate!
  • 3/7/2014 4:57:37 PM

Are you talking about selection bias, Beth? Because I tried to keep my voice as neutral as possible, and it picked up on my lack of sleep, my overwhelming regrets, and my dozens of unrequited crushes easily. 

I think it's done a pretty good job categorizing you!

Regardless, the best way to test the app is by having everybody try it.

Re: Pretty accurate!
  • 3/7/2014 4:25:45 PM

Huh. I think I like this app. It casts me as: "Assertive and tough. High self-confidence." And that I have: "Strong ambition, drive to succeed, joy."

I wonder what it would say about me when analyzing my "not talking to a computer app assessing me by voice" voice. This is fun, but maybe not so telling in this sort of application?

Pretty accurate!
  • 3/7/2014 12:37:24 PM

I'd encourage everyone to give this a try and see how it reflects your emotions and mood. I tried it on a gregarious 3-year-old and got "friendliness, openness, affinity." Spot-on!

<<   <   Page 4 / 4