Voice Analytics: You Sound Emotional

Imagine you ask Siri to find a restaurant nearby. She replies, "You sound kind of down, so I'd recommend a warm and quiet spot a few blocks from here."

How did Siri know how you were feeling? Welcome to the world of intonation analytics.

Humans read emotion in verbal communication all the time: I don't like your tone! -- An edge crept into his voice. -- She spoke soothingly. But the science of sentiment analysis is still relatively new and focused primarily on text.

As explained by data scientist Brian Kolo on the Opera Solutions blog, algorithms have a hard enough time differentiating "that's hot" (as in, good) from "that's hot!" (as in, you'll burn yourself). It seems a stretch to suggest they can understand how we feel based on our intonations.

Or does it? At yesterday's Sentiment Analysis Symposium in New York, executives from Beyond Verbal demonstrated software that seems to pick up on these cues, too.

"It's not what you say, but how you say it," explained Dan Emodi, vice president of marketing at Beyond Verbal. "Vocal intonations convey your mood, attitude, even personality."

The research began 18 years ago, when a physicist and neuropsychologist began exploring the way that babies understand language even before they understand words, Emodi said. The team discovered that intonation transcends language and culture.

"Patterns of happiness, sadness, aggression, introversion, extroversion, apathy, and action-orientation, they're all the same across languages," he said.

On Beyond Verbal's site, vocal analytics are applied to the speech of President Barack Obama, Steve Jobs, and Princess Diana. The company also analyzed Edward Snowden's June interview with The Guardian, finding that his voice betrayed some egocentricity and pride in orchestrating his NSA document leak.

You can listen to me, but do you hear me?
Like most sentiment analysis projects, Beyond Verbal started with a set of samples -- 16,000 audio clips, in this case -- and trained its algorithms using crowdsourced human evaluators. Temper and anger were easiest to detect accurately, Emodi said. "Call centers have plenty of aggressive tones." Over the years, the training sample grew to 32,000 clips.

The potential applications are limited only by imagination, he added. Contact centers, market research firms, dating sites, search services, and many other businesses could benefit from this kind of technology.

I asked about law enforcement, and Emodi said the software has caught the attention of Homeland Security officials. Another audience member asked about healthcare applications, where the software could detect confusion in patients suffering from early stages of dementia, for example.

"We've been contacted by healthcare companies," Emodi said. "When we started this, we never even thought about healthcare."

Beyond Verbal's analytics happen in the cloud, so individuals can try it on the website or via the Moodies iOS app.

I gave it a shot this morning, and as you can see in the screenshot, it seems to have me pegged. "It helps us get a better understanding of our own selves," Emodi said.

What do you think, members? Do you see intonation analytics changing the way we interact with machines? What other applications might benefit from intonation analytics? Make your voice heard (but watch your tone) below.

— Michael Steinhart, Circle me on Google+ Follow me on TwitterVisit my LinkedIn pageFriend me on Facebook, Executive Editor, AllAnalytics.com

Related posts:

Michael Steinhart, Contributing Editor

Michael Steinhart has been covering IT and business computing for 15 years, tracking the rising popularity of virtualization, unified fabric, high-performance computing, and cloud infrastructures. He is editor of The Enterprise Cloud Site, which won the Least Imaginative Site Name award in 2012, and he managed TheITPro.com, a community of IT professionals taking their first steps into cloud computing. From 2006 to 2012, Steinhart worked as an executive editor at Ziff Davis Enterprise, writing and managing research reports, whitepapers, case studies, magazine features, e-newsletters, blog posts, online videos, and podcasts. He also moderated and presented in dozens of webinars and virtual tradeshows. He got his start in IT journalism at CMP Media back in 1998, then moved to PC Magazine, managing the popular Solutions section and then covering business technology and consumer software. He holds a Bachelor of Arts degree in communications/journalism from Ramapo College of New Jersey.

Biosensors, Robots Replacing Human Guards

In prisons and other security settings, electronic devices are predicting and detecting anomalous behavior with high accuracy and reducing the need for human personnel.

Questioning the Ethics of Learning Analytics

Schools around the country are weighing the benefits and ethical questions of learning analytics.

Re: Pretty accurate!
  • 3/11/2014 4:33:48 PM

Yeah, I hear what you're saying. But honestly I'm more impressed with the algorithm's ability to capture the emotional nuances of regular folk, who aren't necessarily so impassioned about what it is they might be speaking of. Setting it against Snowden's voice seems like a gimme to me.

Re: Pretty accurate!
  • 3/11/2014 4:08:06 PM

He might have the honesty to say that he'd rather not be living in exile, all things considered. But the kind of "I did it, and I'd do it again" defiance comes from pride and ego. And again, this isn't a surprise to us. It's just interesting that an algorithm can detect it.

Re: Pretty accurate!
  • 3/11/2014 3:45:43 PM

Given that his decision to do what he did now has him living in exile, what else might we expect him to say? He'd look far less important to himself if he said his actions didn't have any public benefit, right?

Re: Pretty accurate!
  • 3/11/2014 3:16:27 PM

And here's the 'corroboration' for Beyond Verbal's assessment of Ed Snowden:

Snowden was asked... whether he felt the controversial NSA leaks he executed were worth it in retrospect. "I would do it again," Snowden said, noting that he believes the revelations have benefited the public.


Re: Emotions and meanings
  • 3/11/2014 3:14:58 PM

But aside from sales engagements / marketing, and law enforcement, why else would we need to know someone's emotional state? What kinds of interactions will this help facilitate?

Emotions and meanings
  • 3/11/2014 1:57:01 PM

 Michael I think this is a key area.There has been much discussion about text analytics and this is often what is missing; the emotion behind what is written. Text itself is open to interpretation because words mean so many things, even a simple word like" really" can mean multiple things and have multiple connotations. Cracking the code on the emotions will provide the real analytics data.

Re: Pretty accurate!
  • 3/11/2014 9:46:26 AM

I suppose it's just a matter of time before the facial and audio analytics are combined into one comprehensive 'detector.' I just wonder whether it'll have broader uptake in law enforecement or business.

Re: Pretty accurate!
  • 3/11/2014 9:24:22 AM

Now if only Moodies also captured facial expressions -- because of course those hold plenty of the type of evidence I'd like to present to him as well.

Re: Pretty accurate!
  • 3/10/2014 9:06:25 PM


Michael writes

The company that fields it just announced a Google Glass app, so anyone watching you via Glass may also be recording your face and analyzing it for emotional cues. I can't think of any potential for misuse there!


I guess my apprehension is stimulated in part by the misuse of existing technologies such as the venerable lie detector (used mainly by law enforcement) and more recent software over the past 10-15 years that purports to detect whether you're lying, honest, dependable, etc. (used mainly by businesses).

I haven't done deep research, but I have seen a lot of evidence reported that indicates reliability fluctuates. But cops and execs seem to be basing decisions regarding the fates of human beings  on what amounts to somewhat flaky technology.

I would think that Glass could also use a voice analysis app implementing the same basic kind of attitude analysis we've been discussing.


Re: Pretty accurate!
  • 3/10/2014 6:58:40 PM

Next time I'm on the phone with a call center, I may just run this app on myself. Partly to see how I might be coming across.

As these types of tools become better and more common, it certainly won't hurt to be more aware of your own tone.

<<   <   Page 2 / 4   >   >>