Voice Analytics: You Sound Emotional


Imagine you ask Siri to find a restaurant nearby. She replies, "You sound kind of down, so I'd recommend a warm and quiet spot a few blocks from here."

How did Siri know how you were feeling? Welcome to the world of intonation analytics.

Humans read emotion in verbal communication all the time: I don't like your tone! -- An edge crept into his voice. -- She spoke soothingly. But the science of sentiment analysis is still relatively new and focused primarily on text.

As explained by data scientist Brian Kolo on the Opera Solutions blog, algorithms have a hard enough time differentiating "that's hot" (as in, good) from "that's hot!" (as in, you'll burn yourself). It seems a stretch to suggest they can understand how we feel based on our intonations.

Or does it? At yesterday's Sentiment Analysis Symposium in New York, executives from Beyond Verbal demonstrated software that seems to pick up on these cues, too.

"It's not what you say, but how you say it," explained Dan Emodi, vice president of marketing at Beyond Verbal. "Vocal intonations convey your mood, attitude, even personality."

The research began 18 years ago, when a physicist and neuropsychologist began exploring the way that babies understand language even before they understand words, Emodi said. The team discovered that intonation transcends language and culture.

"Patterns of happiness, sadness, aggression, introversion, extroversion, apathy, and action-orientation, they're all the same across languages," he said.

On Beyond Verbal's site, vocal analytics are applied to the speech of President Barack Obama, Steve Jobs, and Princess Diana. The company also analyzed Edward Snowden's June interview with The Guardian, finding that his voice betrayed some egocentricity and pride in orchestrating his NSA document leak.

You can listen to me, but do you hear me?
Like most sentiment analysis projects, Beyond Verbal started with a set of samples -- 16,000 audio clips, in this case -- and trained its algorithms using crowdsourced human evaluators. Temper and anger were easiest to detect accurately, Emodi said. "Call centers have plenty of aggressive tones." Over the years, the training sample grew to 32,000 clips.

The potential applications are limited only by imagination, he added. Contact centers, market research firms, dating sites, search services, and many other businesses could benefit from this kind of technology.

I asked about law enforcement, and Emodi said the software has caught the attention of Homeland Security officials. Another audience member asked about healthcare applications, where the software could detect confusion in patients suffering from early stages of dementia, for example.

"We've been contacted by healthcare companies," Emodi said. "When we started this, we never even thought about healthcare."

Beyond Verbal's analytics happen in the cloud, so individuals can try it on the website or via the Moodies iOS app.

I gave it a shot this morning, and as you can see in the screenshot, it seems to have me pegged. "It helps us get a better understanding of our own selves," Emodi said.

What do you think, members? Do you see intonation analytics changing the way we interact with machines? What other applications might benefit from intonation analytics? Make your voice heard (but watch your tone) below.

— Michael Steinhart, Circle me on Google+ Follow me on TwitterVisit my LinkedIn pageFriend me on Facebook, Executive Editor, AllAnalytics.com

Related posts:

Michael Steinhart, Contributing Editor

Michael Steinhart has been covering IT and business computing for 15 years, tracking the rising popularity of virtualization, unified fabric, high-performance computing, and cloud infrastructures. He is editor of The Enterprise Cloud Site, which won the Least Imaginative Site Name award in 2012, and he managed TheITPro.com, a community of IT professionals taking their first steps into cloud computing. From 2006 to 2012, Steinhart worked as an executive editor at Ziff Davis Enterprise, writing and managing research reports, whitepapers, case studies, magazine features, e-newsletters, blog posts, online videos, and podcasts. He also moderated and presented in dozens of webinars and virtual tradeshows. He got his start in IT journalism at CMP Media back in 1998, then moved to PC Magazine, managing the popular Solutions section and then covering business technology and consumer software. He holds a Bachelor of Arts degree in communications/journalism from Ramapo College of New Jersey.

Biosensors, Robots Replacing Human Guards

In prisons and other security settings, electronic devices are predicting and detecting anomalous behavior with high accuracy and reducing the need for human personnel.

Questioning the Ethics of Learning Analytics

Schools around the country are weighing the benefits and ethical questions of learning analytics.


Pretty accurate!
  • 3/7/2014 12:37:24 PM
NO RATINGS

I'd encourage everyone to give this a try and see how it reflects your emotions and mood. I tried it on a gregarious 3-year-old and got "friendliness, openness, affinity." Spot-on!

Re: Pretty accurate!
  • 3/7/2014 4:25:45 PM
NO RATINGS

Huh. I think I like this app. It casts me as: "Assertive and tough. High self-confidence." And that I have: "Strong ambition, drive to succeed, joy."

I wonder what it would say about me when analyzing my "not talking to a computer app assessing me by voice" voice. This is fun, but maybe not so telling in this sort of application?

Re: Pretty accurate!
  • 3/7/2014 4:57:37 PM
NO RATINGS

Are you talking about selection bias, Beth? Because I tried to keep my voice as neutral as possible, and it picked up on my lack of sleep, my overwhelming regrets, and my dozens of unrequited crushes easily. 

I think it's done a pretty good job categorizing you!

Regardless, the best way to test the app is by having everybody try it.

Re: Pretty accurate!
  • 3/7/2014 5:04:19 PM
NO RATINGS

I'm definitely having my kids try it!

Re: Pretty accurate!
  • 3/10/2014 9:15:07 AM
NO RATINGS

I would be interested in the results. I don't think kids know how they come across- much of the time.

Re: Pretty accurate!
  • 3/10/2014 12:45:00 PM
NO RATINGS

Best if you'd be able to capture the voice surreptitiously, though. With a 12 year old at home, I lately find myself too often, "Watch your attitude." And his reply is typically along the lines of, "What? I didn't say anything!" Now, if I could capture is intonation on the sly, I'd have evidence of his sulleness!

Re: Pretty accurate!
  • 3/10/2014 2:54:14 PM
NO RATINGS

I thought my house is the only one where that scenario plays out, Beth! If you have an iPhone, you can put the Moodies app on and grab samples when he least expects it.

Re: Pretty accurate!
  • 3/11/2014 9:24:22 AM
NO RATINGS

Now if only Moodies also captured facial expressions -- because of course those hold plenty of the type of evidence I'd like to present to him as well.

Re: Pretty accurate!
  • 3/11/2014 3:16:27 PM
NO RATINGS

And here's the 'corroboration' for Beyond Verbal's assessment of Ed Snowden:

Snowden was asked... whether he felt the controversial NSA leaks he executed were worth it in retrospect. "I would do it again," Snowden said, noting that he believes the revelations have benefited the public.

 

Re: Pretty accurate!
  • 3/11/2014 3:45:43 PM
NO RATINGS

Given that his decision to do what he did now has him living in exile, what else might we expect him to say? He'd look far less important to himself if he said his actions didn't have any public benefit, right?

Re: Pretty accurate!
  • 3/11/2014 4:08:06 PM
NO RATINGS

He might have the honesty to say that he'd rather not be living in exile, all things considered. But the kind of "I did it, and I'd do it again" defiance comes from pride and ego. And again, this isn't a surprise to us. It's just interesting that an algorithm can detect it.

Re: Pretty accurate!
  • 3/11/2014 4:33:48 PM
NO RATINGS

Yeah, I hear what you're saying. But honestly I'm more impressed with the algorithm's ability to capture the emotional nuances of regular folk, who aren't necessarily so impassioned about what it is they might be speaking of. Setting it against Snowden's voice seems like a gimme to me.

Re: Pretty accurate!
  • 3/11/2014 6:19:51 PM
NO RATINGS

I do find verbal intonation intriquing, but I wonder if analytics can really be applied to this ?

Re: Pretty accurate!
  • 3/12/2014 8:59:17 AM
NO RATINGS

Beyond Verbal certainly thinks it can be!

Re: Pretty accurate!
  • 3/13/2014 6:09:36 PM
NO RATINGS

I agree, and I also think it's an interesting exercise regardless of how accurate it is. I tried it on an upbeat, cheerful friend, and it detected stress and resentment as primary emotions. That sort of thing can either be an uncomfortable revelation, or an off-base assumption. We were in a noisy environment, so it's possible the algorithm had a hard time detecting intonation.

Re: Pretty accurate!
  • 3/13/2014 6:14:51 PM
NO RATINGS

Just to follow up -- the friend repeated the test and got 'stress' and 'vengeance' again. Then, on a third try, it came up with self-esteem, leadership, and charisma. All of which are true. So maybe it's not as fine-tuned as it seems?

Re: Pretty accurate!
  • 3/13/2014 8:43:09 PM
NO RATINGS

Thanks for the information Michael, I would tend to think there are just too many factors which could affect the results, I mean people fake "happiness" for instance - all the time.  : )

Re: Pretty accurate!
  • 3/14/2014 2:08:45 PM
NO RATINGS

That's a rather substantial either/or, Michael. So much so that doesn't it more or less negate any value of voice analytics? 

Re: Pretty accurate!
  • 3/22/2014 11:04:21 AM
NO RATINGS

As we're separating words from intonation, this technology might be great for travelers, who'll be able to gauge the emotions of people who speak another language. At the very least, it might help a tourist avoid getting ripped off.

Re: Pretty accurate!
  • 3/10/2014 1:43:37 PM
NO RATINGS

..

Michael writes


I'd encourage everyone to give this a try and see how it reflects your emotions and mood


 

Well, darn. The software could not analyze my voice. I used the same microphone that provides over 99% accuracy with Dragon, so I dunno what the problem was. I'd like to think I'm inscrutable, but it was probably something else.

But actually, I'm relieved. I'd hope the spooks, cops, or any surreptitious adversaries would have the same problem.

Guess I may need some anti-paranoia meds, but the privacy-threatening implications of technology like this have me looking over my shoulder.

 

Re: Pretty accurate!
  • 3/10/2014 2:59:21 PM
NO RATINGS

That's a very valid point, Lyndon -- and I'm sure its accuracy would be challenged the instant it was used to prove anything of legal import. (Kind of like the Psychic Hotline disclaimer: This service is for entertainment only...)

But in the (albeit limited) few cases that I've used it, it was spot-on, even detecting a slight undertone of sadness in one subject's otherwise cheery vocalizations.

I wonder whether this kind of technology could help overworked mental health professionals, or serve to power "artificial therapy" apps. 

Re: Pretty accurate!
  • 3/10/2014 4:03:03 PM
NO RATINGS

..

Michael writes



I wonder whether this kind of technology could help overworked mental health professionals, or serve to power "artificial therapy" apps. 


 

I'm sure there are important beneficial uses for this kind of mood sensitivity/detection software, and it's obviously a clever advancement in analytics. As with so many new developments in technology, it tends to come Janus-faced. We all need to be aware of both the positive and negative implications.

 

Re: Pretty accurate!
  • 3/10/2014 5:23:37 PM
NO RATINGS

Sounds like every technology development! I just got finished writing a blog about emotion detection via facial expression analysis. The company that fields it just announced a Google Glass app, so anyone watching you via Glass may also be recording your face and analyzing it for emotional cues. I can't think of any potential for misuse there!

Re: Pretty accurate!
  • 3/10/2014 9:06:25 PM
NO RATINGS

..

Michael writes


The company that fields it just announced a Google Glass app, so anyone watching you via Glass may also be recording your face and analyzing it for emotional cues. I can't think of any potential for misuse there!


 

I guess my apprehension is stimulated in part by the misuse of existing technologies such as the venerable lie detector (used mainly by law enforcement) and more recent software over the past 10-15 years that purports to detect whether you're lying, honest, dependable, etc. (used mainly by businesses).

I haven't done deep research, but I have seen a lot of evidence reported that indicates reliability fluctuates. But cops and execs seem to be basing decisions regarding the fates of human beings  on what amounts to somewhat flaky technology.

I would think that Glass could also use a voice analysis app implementing the same basic kind of attitude analysis we've been discussing.

 

Re: Pretty accurate!
  • 3/11/2014 9:46:26 AM
NO RATINGS

I suppose it's just a matter of time before the facial and audio analytics are combined into one comprehensive 'detector.' I just wonder whether it'll have broader uptake in law enforecement or business.

Re: Pretty accurate!
  • 3/11/2014 6:26:44 PM
NO RATINGS

@Lyndon_Henry   Good point reqarding Lie Detectors - which are loosely based on this notion of measuring responses. I don't think anyone would consider them "fool-proof" and intonation analytics probably faces many of the same challenges.

Re: Pretty accurate!
  • 3/10/2014 5:52:47 PM
NO RATINGS

In theory it's sounds plausible. But I wonder if we "speak" to our mobile devices with the same emotional content as we speak to real people? If there's another "language" going on, there will have to be a separate "device" algorithm to compensate for the differences?

Re: Pretty accurate!
  • 3/10/2014 6:19:43 PM
NO RATINGS

Beth brought up a similar concern, kq4ym, and it's possible that the algorithm accounts for that statistically or by filtering the tension out -- there are probably a number of ways to go about it. Also, when it's being used by a contact center, for instance, the subject doesn't know they're being analyzed.

Re: Pretty accurate!
  • 3/10/2014 6:58:40 PM
NO RATINGS

Next time I'm on the phone with a call center, I may just run this app on myself. Partly to see how I might be coming across.

As these types of tools become better and more common, it certainly won't hurt to be more aware of your own tone.

Re: Pretty accurate!
  • 3/10/2014 5:42:09 PM
NO RATINGS

But in the (albeit limited) few cases that I've used it, it was spot-on

I tried it on a teenager - NOT surreptitiously. Some of the results seemed to agree with my personal assessment -

* Primary - controlled by urges

* Secondary - Argumenative, accusing

It gave us four different results in four consecutive samples. I know teenage moods change often, but every 30 seconds?

Re: Pretty accurate!
  • 3/10/2014 5:51:37 PM
NO RATINGS

That's interesting, PC, because I tried it consecutively on a younger subject, and the results changed subtly, not widely. And according to the folks at Emotient (a facial expression analytics firm), emotions can swing from happy to sad to angry to disgusted in a matter of seconds. I wrote about them in a blog that you should see in a couple of minutes.

Re: Pretty accurate!
  • 3/10/2014 6:14:05 PM
NO RATINGS

Maybe the results do have validity. Looking back, I notice that stress shows up somewhere in three of the four results.

That's not what I'd expect for a teenager at the start of Spring Break, but I know she felt a little like she should 'perform' for me and for the app. It is a little stressful to be recorded and analyzed by a smartphone app, especially one that promises to report your innermost feelings.

Re: Pretty accurate!
  • 3/10/2014 6:22:10 PM
NO RATINGS

I'm not saying this is a window into everyone's soul, per se, but it's certainly interesting. It could make for a good ice breaker or conversation piece at parties, if nothing else.

My daughter (10) was hesitant to try it, but in the end, it revealed some interesting traits that are unique to her. Nothing revelatory, but interesting that it could pick up on them. I'm curious to see how this is phased into 'normal' life over the coming months and years.

Emotions and meanings
  • 3/11/2014 1:57:01 PM
NO RATINGS

 Michael I think this is a key area.There has been much discussion about text analytics and this is often what is missing; the emotion behind what is written. Text itself is open to interpretation because words mean so many things, even a simple word like" really" can mean multiple things and have multiple connotations. Cracking the code on the emotions will provide the real analytics data.

Re: Emotions and meanings
  • 3/11/2014 3:14:58 PM
NO RATINGS

But aside from sales engagements / marketing, and law enforcement, why else would we need to know someone's emotional state? What kinds of interactions will this help facilitate?

Re: Emotions and meanings
  • 3/13/2014 3:06:51 PM
NO RATINGS

Michael while those are key areas, the emotional state can determine the meaning of a word or phrase so it could impact the overall interpretation of what is said at any time. Whenever analysis is performed it is worth considering.

Would do wonders for a marriage (or not)
  • 3/22/2014 10:57:36 AM
NO RATINGS

With this technology, millions of husbands around the world will finally be able to understand their wives. No longer shall we have insensitive men who can't hear what women are not saying. Simply slap this on a wearable and it'll sell like hotcakes, though remember to advertise in wedding fairs, just to be sure.

But it's a double-edged sword. It could totally backfire on the insincere, especially if wives also have this technology.

"I love you." [LYING]

"I'm sorry." [EGO]

"I feel sick; I can't take out the garbage." [LAZY]

INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +