Facial Analytics: What Are You Smiling at?

Some of us have our feelings written all over our faces. Others may pride themselves on being inscrutable. However, when a computer is analyzing our features frame by frame, it can glean insight from even the slightest quirk.

At last week's Sentiment Analysis Symposium in New York, Jacob Whitehill, a research scientist with Emotient demonstrated the company's emotion recognition products. He showed how they isolate the faces in a video stream and track their expressions, from joyful to angry to sad. "It has many commercial applications," he said.

Emotient provides an API that enables "real-time emotional analysis," and offers highly accurate readings of positive, negative, and neutral emotions based on "cognitive science, machine learning, and computer vision," Whitehill said. Mining large datasets of facial expressions, it can find patterns and sometimes even predict the way people will react to given stimuli. In addition to wide smiles and angry nostril flares, the software detects "microexpressions" like flashes of disgust or contempt.

Emotient analyzes 28 facial expressions to determine the subject's emotional range.
Emotient analyzes 28 facial expressions to determine the subject's emotional range.

According to Emotient's website, the API measures 28 facial action units, including eyebrow raises, nose wrinkles, lip curls, and jaw drops.

The obvious use case is for focus groups, with the software noting positive and negative reactions far more quickly and comprehensively than human observers. In research for consumer packaged goods, Whitehill said, facial analysis was a more accurate predictor of "proclivity to buy" than self-reporting by the subjects. It wasn't so much that certain package designs evoked huge smiles. "Lack of negative reaction was a strong predictor."

In audience response to ads, Emotient finds interesting differences and similarities among male and female viewers. This bears out in an analysis of the Volkswagen "Wings" commercial that aired during this year's Super Bowl. The ad, in which German engineers sprout wings when a car reaches 100,000 miles, drew smiles from women during the elevator scene and smirks from males during the bathroom scene.

Emotient superimposes an emotion waveform over the video stimulus. Whitehill said this "enables a fine-grained temporal analysis that would be much harder to do with human annotators."

Affectiva's Affdex (shown) and Emotient layer emotional responses across a video timeline to pinpoint which moments draw which reactions.
Affectiva's Affdex (shown) and Emotient layer emotional responses across a video timeline to pinpoint which moments draw which reactions.

A competing facial recognition provider, Affectiva, provides an interactive demo of this technology. It watches you through your webcam as you view a series of ads. The output seems similar to Emotient's.

Agitation and engagement
Whitehill said the consumer research firm Innerscope used Emotient software during the Super Bowl to observe a focus group divided among Denver Broncos and Seattle Seahawks fans. "The Denver fans grew more agitated during the game" -- not surprising to any human in the audience, but still noteworthy because the software detected the emotional shift.

In another example, Emotient was used to observe students in a classroom setting. "The engagement level of students during class had a very high correlation with test results." (Our own Ariella Brown wrote about this phenomenon in December.)

Stop staring at me
The emotion recognition industry is clearly in an upswing. Just last week, Emotient announced a $6 million round of funding and debuted a Google Glass app for measuring the sentiment of people within the wearer's field of vision. The first target for this app is the retail industry. "Salespeople who wear Glass can use it to measure how customers respond during their interactions and then get feedback that can help tailor their responses," Ingrid Lundgren wrote for TechCrunch. This application is "particularly aimed at training for future situations, but also for real-time feedback."

I don't know about you, but I was OK with the idea of being observed in tightly defined and controlled situations. I was even amused by the accuracy of emotion detection algorithms -- until I read about this Glass project. As valuable as this kind of technology may be for training sales folks (or, for another example, helping individuals with social communication deficits), it's a little too creepy to think that anyone with a Glass headset may be applying analytics to my face.

What do you think? Share a smile or scowl -- verbally -- in our comments.

— Michael Steinhart, Circle me on Google+ Follow me on TwitterVisit my LinkedIn pageFriend me on Facebook, Executive Editor, AllAnalytics.com

Related posts:

Michael Steinhart, Contributing Editor

Michael Steinhart has been covering IT and business computing for 15 years, tracking the rising popularity of virtualization, unified fabric, high-performance computing, and cloud infrastructures. He is editor of The Enterprise Cloud Site, which won the Least Imaginative Site Name award in 2012, and he managed TheITPro.com, a community of IT professionals taking their first steps into cloud computing. From 2006 to 2012, Steinhart worked as an executive editor at Ziff Davis Enterprise, writing and managing research reports, whitepapers, case studies, magazine features, e-newsletters, blog posts, online videos, and podcasts. He also moderated and presented in dozens of webinars and virtual tradeshows. He got his start in IT journalism at CMP Media back in 1998, then moved to PC Magazine, managing the popular Solutions section and then covering business technology and consumer software. He holds a Bachelor of Arts degree in communications/journalism from Ramapo College of New Jersey.

Biosensors, Robots Replacing Human Guards

In prisons and other security settings, electronic devices are predicting and detecting anomalous behavior with high accuracy and reducing the need for human personnel.

Questioning the Ethics of Learning Analytics

Schools around the country are weighing the benefits and ethical questions of learning analytics.

Re: A good idea
  • 3/12/2014 11:41:37 AM

That's probably why 'lack of negative reaction' was a good predictor of purchasing, PC. If you don't immediately dislike something, and then some other variable shifts in favor of it, you'd probably think, "Why not?"

Re: Like minds
  • 3/12/2014 11:36:32 AM

I agree 100%, Beth. I wonder, though, how it will play out in retail settings. Will it be a training tool used with actors, with consumers who've opted in, or just put out into the unsuspecting world? Also, if I start frowning, will Glass instruct the sales rep to give me an instant 10% discount?

Re: goog glass app
  • 3/12/2014 11:34:17 AM

Eventually, that would probably happen, kq4ym, but for the next 3-5 years, at least, it'll be a competitive differentiator. And the services themselves will compete on accuracy and on comprehensiveness (face + voice + text, vs. just one piece of the puzzle). 

As for regional or hemispheric differences in expression, I'm sure the algorithms will evolve to the point where they can pinpoint exactly where a person is from, based on intonation and facial features. We do that without computers all the time. Couldn't you easily differentiate Bostonian, Chicago, and New York accents? What about Australian, British, and South African?

Re: goog glass app
  • 3/12/2014 11:27:28 AM

That's a great scenario, dummy-variable. I don't know if you've read Gary Shteyngart's Super Sad True Love Story, but he envisions a future where all these analytics happen in real time, projected holographically in front of each user. From the New York Times description:

Everyone carries around a device called an äppärät, which can live-stream its owner's thoughts and conversations, and broadcast their "hotness" quotient to others. People are obsessed with their health — Lenny works as a Life Lovers Outreach Coordinator (Grade G) for a firm that specializes in life extension — and shopping is the favorite pastime of anyone with money.

Re: goog glass app
  • 3/12/2014 10:45:07 AM

Having just read Born to Be Good, by Dachter Keltner, professor of psychology at the University of California, Berkely, I was convinced that the facial characteristics for emotion had indeed been worked out, and were proved universal around the world which would indeed prove useful to study and predict behavior.

But, then I read a news release where another study found that facial movements were not universal. So who to believe? Then, what if the computer coiuld analyze correctly, wouldn't every marketer use the same programs and come up with the same results leading to a zero sum game among competitors?

Re: goog glass app
  • 3/11/2014 2:55:59 PM

Right! Emotient needs to turn this into an app like Word Lens, so all daters have to do is hold up their phone and get an instant read on what the other person's face is really telling them. 

Like minds
  • 3/11/2014 2:52:25 PM

Michael, the expression on my face when I read about the Google Glass deal was probably quite similar to the expression on your face when you read about it. I was liking this company and the facial analytics well enough but did a figurative about-face when I came to learn that this app might one day be used to watch me as a I shop. Suddenly I find the idea not so appealing. Like you say, it's OK when this technology is put to use in controlled environments... but outside of that makes me far less comfortable with it.


Re: A good idea
  • 3/11/2014 2:23:03 PM

It's an amazing feat to achieve the degree of accuracy they claim. Definitely applicable to marketing efforts for starters. But I just can't see how they can effectively correlate the perceived expression to a specific item or point of emphasis in a scene. If you have two or more individuals looking at a scene at the simultaneously, they don't necessarily observe it identically. Their point of attraction may focus on different aspects of the scene. The measurements may be accurate but to what is it related specifically? Another question pertains to cultural consideration in interpretation of facial expressions. Was that accounted for?

Re: goog glass app
  • 3/11/2014 2:09:15 PM

Maybe, but there is more to it than just looks.

goog glass app
  • 3/11/2014 2:03:20 PM

this will revolutionize speed dating

<<   <   Page 2 / 3   >   >>