Facial Analytics: What Are You Smiling at?


Some of us have our feelings written all over our faces. Others may pride themselves on being inscrutable. However, when a computer is analyzing our features frame by frame, it can glean insight from even the slightest quirk.

At last week's Sentiment Analysis Symposium in New York, Jacob Whitehill, a research scientist with Emotient demonstrated the company's emotion recognition products. He showed how they isolate the faces in a video stream and track their expressions, from joyful to angry to sad. "It has many commercial applications," he said.

Emotient provides an API that enables "real-time emotional analysis," and offers highly accurate readings of positive, negative, and neutral emotions based on "cognitive science, machine learning, and computer vision," Whitehill said. Mining large datasets of facial expressions, it can find patterns and sometimes even predict the way people will react to given stimuli. In addition to wide smiles and angry nostril flares, the software detects "microexpressions" like flashes of disgust or contempt.

Emotient analyzes 28 facial expressions to determine the subject's emotional range.
Emotient analyzes 28 facial expressions to determine the subject's emotional range.

According to Emotient's website, the API measures 28 facial action units, including eyebrow raises, nose wrinkles, lip curls, and jaw drops.

The obvious use case is for focus groups, with the software noting positive and negative reactions far more quickly and comprehensively than human observers. In research for consumer packaged goods, Whitehill said, facial analysis was a more accurate predictor of "proclivity to buy" than self-reporting by the subjects. It wasn't so much that certain package designs evoked huge smiles. "Lack of negative reaction was a strong predictor."

In audience response to ads, Emotient finds interesting differences and similarities among male and female viewers. This bears out in an analysis of the Volkswagen "Wings" commercial that aired during this year's Super Bowl. The ad, in which German engineers sprout wings when a car reaches 100,000 miles, drew smiles from women during the elevator scene and smirks from males during the bathroom scene.

Emotient superimposes an emotion waveform over the video stimulus. Whitehill said this "enables a fine-grained temporal analysis that would be much harder to do with human annotators."

Affectiva's Affdex (shown) and Emotient layer emotional responses across a video timeline to pinpoint which moments draw which reactions.
Affectiva's Affdex (shown) and Emotient layer emotional responses across a video timeline to pinpoint which moments draw which reactions.

A competing facial recognition provider, Affectiva, provides an interactive demo of this technology. It watches you through your webcam as you view a series of ads. The output seems similar to Emotient's.

Agitation and engagement
Whitehill said the consumer research firm Innerscope used Emotient software during the Super Bowl to observe a focus group divided among Denver Broncos and Seattle Seahawks fans. "The Denver fans grew more agitated during the game" -- not surprising to any human in the audience, but still noteworthy because the software detected the emotional shift.

In another example, Emotient was used to observe students in a classroom setting. "The engagement level of students during class had a very high correlation with test results." (Our own Ariella Brown wrote about this phenomenon in December.)

Stop staring at me
The emotion recognition industry is clearly in an upswing. Just last week, Emotient announced a $6 million round of funding and debuted a Google Glass app for measuring the sentiment of people within the wearer's field of vision. The first target for this app is the retail industry. "Salespeople who wear Glass can use it to measure how customers respond during their interactions and then get feedback that can help tailor their responses," Ingrid Lundgren wrote for TechCrunch. This application is "particularly aimed at training for future situations, but also for real-time feedback."

I don't know about you, but I was OK with the idea of being observed in tightly defined and controlled situations. I was even amused by the accuracy of emotion detection algorithms -- until I read about this Glass project. As valuable as this kind of technology may be for training sales folks (or, for another example, helping individuals with social communication deficits), it's a little too creepy to think that anyone with a Glass headset may be applying analytics to my face.

What do you think? Share a smile or scowl -- verbally -- in our comments.

— Michael Steinhart, Circle me on Google+ Follow me on TwitterVisit my LinkedIn pageFriend me on Facebook, Executive Editor, AllAnalytics.com

Related posts:

Michael Steinhart, Contributing Editor

Michael Steinhart has been covering IT and business computing for 15 years, tracking the rising popularity of virtualization, unified fabric, high-performance computing, and cloud infrastructures. He is editor of The Enterprise Cloud Site, which won the Least Imaginative Site Name award in 2012, and he managed TheITPro.com, a community of IT professionals taking their first steps into cloud computing. From 2006 to 2012, Steinhart worked as an executive editor at Ziff Davis Enterprise, writing and managing research reports, whitepapers, case studies, magazine features, e-newsletters, blog posts, online videos, and podcasts. He also moderated and presented in dozens of webinars and virtual tradeshows. He got his start in IT journalism at CMP Media back in 1998, then moved to PC Magazine, managing the popular Solutions section and then covering business technology and consumer software. He holds a Bachelor of Arts degree in communications/journalism from Ramapo College of New Jersey.

Biosensors, Robots Replacing Human Guards

In prisons and other security settings, electronic devices are predicting and detecting anomalous behavior with high accuracy and reducing the need for human personnel.

Questioning the Ethics of Learning Analytics

Schools around the country are weighing the benefits and ethical questions of learning analytics.


Re: Like minds
  • 3/17/2014 5:31:52 PM
NO RATINGS

Well, that's interesting -- I'm here, but I'm too cool to really be here and I've got far more interesting things to via Google Glass than participate with you people. The New Elitism is born. 

Re: Like minds
  • 3/16/2014 5:09:25 PM
NO RATINGS

..

Beth asks


...Why does this woman feel the need to wear Google Glass everywhere she goes, including to bars? The piece notes that she's a social media strategist. I wonder if it's something she's doing in line with her job, whether she just likes the experience, or whether she just likes to call attention to herself.


 

Or possibly she's engaging in what I perceive as the increasing use of electronic devices to embrace a disengagement and alientation from the here-and-now social environment. Glass sorta takes you to the next level in this process...

 

Glass surveillance & recording implications
  • 3/16/2014 5:04:42 PM
NO RATINGS

..

Michael writes (re: someone recording a scene with Google Glass):


... isn't it the same as bystanders shooting video with their handhelds? That sort of footage has become de rigeur for news outlets.

I guess when you see someone wearing Google Glass, treat them as if they were holding up their cameras or phones.


 

There's a somewhat more surreptitious quality to wearing Glass, and that may be partially what bothers people to the point of anger — an intrusion of the sneaky surveillance that much of the public is now aware of.

Also, there's a big difference between recording, say, a street scene (like an accident or something), and the scene inside a bar or other establishment.

I think the ease of use and low profile of Glass raise a lot of new implications that our society has not encountered before. However, I expect even more surreptitious forms of electronic devices capable of recording audio and video to start becoming more accessible to the general public. After all, tiny "hidden cameras" have been in use by the news media for years.

 

Re: Like minds
  • 3/14/2014 3:01:57 PM
NO RATINGS

I agree. But at the same time, isn't it the same as bystanders shooting video with their handhelds? That sort of footage has become de rigeur for news outlets.

I guess when you see someone wearing Google Glass, treat them as if they were holding up their cameras or phones.

Re: Like minds
  • 3/14/2014 1:53:19 PM
NO RATINGS

After watching the video clip, I have a few questions. 1). Why does this woman feel the need to wear Google Glass everywhere she goes, including to bars? The piece notes that she's a social media strategist. I wonder if it's something she's doing in line with her job, whether she just likes the experience, or whether she just likes to call attention to herself.

2). What are the privacy implications for the individuals this woman captured via Glass and then shared with the news media? 

3) Is the media outlet responsible for obtaining permission from the individuals in the videos taken by Glass and shared by the Glass user? This is not like knowing a news camera is in the bar rolling and deliberately getting into the shot.

Lastly, I'd say that Google needs to do a better job of explaining Glass to the general public before these devices become generally available and in widespread use. The last thing it should want on its hands is violence resulting from Glass use.

Re: Like minds
  • 3/14/2014 9:18:45 AM
NO RATINGS

..

Beth writes


But, honestly, I hope we don't see Glass use widespread for many years to come. I've never seen anybody wearing Glass glasses in real life, but I'm sure it'll be disconcerting when I do.


 

You can see a real-life example, plus a report of the consequences, in my recent posting Google Glass sparks brawl in SF bar.

This is one of a number of recent indications of growing public discomfort and even animosity toward the privacy intrusion represented by Glass (and probably other techno devices that push this envelope).

 

Re: Like minds
  • 3/12/2014 11:30:31 PM
NO RATINGS

I haven't actually seen anyone wearing google glass in this part of the world yet. Before reading this article I wouldn't have given much thought to it but now that I know the possibility of being observed it makes me uncomfortable. I will watch out for google glass wearers.

Re: Like minds
  • 3/12/2014 12:47:08 PM
NO RATINGS

Michael, that is interesting re: Glass-wearing New Yorkers. I would have anticipated seeing a lot in San Francisco/Google HQ but not necessarily NYC... yet. That was silly of me. I don't get out to the streets of downtown Chicago during the week, working as I do from home, so I'm not sure what I'd see among the urban professionals here. Wearing Google Glasses seems rather pretentious to me and we're so much more down to earth here than there, so I'm guessing not. ;-)

As for the Volkswagen commercial... right. What is up with that?!

 

Re: Like minds
  • 3/12/2014 12:40:00 PM
NO RATINGS

Here in New York, I see at least one or two people per week wearing Google Glass. And on an unrelated subject, I thought about it overnight and had a chance to re-watch just now -- all the "German engineers" in the Volkswagen commercial are men. What's up with that?

Re: Like minds
  • 3/12/2014 12:28:54 PM
NO RATINGS

I don't know Michael, but if the latter is the case, I'll have my scowl at the ready. (But, honestly, I hope we don't see Glass use widespread for many years to come. I've never seen anybody wearing Glass glasses in real life, but I'm sure it'll be disconcerting when I do.)

Page 1 / 3   >   >>
INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +