Facial Analytics: What Are You Smiling at?


Some of us have our feelings written all over our faces. Others may pride themselves on being inscrutable. However, when a computer is analyzing our features frame by frame, it can glean insight from even the slightest quirk.

At last week's Sentiment Analysis Symposium in New York, Jacob Whitehill, a research scientist with Emotient demonstrated the company's emotion recognition products. He showed how they isolate the faces in a video stream and track their expressions, from joyful to angry to sad. "It has many commercial applications," he said.

Emotient provides an API that enables "real-time emotional analysis," and offers highly accurate readings of positive, negative, and neutral emotions based on "cognitive science, machine learning, and computer vision," Whitehill said. Mining large datasets of facial expressions, it can find patterns and sometimes even predict the way people will react to given stimuli. In addition to wide smiles and angry nostril flares, the software detects "microexpressions" like flashes of disgust or contempt.

Emotient analyzes 28 facial expressions to determine the subject's emotional range.
Emotient analyzes 28 facial expressions to determine the subject's emotional range.

According to Emotient's website, the API measures 28 facial action units, including eyebrow raises, nose wrinkles, lip curls, and jaw drops.

The obvious use case is for focus groups, with the software noting positive and negative reactions far more quickly and comprehensively than human observers. In research for consumer packaged goods, Whitehill said, facial analysis was a more accurate predictor of "proclivity to buy" than self-reporting by the subjects. It wasn't so much that certain package designs evoked huge smiles. "Lack of negative reaction was a strong predictor."

In audience response to ads, Emotient finds interesting differences and similarities among male and female viewers. This bears out in an analysis of the Volkswagen "Wings" commercial that aired during this year's Super Bowl. The ad, in which German engineers sprout wings when a car reaches 100,000 miles, drew smiles from women during the elevator scene and smirks from males during the bathroom scene.

Emotient superimposes an emotion waveform over the video stimulus. Whitehill said this "enables a fine-grained temporal analysis that would be much harder to do with human annotators."

Affectiva's Affdex (shown) and Emotient layer emotional responses across a video timeline to pinpoint which moments draw which reactions.
Affectiva's Affdex (shown) and Emotient layer emotional responses across a video timeline to pinpoint which moments draw which reactions.

A competing facial recognition provider, Affectiva, provides an interactive demo of this technology. It watches you through your webcam as you view a series of ads. The output seems similar to Emotient's.

Agitation and engagement
Whitehill said the consumer research firm Innerscope used Emotient software during the Super Bowl to observe a focus group divided among Denver Broncos and Seattle Seahawks fans. "The Denver fans grew more agitated during the game" -- not surprising to any human in the audience, but still noteworthy because the software detected the emotional shift.

In another example, Emotient was used to observe students in a classroom setting. "The engagement level of students during class had a very high correlation with test results." (Our own Ariella Brown wrote about this phenomenon in December.)

Stop staring at me
The emotion recognition industry is clearly in an upswing. Just last week, Emotient announced a $6 million round of funding and debuted a Google Glass app for measuring the sentiment of people within the wearer's field of vision. The first target for this app is the retail industry. "Salespeople who wear Glass can use it to measure how customers respond during their interactions and then get feedback that can help tailor their responses," Ingrid Lundgren wrote for TechCrunch. This application is "particularly aimed at training for future situations, but also for real-time feedback."

I don't know about you, but I was OK with the idea of being observed in tightly defined and controlled situations. I was even amused by the accuracy of emotion detection algorithms -- until I read about this Glass project. As valuable as this kind of technology may be for training sales folks (or, for another example, helping individuals with social communication deficits), it's a little too creepy to think that anyone with a Glass headset may be applying analytics to my face.

What do you think? Share a smile or scowl -- verbally -- in our comments.

— Michael Steinhart, Circle me on Google+ Follow me on TwitterVisit my LinkedIn pageFriend me on Facebook, Executive Editor, AllAnalytics.com

Related posts:

Michael Steinhart, Contributing Editor

Michael Steinhart has been covering IT and business computing for 15 years, tracking the rising popularity of virtualization, unified fabric, high-performance computing, and cloud infrastructures. He is editor of The Enterprise Cloud Site, which won the Least Imaginative Site Name award in 2012, and he managed TheITPro.com, a community of IT professionals taking their first steps into cloud computing. From 2006 to 2012, Steinhart worked as an executive editor at Ziff Davis Enterprise, writing and managing research reports, whitepapers, case studies, magazine features, e-newsletters, blog posts, online videos, and podcasts. He also moderated and presented in dozens of webinars and virtual tradeshows. He got his start in IT journalism at CMP Media back in 1998, then moved to PC Magazine, managing the popular Solutions section and then covering business technology and consumer software. He holds a Bachelor of Arts degree in communications/journalism from Ramapo College of New Jersey.

Biosensors, Robots Replacing Human Guards

In prisons and other security settings, electronic devices are predicting and detecting anomalous behavior with high accuracy and reducing the need for human personnel.

Questioning the Ethics of Learning Analytics

Schools around the country are weighing the benefits and ethical questions of learning analytics.


A good idea
  • 3/10/2014 7:05:00 PM
NO RATINGS

Lack of negative reaction was a strong predictor.

Too bad this wasn't available back in 1958. Maybe it could have saved us from the Edsel debacle.

Re: A good idea
  • 3/10/2014 8:34:56 PM
NO RATINGS

That, and many other marketing disasters, I'm sure. But what about a company that follows similar practices (like maintaining a shroud of secrecy around new releases) and still makes money hand over fist? I'm talking about Apple, naturally. What does it do differently?

Re: A good idea
  • 3/11/2014 1:02:23 AM
NO RATINGS

Apple may be secretive, but they don't have a head-in-the-sand approach to marketing.

One of the challenges in technology fields is the rapid rates of change. A product that is state of the art today will be ho-hum in a short time. This forces all technology companies to listen to the scientists and engineers. Too often this gets out of hand and the engineers control everything. We've all seen ads for electronics or other high tech products that were just a list of gobblygook features. These ads were written in techno speak by engineers.

Apple's success is using technology to make products that your 3-year old and your grandmother can use. Products with features that regular people can understand.

Re: A good idea
  • 3/11/2014 8:23:22 AM
NO RATINGS

It's funny - I have nothing but problems trying to use iProducts. It's probably based on prejudice more than anything else. I wonder how that would register on my face...

Re: A good idea
  • 3/11/2014 9:44:10 AM
NO RATINGS

I agree. I wonder what would register with this product? Maybe Apple could use ideas like this for the design of future products. I suspect Steve Jobs was able to do the same thing inately.

Re: A good idea
  • 3/11/2014 10:39:42 AM
NO RATINGS

Liking a product is not the same as liking a commercial.

Facial analytics is perfect for analyzing the reaction to a Super Bowl commercial.  Reactions can change second-by-second and you can see this happen as people watch the commercial.  If you analyze enough people you can even do a demographic analysis to make sure there aren't some negatives somewhere that you've missed.

But commercials have your full attention and only last seconds or a couple of minutes at most. Products are a much longer term relationship. Many times when you're interacting with a product, the expression on your face will be driven by something other than the product.

Re: A good idea
  • 3/11/2014 2:23:03 PM
NO RATINGS

It's an amazing feat to achieve the degree of accuracy they claim. Definitely applicable to marketing efforts for starters. But I just can't see how they can effectively correlate the perceived expression to a specific item or point of emphasis in a scene. If you have two or more individuals looking at a scene at the simultaneously, they don't necessarily observe it identically. Their point of attraction may focus on different aspects of the scene. The measurements may be accurate but to what is it related specifically? Another question pertains to cultural consideration in interpretation of facial expressions. Was that accounted for?

Re: A good idea
  • 3/12/2014 11:41:37 AM
NO RATINGS

That's probably why 'lack of negative reaction' was a good predictor of purchasing, PC. If you don't immediately dislike something, and then some other variable shifts in favor of it, you'd probably think, "Why not?"

Smiling while grimacing
  • 3/11/2014 1:51:34 PM
NO RATINGS

I have always found this type of analytics fascinating, especially when you compare a person's facial expression and their actual behavior. It allows the determination of performance behavior versus true feelings. In my early research days we used observation research to watch shoppers and then analyze their answers to questions the disparity was incredible. Much of it was attributed to social performance ques. This may help companies determine some true feeling about products and services rather than socially acceptable answers.

goog glass app
  • 3/11/2014 2:03:20 PM
NO RATINGS

this will revolutionize speed dating

Re: goog glass app
  • 3/11/2014 2:09:15 PM
NO RATINGS

Maybe, but there is more to it than just looks.

Re: goog glass app
  • 3/12/2014 10:45:07 AM
NO RATINGS

Having just read Born to Be Good, by Dachter Keltner, professor of psychology at the University of California, Berkely, I was convinced that the facial characteristics for emotion had indeed been worked out, and were proved universal around the world which would indeed prove useful to study and predict behavior.

But, then I read a news release where another study found that facial movements were not universal. So who to believe? Then, what if the computer coiuld analyze correctly, wouldn't every marketer use the same programs and come up with the same results leading to a zero sum game among competitors?

Re: goog glass app
  • 3/12/2014 11:34:17 AM
NO RATINGS

Eventually, that would probably happen, kq4ym, but for the next 3-5 years, at least, it'll be a competitive differentiator. And the services themselves will compete on accuracy and on comprehensiveness (face + voice + text, vs. just one piece of the puzzle). 

As for regional or hemispheric differences in expression, I'm sure the algorithms will evolve to the point where they can pinpoint exactly where a person is from, based on intonation and facial features. We do that without computers all the time. Couldn't you easily differentiate Bostonian, Chicago, and New York accents? What about Australian, British, and South African?

Re: goog glass app
  • 3/11/2014 2:55:59 PM
NO RATINGS

Right! Emotient needs to turn this into an app like Word Lens, so all daters have to do is hold up their phone and get an instant read on what the other person's face is really telling them. 

Re: goog glass app
  • 3/12/2014 11:27:28 AM
NO RATINGS

That's a great scenario, dummy-variable. I don't know if you've read Gary Shteyngart's Super Sad True Love Story, but he envisions a future where all these analytics happen in real time, projected holographically in front of each user. From the New York Times description:

Everyone carries around a device called an äppärät, which can live-stream its owner's thoughts and conversations, and broadcast their "hotness" quotient to others. People are obsessed with their health — Lenny works as a Life Lovers Outreach Coordinator (Grade G) for a firm that specializes in life extension — and shopping is the favorite pastime of anyone with money.

Like minds
  • 3/11/2014 2:52:25 PM
NO RATINGS

Michael, the expression on my face when I read about the Google Glass deal was probably quite similar to the expression on your face when you read about it. I was liking this company and the facial analytics well enough but did a figurative about-face when I came to learn that this app might one day be used to watch me as a I shop. Suddenly I find the idea not so appealing. Like you say, it's OK when this technology is put to use in controlled environments... but outside of that makes me far less comfortable with it.

 

Re: Like minds
  • 3/12/2014 11:36:32 AM
NO RATINGS

I agree 100%, Beth. I wonder, though, how it will play out in retail settings. Will it be a training tool used with actors, with consumers who've opted in, or just put out into the unsuspecting world? Also, if I start frowning, will Glass instruct the sales rep to give me an instant 10% discount?

Re: Like minds
  • 3/12/2014 12:28:54 PM
NO RATINGS

I don't know Michael, but if the latter is the case, I'll have my scowl at the ready. (But, honestly, I hope we don't see Glass use widespread for many years to come. I've never seen anybody wearing Glass glasses in real life, but I'm sure it'll be disconcerting when I do.)

Re: Like minds
  • 3/12/2014 12:40:00 PM
NO RATINGS

Here in New York, I see at least one or two people per week wearing Google Glass. And on an unrelated subject, I thought about it overnight and had a chance to re-watch just now -- all the "German engineers" in the Volkswagen commercial are men. What's up with that?

Re: Like minds
  • 3/12/2014 12:47:08 PM
NO RATINGS

Michael, that is interesting re: Glass-wearing New Yorkers. I would have anticipated seeing a lot in San Francisco/Google HQ but not necessarily NYC... yet. That was silly of me. I don't get out to the streets of downtown Chicago during the week, working as I do from home, so I'm not sure what I'd see among the urban professionals here. Wearing Google Glasses seems rather pretentious to me and we're so much more down to earth here than there, so I'm guessing not. ;-)

As for the Volkswagen commercial... right. What is up with that?!

 

Re: Like minds
  • 3/12/2014 11:30:31 PM
NO RATINGS

I haven't actually seen anyone wearing google glass in this part of the world yet. Before reading this article I wouldn't have given much thought to it but now that I know the possibility of being observed it makes me uncomfortable. I will watch out for google glass wearers.

Re: Like minds
  • 3/14/2014 9:18:45 AM
NO RATINGS

..

Beth writes


But, honestly, I hope we don't see Glass use widespread for many years to come. I've never seen anybody wearing Glass glasses in real life, but I'm sure it'll be disconcerting when I do.


 

You can see a real-life example, plus a report of the consequences, in my recent posting Google Glass sparks brawl in SF bar.

This is one of a number of recent indications of growing public discomfort and even animosity toward the privacy intrusion represented by Glass (and probably other techno devices that push this envelope).

 

Re: Like minds
  • 3/14/2014 1:53:19 PM
NO RATINGS

After watching the video clip, I have a few questions. 1). Why does this woman feel the need to wear Google Glass everywhere she goes, including to bars? The piece notes that she's a social media strategist. I wonder if it's something she's doing in line with her job, whether she just likes the experience, or whether she just likes to call attention to herself.

2). What are the privacy implications for the individuals this woman captured via Glass and then shared with the news media? 

3) Is the media outlet responsible for obtaining permission from the individuals in the videos taken by Glass and shared by the Glass user? This is not like knowing a news camera is in the bar rolling and deliberately getting into the shot.

Lastly, I'd say that Google needs to do a better job of explaining Glass to the general public before these devices become generally available and in widespread use. The last thing it should want on its hands is violence resulting from Glass use.

Re: Like minds
  • 3/14/2014 3:01:57 PM
NO RATINGS

I agree. But at the same time, isn't it the same as bystanders shooting video with their handhelds? That sort of footage has become de rigeur for news outlets.

I guess when you see someone wearing Google Glass, treat them as if they were holding up their cameras or phones.

Glass surveillance & recording implications
  • 3/16/2014 5:04:42 PM
NO RATINGS

..

Michael writes (re: someone recording a scene with Google Glass):


... isn't it the same as bystanders shooting video with their handhelds? That sort of footage has become de rigeur for news outlets.

I guess when you see someone wearing Google Glass, treat them as if they were holding up their cameras or phones.


 

There's a somewhat more surreptitious quality to wearing Glass, and that may be partially what bothers people to the point of anger — an intrusion of the sneaky surveillance that much of the public is now aware of.

Also, there's a big difference between recording, say, a street scene (like an accident or something), and the scene inside a bar or other establishment.

I think the ease of use and low profile of Glass raise a lot of new implications that our society has not encountered before. However, I expect even more surreptitious forms of electronic devices capable of recording audio and video to start becoming more accessible to the general public. After all, tiny "hidden cameras" have been in use by the news media for years.

 

Re: Like minds
  • 3/16/2014 5:09:25 PM
NO RATINGS

..

Beth asks


...Why does this woman feel the need to wear Google Glass everywhere she goes, including to bars? The piece notes that she's a social media strategist. I wonder if it's something she's doing in line with her job, whether she just likes the experience, or whether she just likes to call attention to herself.


 

Or possibly she's engaging in what I perceive as the increasing use of electronic devices to embrace a disengagement and alientation from the here-and-now social environment. Glass sorta takes you to the next level in this process...

 

Re: Like minds
  • 3/17/2014 5:31:52 PM
NO RATINGS

Well, that's interesting -- I'm here, but I'm too cool to really be here and I've got far more interesting things to via Google Glass than participate with you people. The New Elitism is born. 

INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +