Why One-Question Surveys Make for Bad Analytics

If I had to describe Netflix's telephone customer service in one word, that word would be "streamlined."

When you call Netflix customer service, you get no prompting menus, and you are placed on hold for only a few minutes at most. You can even skip to the head of the queue if you use an easily accessed service code from the Netflix Website.

Additionally, Netflix refuses to accept too many calls if the lines get too full. If you happen to call when a major service issue is causing many customers to call in, you will hear -- just before being disconnected -- a curt recording telling you to call back another time. As terrible as this sounds, it may help keep Netflix's customer service representatives from getting jaded by protecting them from potential abuse from customers angry about being on hold for 45 minutes.

However, nowhere is Netflix's streamlined approach to customer service more apparent than at the end of the phone call. After you wrap up your call and the customer service representative wishes you well, you are immediately directed to a customer satisfaction survey.

The survey is one question long:

"We would like to hear about your experience with Netflix. If you are satisfied, press 3. If you are dissatisfied, press 4."

The problem with this question is that it involves major guesswork on the customer's part. Consequently, the response may say more about the respondent than about Netflix -- and therefore may be of little analytical value.

For one thing, it is unclear whether it is asking about satisfaction with the company in general or with the particular customer service phone call. I have always assumed the latter, given the context, but I could be wrong.

For another example, once or twice in the past, I have been assured something would be handled by the exceedingly friendly Netflix customer service representative, pressed 3 on the survey, and found later that the issue was handled incorrectly.

Now I have a dilemma. Do I call Netflix back to complain? If the representative handling that call takes care of it for me, do I tell the survey that I'm satisfied (truthful but doing nothing to mitigate the prior "satisfied" rating) or not satisfied (untruthful but potentially more accurate in the aggregate)? The additional problem with the latter approach is that I don't want the person actually helping me to be punished.

On a related note, what if the employees try their best, but I still have the problem? Recently, I called Netflix about some streaming issues I've been having. Ultimately, the representative could not assist me. I honestly have no way of knowing whether the representative was truly not in a position to assist me (in which case, I should vote satisfied, because he tried his best and was very courteous), or if there was something more he could have done, and he either didn't know what it was or just didn't bother (in which case, I should vote dissatisfied). Torn, I hung up on the survey without answering.

Netflix isn't the only company that has these one-question surveys. As many companies overwhelm customers with a laundry list of increasingly complex feedback questions, others -- in the interest of increasing customer participation -- are taking the opposite approach: brutal simplicity.

However, this kind of all-or-nothing survey, leaving no room between two extremes (satisfied and dissatisfied), has serious weaknesses. Sure, with only a straightforward yes-or-no question, the survey is quick and simple, but "quick and simple" is not synonymous with "efficient" -- or, for that matter, "accurate."

Moreover, its usefulness is limited. Netflix is getting little to no insight as to why the customer is satisfied or dissatisfied. Additionally, unless Netflix keeps track of customers who can't get through during a busy time, the customer survey fails to take into account perhaps the most dissatisfied customers of all.

Accuracy, not simplicity, is the touchstone of analytics. Organizations must therefore take a more balanced approach to surveys. Yes, it may make the analytics more difficult, but thinking like an actual customer yields far more accurate -- and actionable -- results.

Joe Stanganelli, Attorney & Marketer

Joe Stanganelli is founder and principal of Beacon Hill Law, a Boston-based general practice law firm.  His expertise on legal topics has been sought for several major publications, including U.S. News and World Report and Personal Real Estate Investor Magazine. 

Joe is also a communications consultant.  He has been working with social media for many years -- even in the days of local BBSs (one of which he served as Co-System Operator for), well before the term "social media" was invented.

From 2003 to 2005, Joe ran Grandpa George Productions, a New England entertainment and media production company. He has also worked as a professional actor, director, and producer.  Additionally, Joe is a produced playwright.

When he's not lawyering, marketing, or social-media-ing, Joe writes scripts, songs, and stories.

He also finds time to lose at bridge a couple of times a month.

Follow Joe on Twitter: @JoeStanganelli

Also, check out his blog .

Clean Your Data with Visualization and Algorithmic Tests

Speakers at Bio-IT World explore techniques for biotech researchers and others working with big data to identify the accurate data in their data files.

Data Sharing: A Matter of Life and Death

Cooperation among medical researchers -- done right -- very simply can mean lives saved, but the research community needs education on how to execute on that collaboration.

Re: Survey frustration
  • 8/26/2011 10:08:10 AM

Coming from a customer support background the majority of people don't take surverys, especially if they're long.

Netflix gets that. This question is more likely to get answered and can get you a quick temperature of your client satisfaction. This doesn't mean that its going to be completely accurate, but you get a feel.

Thinking to deep
  • 8/26/2011 10:05:47 AM

Honestly I think you might be thinking into this way too deep. I'm a huge fan of netflix and its my opinion that they've done a terrific job of simplifying almost everything (yes, even their interface ;)), for the majority of a users needs.

What they're looking for at at the end of the call was one thing. Are you satisfied? Not are you satisfied with the service, the support, the qulity of the picture (which they do send out frequently), but the overall satisfaction of the client.

With any good metric this would need to be chopped down a little more, and maybe they're planning on it, but it seems like they're goal was to find overall satisfaction for the time being.




Re: Survey frustration
  • 8/25/2011 12:24:06 PM
1 saves

Joe, in one of your other posts you use the phrase "customer service theater." I love it. That really does say what's going on at a lot of companies in a nutshell. Lots of staging around presenting the picture of being truly interested in serving customers and catering to their needs.

Re: Survey frustration
  • 8/25/2011 10:42:29 AM

There is nothing *inherently* wrong with a short survey, so long as it is constructed to provide enough options to the taker to be meaningful and enough meaningful data to the organization to be useful and actionable.

One question, with a yes or no answer, and nothing in between, seems hardly the way to go when it comes to measuring customer satisfaction.

Although, I freely admit that, as a lawyer, I am predisposed towards seeing things in shades of gray.

Re: Survey frustration
  • 8/24/2011 11:42:56 PM


Still the trend is very short surveys. I took one recently that was just two questions long.

Re: Netflix perspective
  • 8/24/2011 9:33:30 PM

Hah!  Good luck, since they famously don't even respond to customer feedback.

Do you have a contact at Netflix?  Generally, from as far as I can tell, the person who usually talks to the press is Steve Swasey, VP of Corporate Communications.

Re: Need Marketing researcher
  • 8/24/2011 9:31:19 PM

Hi, Maryam.

Ditto for the UI change to Instant Play, which also caused a large backlash (though quite small compared to the reaction to the price hike).  Curiously, they insisted that that change was based upon market research.

As Scott Adams reminds us, however, research -- or claims thereof -- cannot always be trusted.

Re: Survey frustration
  • 8/24/2011 9:27:59 PM

Hi, tinym.

Perhaps the goal is to measure how customer service impacts change in satisfaction levels in the company, assuming every customer who calls customer service is dissatisfied with Netflix?

But that would be stupid, because not every customer service call is from someone who is dissatisfied.  Half the time when I call, I'm not dissatisfied per se; I just need something taken care of that only customer service can take care of.

Re: Survey frustration
  • 8/24/2011 9:25:11 PM

Hi, Shawn.

Except I'm not convinced Netflix actually acts on that information; they certainly don't act on customer feedback, if the UI change and price hike debacles are any indication.

It's not customer service; it's customer service theatre.

Re: Survey frustration
  • 8/24/2011 9:23:13 PM

Hi, Gil.

"Little data"... I like it!

You'd think that, if they're engaging in the kind of sophisticated analytics you are talking about, they'd have at least a mildly better better approach to a customer satisfaction survey.

I think they could get almost as many people to participate in, say, a three-quesiton survey, that was crafted much more precisely and yielded high-quality information.

<<   <   Page 2 / 4   >   >>