Joe Stanganelli

Why One-Question Surveys Make for Bad Analytics

NO RATINGS
View Comments: Threaded | Newest First | Oldest First
BethSchultz
User Rank
Blogger
Survey frustration
BethSchultz   8/22/2011 10:49:27 AM
NO RATINGS
Joe, I know this frustration well having recently had a similar experience with Comcast. I spent way too much time on a recent Sunday afternoon working with myriad Comcast techs as I tried to update my cable modem/router setup. Each time I had to initiate another call -- which happened multiple times during this process for a variety of reasons, including several instances of misrouting and disconnections, I was asked if I'd mind taking a quick survey at the end of my call. I did once, after talking to the first guy who was nice and helpful and even though he ended up not being able to help me he at least acknowledged that and successfully passed me along to the next tech down the line. She, too, was helpful -- or so I thought and so I rated the overall experience as positive. However, in thinking about the experience after the fact, I kind of think she just wanted me off the phone because she didn't know how to fix my problem. She told me to call back after my new modem had time to download all the appropriate software from the network, but that ended up resulting in about 2 hours and all sorts of misdirection. Ultimately, I had to reinstall my old modem. I should have taken multiple surveys expressing my irritation, but as the day wore on all I wanted was my dang connection reestablished and the heck to providing Comcast with usable feedback!

Joe Stanganelli
User Rank
Blogger
Re: Survey frustration
Joe Stanganelli   8/22/2011 11:27:10 AM
NO RATINGS
Hi, Beth.

Yes, I also have issues with Comcast's surveys.

I once made the mistake of agreeing to take a survey, but then deciding that I didn't want to after hearing the loaded question, and hanging up.  Comcast continued to auto-call me for WEEKS after the fact.  Now I never take their surveys.

Plus, you can't take the survey when a particularly dull-witted Comcast employee abruptly disconnects you -- which happened to me about a week ago.

Jaime
User Rank
Statistician
Re: Survey frustration
Jaime   8/22/2011 1:47:28 PM
I like their idea of the picture quality 1-question survey you randomly get after viewing a movie. But there is also a big problem with the question as well. Can you guess what it is? The email question reads:

Survey: How was the Picture and Audio Quality?

Dear Jaime,

You recently watched Goya's Ghosts. To help us ensure a great experience for all members, would you take a moment to tell us about the picture and audio quality?

The quality was very good

The quality was acceptable

The quality was unacceptable

-------

The problem is that it is trying to rate two different aspects of the movie. What if the video was good but not the audio? This composite question should be separated into two or show me only audio or video and the next person who watches gets the other one. With as many movie watchers, I am sure Nextflix would get a good sample and be able to react when a movie's audio or video is bad. Fortunately, I've always rated their instant viewing positively and never had a problem with video or audio.

Your post reminds me of the 1-question you get from a restaurant waiter. "Was the food good?" I have to say that even when it wasn't, I always say yes. By being more specific, it would entice the customer to be more honest. For example, was the chicken seasoned correctly? Was the milkshake too thick? Restaurants can gather lots of info if they just had the right 1-question selected.

Shawn Hessinger
User Rank
Blogger
Re: Survey frustration
Shawn Hessinger   8/22/2011 2:44:36 PM
NO RATINGS
Jaime,
Agree totally. I think on one hand they've tried to make the process simple and, well, short, but they've overdone it. They are also trying to simplify the data being collected but there are too many ways to interpret the answers for the process to be helpful to the company, which should be the whole point of the survey in the first place.

BethSchultz
User Rank
Blogger
Re: Survey frustration
BethSchultz   8/22/2011 3:41:57 PM
NO RATINGS
The Netflix satisfaction survey, and so many others out there, smacks to me of somebody THINKING they ought to be collecting data but not really analyzing the why of it or, more importantly, putting themselves in the customer's seat and judging the survey question from that perspective. (Or maybe they did, but determined there was no way to get to the info they really wanted in a simple, straightforward fashion and this is simply their "throw in the towel" approach!)

GilPress
User Rank
Blogger
Re: Survey frustration
GilPress   8/22/2011 4:16:15 PM
NO RATINGS
1 saves
 

Indeed, getting more customers to participate in satisfaction surveys may yield more data but does not guarantee better insights. Maybe we should start a movement called "little data"... :-)

As for Netflix, it is probably more interested in analyzing (or at least watching closely) the data they get from their customers anyway, without the need to beg for participation. This leads to reduced calls to the streamlined call center and to proactive customer service.  Earlier this year, I was playing with their video streaming service, starting and stopping after a few minutes a number of movies. The next day (possibly the next hour),  I got an email from Netflix apologizing for the difficulties I had in using their video service and offering me a few dollars off my next monthly bill.  I'm sure Netflixers are much more excited about performing these kinds of customer delighters than designing and analyzing traditional satisfaction surveys. Given that Netflix Instant accounts for 20% of peak U.S. bandwidth use, their analysts and Data Scientists probably prefer to spend their time analyzing Internet usage patterns, capacity constraints, and performance, everywhere in the U.S., knowledge that must represent an enviable competitive and market differentiation and drives their proactive customer service.

 

Shawn Hessinger
User Rank
Blogger
Re: Survey frustration
Shawn Hessinger   8/22/2011 9:10:59 PM
NO RATINGS
1 saves
GilPress,

You bring up an interesting point, namely, is the customer survey much of a survey at all or is it a kind of customer service, one more example of how Netflix welcomes feedback even though they limit the calls they will receive? I remember when we started using Netflix in our home and how difficult it became to justify a trip to the video rental store...when there still was one in business locally! Why? Because for the cost and the number of titles offered not only with streaming video but also with home delivery, there wasn't much competition. I don't mean this to sound like a Netflix ad. The point I am making is that it's almost like complaining about free services from Google. Due to the adoption rate of these technologies, how helpful could such surveys really be and aren't there likely better places to apply analytics in terms of mapping growth and examining the next new opportunity.

Joe Stanganelli
User Rank
Blogger
Re: Survey frustration
Joe Stanganelli   8/24/2011 9:25:11 PM
NO RATINGS
Hi, Shawn.

Except I'm not convinced Netflix actually acts on that information; they certainly don't act on customer feedback, if the UI change and price hike debacles are any indication.

It's not customer service; it's customer service theatre.

Joe Stanganelli
User Rank
Blogger
Re: Survey frustration
Joe Stanganelli   8/24/2011 9:23:13 PM
NO RATINGS
Hi, Gil.

"Little data"... I like it!

You'd think that, if they're engaging in the kind of sophisticated analytics you are talking about, they'd have at least a mildly better better approach to a customer satisfaction survey.

I think they could get almost as many people to participate in, say, a three-quesiton survey, that was crafted much more precisely and yielded high-quality information.

Shawn Hessinger
User Rank
Blogger
Re: Survey frustration
Shawn Hessinger   8/24/2011 11:42:56 PM
NO RATINGS
Joe,

Still the trend is very short surveys. I took one recently that was just two questions long.

Joe Stanganelli
User Rank
Blogger
Re: Survey frustration
Joe Stanganelli   8/25/2011 10:42:29 AM
NO RATINGS
There is nothing *inherently* wrong with a short survey, so long as it is constructed to provide enough options to the taker to be meaningful and enough meaningful data to the organization to be useful and actionable.

One question, with a yes or no answer, and nothing in between, seems hardly the way to go when it comes to measuring customer satisfaction.

Although, I freely admit that, as a lawyer, I am predisposed towards seeing things in shades of gray.

scucci
User Rank
Data Doctor
Re: Survey frustration
scucci   8/26/2011 10:08:10 AM
NO RATINGS
Coming from a customer support background the majority of people don't take surverys, especially if they're long.

Netflix gets that. This question is more likely to get answered and can get you a quick temperature of your client satisfaction. This doesn't mean that its going to be completely accurate, but you get a feel.

Cordell
User Rank
Blogger
Re: Survey frustration
Cordell   8/26/2011 2:41:13 PM
NO RATINGS
An anecdote about phone trees. I was reading an article complaining about the elaborate phone trees your bank requires you to navigate when you call that included this gem - "It's almost like they'd rather you didn't call at all"  Um, that's exactly what it is.  For a bank calls=cost.  If the CSR can't cross-sell something when they get you on the line then they'd rather not take the call at all when a machine can do it.  Shortsighted? Maybe but seriously, when was the last time you switched banks?  What some banks may interpret as loyalty is nothing more than being averse to switching!

Netflix might do the same, mistaking loyalty for a lack of alternatives.

Michelle
User Rank
Data Doctor
Re: Survey frustration
Michelle   8/26/2011 4:47:47 PM
NO RATINGS
That's an excellent point about banks and loyalty.

louisw900
User Rank
Blogger
Re: Survey frustration
louisw900   9/24/2011 6:27:12 PM
NO RATINGS
@Cordell  Excellent point regarding Banks, an industry fraught with "agnostic arrogance" regarding customer service.  And you are so right, banks are missing the point, people are not changing banks as often due to this disregard for their time and business, as they have little choice.  So what is analytics to make of this ?  Not much unless you measure the right variables and come from a place of geniune concern for the customer.

SethBreedlove
User Rank
Data Doctor
Re: Survey frustration
SethBreedlove   10/4/2011 2:16:13 PM
NO RATINGS
I'm wondering if it is truly a one question survey. If a person presses 4 for being dissatisfied are they emailed a follow up survey to get more information? 

The reason I wonder is that Netflix is consistently rated one of the highest web  in customer satisifaction.  This year they are only one point behind Amazon. 

Joe Stanganelli
User Rank
Blogger
Re: Survey frustration
Joe Stanganelli   8/24/2011 9:20:53 PM
NO RATINGS
Agreed, Beth.  All too often (and not just with analytics), businesses feel like they should be using a particular solution because everyone else is using it, but they don't understand it, and are too cheap to implement it properly, so they just go halfway with it.

It's the same phenomenon as when a company hires an unqualified college kid as an underpaid intern to "do social media" for them.

BethSchultz
User Rank
Blogger
Re: Survey frustration
BethSchultz   8/25/2011 12:24:06 PM
NO RATINGS
1 saves
Joe, in one of your other posts you use the phrase "customer service theater." I love it. That really does say what's going on at a lot of companies in a nutshell. Lots of staging around presenting the picture of being truly interested in serving customers and catering to their needs.

Michelle
User Rank
Data Doctor
Re: Survey frustration
Michelle   8/22/2011 11:47:17 PM
NO RATINGS
I'm a Netflix subscriber too. I always long for a second question or at least a comment box on thier one-question surveys. I don't want to spend a long time completing a survey but I want to give accurate feedback. I have wondered what data they were most interested in with these seemingly one-off questions.

Joe Stanganelli
User Rank
Blogger
Re: Survey frustration
Joe Stanganelli   8/24/2011 9:27:59 PM
NO RATINGS
Hi, tinym.

Perhaps the goal is to measure how customer service impacts change in satisfaction levels in the company, assuming every customer who calls customer service is dissatisfied with Netflix?

But that would be stupid, because not every customer service call is from someone who is dissatisfied.  Half the time when I call, I'm not dissatisfied per se; I just need something taken care of that only customer service can take care of.

louisw900
User Rank
Blogger
Wasted Effort
louisw900   8/23/2011 12:23:28 AM
NO RATINGS

@Joe   As I read your blog ( which is excellent) I could not really believe what I was reading!  One question surveys ?  A sort of true and false scenario ?  You have got to be kidding me, yes I know you are not.  

Anyhow, this is discouraging at best, it looks like Netflix and companies like them are trying to make Analytics as easy as possible and are as you say missing the boat entirely.

What can they possibly hope to garner from such an exercise ?  I cannot for the life of me understand why a company would foolishly waste the effort.


impactnow
User Rank
Blogger
Need Marketing researcher
impactnow   8/23/2011 2:21:27 AM
NO RATINGS

Truly sad, they might have used the same survey to determine the price increase they recently implemented that has caused mass user backlash. It is sad that so many companies are cutting back on research as a result of the economy, if they had a marketing researcher on staff or on contract such a mistake would not have been made. Some of the “do it yourself” research tools on the web have made companies think they can cut out researchers and the software will do it for you—this is a critical mistake that they will see in their erroneous results.

Joe Stanganelli
User Rank
Blogger
Re: Need Marketing researcher
Joe Stanganelli   8/24/2011 9:31:19 PM
NO RATINGS
Hi, Maryam.

Ditto for the UI change to Instant Play, which also caused a large backlash (though quite small compared to the reaction to the price hike).  Curiously, they insisted that that change was based upon market research.

As Scott Adams reminds us, however, research -- or claims thereof -- cannot always be trusted.

scucci
User Rank
Data Doctor
Re: Need Marketing researcher
scucci   8/26/2011 10:12:38 AM
NO RATINGS
First let me state that I don't work for Netflix (LOL) and that I might be the only person on the this and other boards that like the UI.

What we have to remember is that we're used to having the ability to customize our systems when we don't like an option.

What Netflix is doing with their UI is making it as EASY to use as possible for the AVERAGE user. When I say average user I'm thinking about my son and grandmother. They're not concerned that it takes slightly longer or doesn't have the options that we want, because they want to streamline everything to make it easy to use.

This gives better customer satisfaction since they know how to use it without reading directions or calling customer support.

I think market research is right on the money if you're an average user.

BethSchultz
User Rank
Blogger
Netflix perspective
BethSchultz   8/23/2011 11:55:28 AM
NO RATINGS
So with all this pondering on Netflix's survey/analytics strategy, I'm hoping to get word directly from the horse's mouth, so to speak. Looking to line up an interview ... and hopefully will be able to share more in a bit!

 

louisw900
User Rank
Blogger
Re: Netflix perspective
louisw900   8/23/2011 2:03:00 PM
NO RATINGS
@ Beth    Awesome.  Surely looking forward to hearing from Netflix on the rationale behind this puzzling practice.

Shawn Hessinger
User Rank
Blogger
Re: Netflix perspective
Shawn Hessinger   8/23/2011 10:24:30 PM
NO RATINGS
Hi Beth,

I think much could be learned from such an interview and beyond just the basis for their single question survey. I'm hoping the interview will include other aspects of Netflix analytics which I'm sure they use for everything from marketing to home delivery to information on download usage and more.

BethSchultz
User Rank
Blogger
Re: Netflix perspective
BethSchultz   8/24/2011 7:48:31 AM
NO RATINGS
Well, no response on my interview request yet -- but I will persist!

Joe Stanganelli
User Rank
Blogger
Re: Netflix perspective
Joe Stanganelli   8/24/2011 9:33:30 PM
NO RATINGS
Hah!  Good luck, since they famously don't even respond to customer feedback.

Do you have a contact at Netflix?  Generally, from as far as I can tell, the person who usually talks to the press is Steve Swasey, VP of Corporate Communications.

scucci
User Rank
Data Doctor
Thinking to deep
scucci   8/26/2011 10:05:47 AM
NO RATINGS
Honestly I think you might be thinking into this way too deep. I'm a huge fan of netflix and its my opinion that they've done a terrific job of simplifying almost everything (yes, even their interface ;)), for the majority of a users needs.

What they're looking for at at the end of the call was one thing. Are you satisfied? Not are you satisfied with the service, the support, the qulity of the picture (which they do send out frequently), but the overall satisfaction of the client.

With any good metric this would need to be chopped down a little more, and maybe they're planning on it, but it seems like they're goal was to find overall satisfaction for the time being.

 

 

 

Joe Stanganelli
User Rank
Blogger
Re: Thinking to deep
Joe Stanganelli   8/26/2011 10:18:34 AM
NO RATINGS
But scucci, the knowledge as to whether I am "satisfied" in some vague, generic way is not actionable.  It does not tell you why I feel that way, what I feel that way about, or what you can do to maintain or change that feeling.

It's meaningless, and can only lead to blind guesswork.

At least the e-mail surveys asking about picture and audio quality, while they do have their weaknesses (as an earlier commenter pointed out), tell you *something.*  So to with the e-mail surveys asking when a particular DVD arrived or was shipped.

But "yes or no: are you satisfied" with no in-between or explanation (either of what the question really means or what the answers really mean)?  You may be able to impress a clueless senior executive or board member with a measurement of the numbers on that metric, but it doesn't tell you anything productive.  Strictly from the perspective of gathering and applying analytics, you may as well abandon the question and just look at sales volume and costumer service call volume.

scucci
User Rank
Data Doctor
Re: Thinking to deep
scucci   8/26/2011 11:44:39 AM
NO RATINGS
I haven't taken one of the customer support surveys, so I don't know what if any other options are available after selecting a choice.

It there aren't any, you get high-level overview of what your customers think (as I mentioned not very useful in its own, but gives you the temperature). What they should do is break this down a few more steps afterwards.

This sounds to me like they're getting something started and are taking the approach very slowly and simplistically. Too much change for a user is scary.

If this is all they have planned for the future, than I agree, the data isn't going to be worth anything long term. 

aaphil
User Rank
Blogger
Re: Thinking to deep
aaphil   8/26/2011 1:08:50 PM
NO RATINGS
Joe,

True. It's all about action from a service P.O.V. If you can't act decisively based on feedback then, in essence, you have nothing. SO much like standard marketing with calls to action, surveys must provide the same answers to those who are looking for answers.

Cordell
User Rank
Blogger
Re: Thinking to deep
Cordell   8/26/2011 2:28:03 PM
NO RATINGS
Okay let me turn this scenario on it's head.  I'm speculating of course but what if the survey isn't really designed to evaluate specific customer feedback per se.  Instead it's a tool to evaluate the CSR's. (Beth a good question when/if you get the interview!). This would lead to all kinds of unusual behavior like reps disconnecting before you could respond to a bad interaction or encouraging you to call later to push you off on another rep.

Now let's really get crazy and imagine that someone at Netflix decided that this was useful market data and upon observing that that most customers are "satisfied" (because the data's been skewed) they conclude that they must be less price sensitive - hence the change in pricing structure.  Well you can't say they didn't take action!

Bad decision based on bad data?  I'm positive I'm overthinking now. Surely Netflix realizes that their loyal customer base isn't going anywhere based on more than thier phone surveys.  The lesson here is that not thinking through design of the survey and the process for data gathering can bring unintended consequences.

Zimana
User Rank
Blogger
Cordell's point about customer service = cost
Zimana   9/24/2011 3:52:31 AM
NO RATINGS
When I read about the customer service, I felt a similar vibe to Cordell's point about cost and customer service.  He's right but it's not just banks. Many businesses consider customer service as strictly a cost rather than a chance to change the value of the operation.  That's essential what Zappos proved with its customer service (and with its hiring practice of paying prospective employees to quit).  Now that there's been a succesful model, more consideration of how to improve customer service will come into practice.  The only question left is which companies will continue to refine and succeed.

Information Resources
More Blogs from Joe Stanganelli
Dun&Bradstreet is tapping its huge database of company information to identify suppliers that have potential human trafficking violations that would be of concern to customers concerned about sustainability.
Speakers at Bio-IT World explore techniques for biotech researchers and others working with big data to identify the accurate data in their data files.
Cooperation among medical researchers -- done right -- very simply can mean lives saved, but the research community needs education on how to execute on that collaboration.
Research into the digital marketing campaigns of some of the presidential candidates raises the question of how much voter messaging is appropriate and who should get which types of messages.
Predictive asset maintenance analytics is finding its niche across industries, from automotive to oil and gas.
Radio Show
A2 Conversations
UPCOMING
Jessica Davis
Monetize Your Data: Turning Insights Into Action


3/29/2017   REGISTER   0
ARCHIVE
James M. Connolly
Evolution of the Data Scientist Role


3/23/2017  LISTEN   31
ARCHIVE
Jessica Davis
Data Analysts in Training: Meeting Tomorrow's Demand


3/8/2017  LISTEN   63
ARCHIVE
Jessica Davis
Our Bodies, Our Data: Medical Records For Sale


2/21/2017  LISTEN   63
ARCHIVE
Jessica Davis
Energy Analytics: Using Data to Find Savings


2/14/2017  LISTEN   44
ARCHIVE
Jessica Davis
Sharpen Your Analytics & Data Management Strategy


2/8/2017  LISTEN   74
ARCHIVE
Jessica Davis
Analytics: Make the Most of Data's Potential in 2017


1/19/2017  LISTEN   19
ARCHIVE
Jessica Davis
A2 Radio: Can You Trust Your Data?


12/20/2016  LISTEN   70
ARCHIVE
James M. Connolly
Retail Analytics: See Where Style Meets Statistics


12/6/2016  LISTEN   53
ARCHIVE
James M. Connolly
Why the IoT Matters to Your Business


11/29/2016  LISTEN   45
ARCHIVE
James M. Connolly
Will Data and Humans Become Friends in 2017?


11/22/2016  LISTEN   40
Information Resources
Quick Poll
Quick Poll
About Us  |  Contact Us  |  Help  |  Register  |  Twitter  |  Facebook  |  RSS