Joe Stanganelli

Why One-Question Surveys Make for Bad Analytics

NO RATINGS
View Comments: Oldest First | Newest First | Threaded View
Page 1 / 4   >   >>
BethSchultz
User Rank
Blogger
Survey frustration
BethSchultz   8/22/2011 10:49:27 AM
NO RATINGS
Joe, I know this frustration well having recently had a similar experience with Comcast. I spent way too much time on a recent Sunday afternoon working with myriad Comcast techs as I tried to update my cable modem/router setup. Each time I had to initiate another call -- which happened multiple times during this process for a variety of reasons, including several instances of misrouting and disconnections, I was asked if I'd mind taking a quick survey at the end of my call. I did once, after talking to the first guy who was nice and helpful and even though he ended up not being able to help me he at least acknowledged that and successfully passed me along to the next tech down the line. She, too, was helpful -- or so I thought and so I rated the overall experience as positive. However, in thinking about the experience after the fact, I kind of think she just wanted me off the phone because she didn't know how to fix my problem. She told me to call back after my new modem had time to download all the appropriate software from the network, but that ended up resulting in about 2 hours and all sorts of misdirection. Ultimately, I had to reinstall my old modem. I should have taken multiple surveys expressing my irritation, but as the day wore on all I wanted was my dang connection reestablished and the heck to providing Comcast with usable feedback!

Joe Stanganelli
User Rank
Blogger
Re: Survey frustration
Joe Stanganelli   8/22/2011 11:27:10 AM
NO RATINGS
Hi, Beth.

Yes, I also have issues with Comcast's surveys.

I once made the mistake of agreeing to take a survey, but then deciding that I didn't want to after hearing the loaded question, and hanging up.  Comcast continued to auto-call me for WEEKS after the fact.  Now I never take their surveys.

Plus, you can't take the survey when a particularly dull-witted Comcast employee abruptly disconnects you -- which happened to me about a week ago.

Jaime
User Rank
Statistician
Re: Survey frustration
Jaime   8/22/2011 1:47:28 PM
I like their idea of the picture quality 1-question survey you randomly get after viewing a movie. But there is also a big problem with the question as well. Can you guess what it is? The email question reads:

Survey: How was the Picture and Audio Quality?

Dear Jaime,

You recently watched Goya's Ghosts. To help us ensure a great experience for all members, would you take a moment to tell us about the picture and audio quality?

The quality was very good

The quality was acceptable

The quality was unacceptable

-------

The problem is that it is trying to rate two different aspects of the movie. What if the video was good but not the audio? This composite question should be separated into two or show me only audio or video and the next person who watches gets the other one. With as many movie watchers, I am sure Nextflix would get a good sample and be able to react when a movie's audio or video is bad. Fortunately, I've always rated their instant viewing positively and never had a problem with video or audio.

Your post reminds me of the 1-question you get from a restaurant waiter. "Was the food good?" I have to say that even when it wasn't, I always say yes. By being more specific, it would entice the customer to be more honest. For example, was the chicken seasoned correctly? Was the milkshake too thick? Restaurants can gather lots of info if they just had the right 1-question selected.

Shawn Hessinger
User Rank
Blogger
Re: Survey frustration
Shawn Hessinger   8/22/2011 2:44:36 PM
NO RATINGS
Jaime,
Agree totally. I think on one hand they've tried to make the process simple and, well, short, but they've overdone it. They are also trying to simplify the data being collected but there are too many ways to interpret the answers for the process to be helpful to the company, which should be the whole point of the survey in the first place.

BethSchultz
User Rank
Blogger
Re: Survey frustration
BethSchultz   8/22/2011 3:41:57 PM
NO RATINGS
The Netflix satisfaction survey, and so many others out there, smacks to me of somebody THINKING they ought to be collecting data but not really analyzing the why of it or, more importantly, putting themselves in the customer's seat and judging the survey question from that perspective. (Or maybe they did, but determined there was no way to get to the info they really wanted in a simple, straightforward fashion and this is simply their "throw in the towel" approach!)

GilPress
User Rank
Blogger
Re: Survey frustration
GilPress   8/22/2011 4:16:15 PM
NO RATINGS
1 saves
 

Indeed, getting more customers to participate in satisfaction surveys may yield more data but does not guarantee better insights. Maybe we should start a movement called "little data"... :-)

As for Netflix, it is probably more interested in analyzing (or at least watching closely) the data they get from their customers anyway, without the need to beg for participation. This leads to reduced calls to the streamlined call center and to proactive customer service.  Earlier this year, I was playing with their video streaming service, starting and stopping after a few minutes a number of movies. The next day (possibly the next hour),  I got an email from Netflix apologizing for the difficulties I had in using their video service and offering me a few dollars off my next monthly bill.  I'm sure Netflixers are much more excited about performing these kinds of customer delighters than designing and analyzing traditional satisfaction surveys. Given that Netflix Instant accounts for 20% of peak U.S. bandwidth use, their analysts and Data Scientists probably prefer to spend their time analyzing Internet usage patterns, capacity constraints, and performance, everywhere in the U.S., knowledge that must represent an enviable competitive and market differentiation and drives their proactive customer service.

 

Shawn Hessinger
User Rank
Blogger
Re: Survey frustration
Shawn Hessinger   8/22/2011 9:10:59 PM
NO RATINGS
1 saves
GilPress,

You bring up an interesting point, namely, is the customer survey much of a survey at all or is it a kind of customer service, one more example of how Netflix welcomes feedback even though they limit the calls they will receive? I remember when we started using Netflix in our home and how difficult it became to justify a trip to the video rental store...when there still was one in business locally! Why? Because for the cost and the number of titles offered not only with streaming video but also with home delivery, there wasn't much competition. I don't mean this to sound like a Netflix ad. The point I am making is that it's almost like complaining about free services from Google. Due to the adoption rate of these technologies, how helpful could such surveys really be and aren't there likely better places to apply analytics in terms of mapping growth and examining the next new opportunity.

Michelle
User Rank
Data Doctor
Re: Survey frustration
Michelle   8/22/2011 11:47:17 PM
NO RATINGS
I'm a Netflix subscriber too. I always long for a second question or at least a comment box on thier one-question surveys. I don't want to spend a long time completing a survey but I want to give accurate feedback. I have wondered what data they were most interested in with these seemingly one-off questions.

louisw900
User Rank
Blogger
Wasted Effort
louisw900   8/23/2011 12:23:28 AM
NO RATINGS

@Joe   As I read your blog ( which is excellent) I could not really believe what I was reading!  One question surveys ?  A sort of true and false scenario ?  You have got to be kidding me, yes I know you are not.  

Anyhow, this is discouraging at best, it looks like Netflix and companies like them are trying to make Analytics as easy as possible and are as you say missing the boat entirely.

What can they possibly hope to garner from such an exercise ?  I cannot for the life of me understand why a company would foolishly waste the effort.


impactnow
User Rank
Blogger
Need Marketing researcher
impactnow   8/23/2011 2:21:27 AM
NO RATINGS

Truly sad, they might have used the same survey to determine the price increase they recently implemented that has caused mass user backlash. It is sad that so many companies are cutting back on research as a result of the economy, if they had a marketing researcher on staff or on contract such a mistake would not have been made. Some of the “do it yourself” research tools on the web have made companies think they can cut out researchers and the software will do it for you—this is a critical mistake that they will see in their erroneous results.

Page 1 / 4   >   >>
Information Resources
More Blogs from Joe Stanganelli
Dun&Bradstreet is tapping its huge database of company information to identify suppliers that have potential human trafficking violations that would be of concern to customers concerned about sustainability.
Speakers at Bio-IT World explore techniques for biotech researchers and others working with big data to identify the accurate data in their data files.
Cooperation among medical researchers -- done right -- very simply can mean lives saved, but the research community needs education on how to execute on that collaboration.
Research into the digital marketing campaigns of some of the presidential candidates raises the question of how much voter messaging is appropriate and who should get which types of messages.
Predictive asset maintenance analytics is finding its niche across industries, from automotive to oil and gas.
Radio Show
A2 Conversations
UPCOMING
Jessica Davis
Our Bodies, Our Data: Medical Records For Sale


2/21/2017   REGISTER   1
ARCHIVE
Jessica Davis
Energy Analytics: Using Data to Find Savings


2/14/2017  LISTEN   44
ARCHIVE
Jessica Davis
Sharpen Your Analytics & Data Management Strategy


2/8/2017  LISTEN   74
ARCHIVE
Jessica Davis
Analytics: Make the Most of Data's Potential in 2017


1/19/2017  LISTEN   19
ARCHIVE
Jessica Davis
A2 Radio: Can You Trust Your Data?


12/20/2016  LISTEN   70
ARCHIVE
James M. Connolly
Retail Analytics: See Where Style Meets Statistics


12/6/2016  LISTEN   53
ARCHIVE
James M. Connolly
Why the IoT Matters to Your Business


11/29/2016  LISTEN   45
ARCHIVE
James M. Connolly
Will Data and Humans Become Friends in 2017?


11/22/2016  LISTEN   40
ARCHIVE
James M. Connolly
We Can Build Smarter Cities


10/20/2016  LISTEN   31
ARCHIVE
James M. Connolly
Visualization: Let Your Data Speak


10/13/2016  LISTEN   70
ARCHIVE
James M. Connolly
How Colleges and Tech Are Grooming Analytics Talent


9/7/2016  LISTEN   56
Information Resources
Quick Poll
Quick Poll
About Us  |  Contact Us  |  Help  |  Register  |  Twitter  |  Facebook  |  RSS