Don't Go Steady (With 1 Data Analysis Method)

Analysts sometimes fall so deeply in love with a particular analytic method that they use it all the time.

You've witnessed it. There's the consultant who heard that a certain sample size was good, and went on to use it for everything. And the engineer whose attraction to the elegance of linear models led to myriad elaborate data transformations in the name of linearization. Analysts who grumble during seminars, pooh-poohing any suggestion that new techniques have value.

In dating, there may be good reasons to stick with just one special someone, but in data analysis, it's ridiculous.

I'm not suggesting that you dump your favorites. But it's not uncommon to get so comfortable with certain tools or techniques that you use them out of habit, without giving much thought to whether that comfortable approach is really the best fit for your needs. Smart and experienced analysts are sometimes spotted using methods that are not right for the data in question.

You may have heard about the Netflix Prize: The video rental and streaming giant offered a $1 million bounty for a better model to predict movie ratings. One thing about that competition grabbed my attention -- one that I've not heard anyone else mention.

Netflix movie ratings are star ratings -- one star for a movie you detest, five for a movie you adore. These ratings are ordinal (suitable for ranking but not for adding, dividing, and so on). Yet the metric used to judge the competitors (root mean square error) isn't appropriate for ordinal metrics. This is a fact stated in thousands of textbooks, though often ignored in practice.

People treat ordinal ratings as ratio variables (variables you can use in numeric calculations) all the time. I'm no purist. If it works for them, it's usually OK with me. But that's acceptable when a rough solution is good enough, not when your corporate lifeblood depends on it. Not when you employ a staff of analysts. Not when you're offering a million-dollar bounty for a better model.

Think it over. Does the method you're using really fit the application? You just might find an alternative that gives better results, more defensible results, results that can give you a competitive edge. Want to learn some new tricks? Get a fresh point of view by researching how professionals in fields other than your own approach problems similar to yours. Attend a new conference, or just search online for presentations that ­many speakers post on conference sites or sharing sites (such as SlideShare). Even YouTube has some thought-provoking content on data analysis.

Today, give some thought to whether you might be using some methods due to habit instead of for better reasons. Is it time for you to play the field with some new analytic methods?

Meta S. Brown, Business Analytics Consultant

Meta S. Brown is a consultant, speaker, and writer who promotes the use of business analytics. A hands-on analyst who has tackled projects with up to $900 million at stake, she is a recognized expert in cutting-edge business analytics. She has conducted more than 4,000 hours of presentations about business analytics, and written guides on neural networks, quality improvement, statistical process control, and many other statistical methods. Meta's seminars have attracted thousands of attendees from across the US and Canada, from novices to professors.

Tell Me a Story: Why Data Analysts Must Be Storytellers, Too

Data alone won't make an analyst's work memorable or actionable in the eyes of a business executive. A story puts it into perspective.

It's the Data, Stupid

When it comes to acquiring the data that will feed your analytics initiative, "free" isn't always the best approach.

Re: Good idea
  • 10/22/2013 2:22:45 PM

Having a good star rating for Netflix may not initially increase subscribers and revenue but it will help them stay competitive. The star rating keeps their current subscribers happy which hopefully through word of mouth increase new customers.


Re: A message of innovation
  • 10/14/2013 1:30:57 PM

I know. I love that one, too! Says it just perfectly, right?

Re: A message of innovation
  • 10/14/2013 1:02:07 PM

"The Bozone Layer" -- can I steal that?

A message of innovation
  • 10/14/2013 11:37:00 AM

Hi Meta. I'm attending the Direct Marketing Association conference here in Chicago, and having just listening to today's keynote address with Terry Jones, of Travelocity & fame, I'm finding his message of innovation quite applicable to this post. While Jones directed his mantra of "innovate, innovate, innovate" to his marketing audience (one presumably highly data driven) he could easily have been talking to data analysts. I would imagine, if so, he'd advise data analysts to try new models. If they fail, try again. You can't get ahead by sticking with the same old, same old -- especially when it no longer applies or isn't the right way to go about it in the first place.

Re: Good idea
  • 10/14/2013 10:26:02 AM

I suspect it was more PR than actual value. If it really ties to revenue, it is probably worth far nore that $1MM,

Re: Good idea
  • 10/14/2013 10:22:22 AM

Netflix emplys a team of trained analysts, so I don't doubt that they had a reason to belive it was important. For example, I wouldn't expect someone who had not liked the movies he'd seen would not be likely to renew - so, there's a revenue connection.

I'd like to hear how they made the case to offer a $1 million prize. Was it purely based on how they valued the model, or was PR the real goal?

Good idea
  • 10/14/2013 9:55:08 AM

If you think about it, of course one size does not fit all. I wonder if Netflix realizes that they are probably not even asking the right questions. Is the star rating what they are after or is it an icrease in subscribers and revenue? Does that really corelate with the star ratings of the content?