Beware the Sentimental Metric


A public relations executive approached me for advice. She was interested in analyzing text, specifically, media mentions of her employer. Her intention was to relate what was said in the text to certain performance metrics.

First, I needed to know more about these metrics, understand why they were important to her and the business, how they were defined, and how well they performed as business indicators. Guess what? They were the unique creations of the organization, each a mélange of several factors. There was no obvious justification for their use as a means of assessing the health of the business. Yet they were routinely used as a basis for decision making. In fact, the executive was compensated based on these metrics, even though they were not measures of anything that could be directly influenced by public relations.

People stick with metrics that are familiar and accepted even when they don't know how or why the metric originated, whether it performs well for the intended purpose, or whether there might be a better alternative. In short, we sometimes use metrics that have become sentimental favorites rather than effective tools. When these ineffective metrics are used as a basis for decision making, it's bad for the business.

Would it surprise you to hear that I could tell many more stories like these? Let's look at another, and explore how this happens and what you can do to address the problem.

An engineering consultant delivered a final report to a client, who asked me about the sample size used in the project. The report didn't fully explain how the sample size had been selected. Was it appropriate?

The details hadn't been explained because the consultant didn't know the proper techniques for estimating sample sizes for statistical analysis. He had heard, somewhere, that a certain number was a good sample size. So, he used that sample size ever after. It happened that the number he had learned was a good number, for a specific type of analysis, under specific circumstances. Alas, it was not at all appropriate for the work he had just delivered to his client.

How does this happen? Often, it starts with something reasonable. Take the example of that engineering consultant. No doubt it started with someone explaining, and perhaps even showing calculations for, the sample size for a specific situation. Then someone tells someone else that this is a good sample size, but without the explanation of the relevant conditions. And so on, and so on, until the number becomes a guideline for all sorts of things, with no real understanding of reasons or requirements.

He had gotten away with this for years. Why? Because his clients didn't ask enough questions!

So what can you do to avoid sentimental metrics? Ask questions!

Ask many questions, and ask in different ways. Before a project begins, ask about methods. Ask why? Ask for documentation. Ask about assumptions. Ask about these same things as the work progresses, and when you review the final report. Confronted with a new metric -- ask how it is calculated. Ask what the metric is meant to measure. Ask for evidence that it does what it should. Ask about statistical analysis of the metric's performance. Ask about audits of the analysis process.

Got the picture? Ask away, and don't let up until you get the evidence and are able to understand it.

Do you think your company uses sentimental metrics? Share your examples below.

Meta S. Brown, Business Analytics Consultant

Meta S. Brown is a consultant, speaker, and writer who promotes the use of business analytics. A hands-on analyst who has tackled projects with up to $900 million at stake, she is a recognized expert in cutting-edge business analytics. She has conducted more than 4,000 hours of presentations about business analytics, and written guides on neural networks, quality improvement, statistical process control, and many other statistical methods. Meta's seminars have attracted thousands of attendees from across the US and Canada, from novices to professors.

Tell Me a Story: Why Data Analysts Must Be Storytellers, Too

Data alone won't make an analyst's work memorable or actionable in the eyes of a business executive. A story puts it into perspective.

It's the Data, Stupid

When it comes to acquiring the data that will feed your analytics initiative, "free" isn't always the best approach.


Re: Magic And Logic
  • 5/31/2013 12:14:27 AM
NO RATINGS

So what's wrong with wanting it all?

Re: Magic And Logic
  • 5/30/2013 11:09:50 PM
NO RATINGS

@kq4ym, that is a fine line. We want managers and leaders with the intuition to know to think beyond the analytics reports and predictive models and tap into their years of real-world experience for answers ... while we don't want leaders who follow intuition when the facts of data say to do something contrary.

Re: I have seen this so often!
  • 5/30/2013 12:14:46 PM
NO RATINGS

I'm not sure that "no one" is always worde than the two cheaper people. At least "no one" can't actively spread misinformation.

Re: I have seen this so often!
  • 5/30/2013 10:32:40 AM
NO RATINGS

@Meta,

No problem :) That's the only downside to all this typing, can never tell a person's true meaning behind their words.

Re: I have seen this so often!
  • 5/30/2013 10:30:43 AM
NO RATINGS

"don't forget the alternative version of that HR play -- replacing the great, high-paid employee with ... no one."

@Broadway,

Yeah that's always a good one too.  Definitely witnessed that one recently.

Magic And Logic
  • 5/30/2013 6:24:01 AM
NO RATINGS

A lot of "magical" thinking goes on in our heads. We think we can figure out complex questions by answering a simple question. It's not all that easy of course, and the companies and management that start fielding metrics to analyze, often may be guilty of such thinking. It take someone with a bit more logic to field the correct ways to look at the problem to be solved.

Re: I have seen this so often!
  • 5/29/2013 2:52:47 PM
NO RATINGS

Ah ... Yes broadway -- even worse

Re: I have seen this so often!
  • 5/29/2013 2:48:42 PM
NO RATINGS

@noreen, don't forget the alternative version of that HR play -- replacing the great, high-paid employee with ... no one.

Re: costs and results
  • 5/29/2013 11:20:40 AM
NO RATINGS

I'm not sure I want my highly paid researcher doing this work. :)  Maybe we need better procedures for those collecting the data?  Or, and this is likely, I just don't understand your point.

Re: I have seen this so often!
  • 5/29/2013 11:20:25 AM
NO RATINGS

Oops, sorry about that. It's easy to miss sarcasm in text (side note - that's one of the reasons sentiment analysis is so hard).


I have seen plenty of power struggles like that!

Page 1 / 3   >   >>
INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +