A Skeptic's Guide to Analytic Results

Situation: You have read a report that describes the objective, methods, results, and significance of an analytic inquiry. The authorís objective is not 100% clear to you, but you attribute that to the fact that highly intelligent people communicate at a pitch that only Mensa members can hear.

The literature review references doctoral dissertations written in German. Every fifth sentence uses a Latin phrase (like Ceteris paribus -- all things being equal). There is enough linear notation to require a degree in quantum physics to understand. The explained variance in the regression model is less than 3%, but the author frames the results in language that suggests the results are ground-breaking.

There are even several charts and graphs that tell a story in pictures, but you are too ashamed to tell someone that they do not make sense to you. When you Google the author, you find that he or she hosts YouTube tutorials and authors several blogs.

Question: As a professional analyst, should you readily accept the contents of the report?

Answer: Not on your life! As an analytics professional, especially if you are just getting started in the field, it is important to know that everything that is presented or published is not always correct or unchallengeable. One of the skills you will need to be successful in analytics is the ability to respectfully question and push back on statements that do not make sense.

The analytics profession bears the burden of producing results. When the results are more mind-numbing than awe-inspiring (i.e., studies show that being alive is the largest risk factor of experiencing death), authors can be tempted to "put lipstick on the pig" to justify the project's funding. Here are some embellishment techniques that you should look for in the analytic report:

  • Attempting to establish a big truth with a small sample size. Nine out of ten doctors may recommend a particular glucose drug for diabetics, but how many doctors (in the US, Brazil, China -- location is important) does this constitute? It could be nine out of ten doctors in the same HMO group in Rhode Island or Wyoming.

  • Distorting details of tests. Stating that rats who ate chocolate developed cancer does not state the proportion of chocolate consumed. In some of the animal studies, the test subjects consume twice their body weight of a substance for a period of time, and then develop a morbid condition. But any human who consumes twice his/her body weight of anything will also develop a morbid condition.

  • Understating or misstating units of measurement. One bad habit I have seen is when an author uses charts/graphs from other studies as points of comparison, but the units of measurement are not the same. This becomes more problematic when trying to overlay two graphs that are scaled differently.

  • Presenting misleading comparisons. This is one of my favorites: "Software product A is faster than software product B." The question to ask here is, do the two products execute in the same manner? Some products write to disk while others write to RAM. If you purchase the RAM-based product without also buying more RAM, the speed differential may be non-existent.

  • Changing the hypothesis to fit the results. This is one is self-explanatory. It's unethical and only done by those who feel that it is in their best interests to do so.

In short, part of being an analytics professional is assessing the work of other analysts. While we seek to leverage our curiosity and tenacity for learning, a healthy dose of skepticism is in order. We are a self-governing body and have to keep each other accountable in advancing the profession. Respectfully pushing back helps to push us forward.

Related posts:

Bryan Beverly, Statistician, Bureau of Labor Statistics

Bryan K. Beverly is from Baltimore. He has a BA in sociology from Morgan State University and an MAS degree in IT management from Johns Hopkins University. His continuing education consists of project management training through the ESI International/George Washington University programs. He began his career in 1984, the same year he was introduced to SAS software. Over the course of nearly 30 years, he has used SAS for data processing, analytics, report generation, and application development on mainframes, mini-computers, and PCs. Bryan has worked in the private sector, public sector, and academia in the Baltimore/Washington region. His work initially focused on programming, but over the years has expanded into project management and business development. Bryan has participated in in-house SAS user groups and SAS user group conferences, and has published in SAS newsletters, as well as company-based newsletters. Over time, his publications have expanded from providing SAS technical tips to examining the sociological, philosophical, financial, and political contexts in which IT is deployed. He believes that the key to a successful IT career is to maintain your skills and think like the person who signs your paycheck.

Tale Of Two Cities: Visual Analysis of Baltimore Crime

Why has the crime rate shifted in Baltimore? A visual analytics video tells the story.

When Good Algorithms, Tech Stop Being Good

What happens when an algorithm designed for good gets used for illegal purposes? Technology designed to fix a problem can ultimately lead to negative consequences.

Re: Respectfully pushing back
  • 2/12/2014 8:07:24 PM

I would add a couple additional "leadership" animals --- the snake, the passive-aggressive types who will destroy you when you turn your back on them, and the vultures, who wait out all the foxes and lions and pick their carcasses after the fight. 

Re: Respectfully pushing back
  • 2/12/2014 5:04:09 PM

@Broadway, I greatly admire your optimism; I used to think just like that when I younger. It's wonderful to be a middle-aged person, but the downside is that time has a way of making one a skeptic. I agree that over time, work places will be less combat-oriented. I think that the gradual increase in women in the white-collar labor force is slowly forcing work place culture to be less Spartan-like (or at least the lawyers are doing that). But your hopes for a better day remind me of Vilfredo Pareto's theory of the circulation of elites. He said that leadership styles cycle through periods of lions (traditional male brute force and intimidation) and foxes (people who can covertly manipulate, massage and convince people to do their bidding). I can see a day when cultures are kinder and gentler. But I would warn you that whether you work with lions or foxes, the outcome for the slow and weak will be the same. Foxes are just as predatory as lions, they just are kinder and gentler in their approach. But ultimately, the goal is to 'do you in'; a soft approach but just as deadly. So keep your whether combative or diplomatic, keep your skepticism radar up - trust but verify!

Re: Respectfully pushing back
  • 2/11/2014 10:55:47 PM

@bkbeverly, you are right. Some "old dogs" still like to fight like young pups. And there are still plenty of dogs that will wait till your not looking and will nip at your heels. One day all company cultures will be kinder and gentler but until then ...

Re: And then there is bias
  • 2/11/2014 10:14:15 AM

Meta, Could not agree with you more. Bias is a chronic problem in sampling. As in the case you mentioned, it was a random sample of people with at least a BA/BS degree. Bias is also a problem in survey design and reporting. People with advanced degrees tend to write to the level of their peers. Some of these survey results represent a 'privileged class' and not the thoughts of Joe or Jane Lunchbucket. Nothing wrong with that as long as you state that the external validity applies to white collar professionals. When your colleagues, neighbors and Facebook friends are in the same socioeconomic status groups, then one's perception of applicable analytical truths is skewed. Not intentional, not malicious, but innate and habitual. When the only people in your circle of associations are like you, then the assumptions or assertions you make should be 'viewed with a jaundiced eye'. Good observation (as always).

Re: Respectfully pushing back
  • 2/10/2014 4:36:59 PM

@Broadway - Not a specific event. Just a gradual realization that it was a hollow victory to attack a person and not constructively work toward refining ideas. Particularly since many environments are charged with testosterone, challenging a coworker or a superior that you wanted to knock off was expected. When you were a young pup, you wanted to prove that you were ready to be a big dog. But when you are a young pup, you do not know that every big dog one day becomes an old dog. No not a specific incident in this case - just a gradual maturing. Every one grows old, but some never grow up; there are some gray haired babies whose insecurities find it better to reaffirm themselves by attacking others rather than preserving the dignity of others and helping them to shape their ideas. It is nothing wrong with being a skeptic, but it can turn out wrong if it is done the wrong way for the wrong reason. So all that to say is that these are the lessons learned from an old dog. I can't learn any new tricks, but I know when someone is trying to play them.

Re: healthy skepticism
  • 2/10/2014 9:18:00 AM

From your blog post to a real-world example of healthy skepticism. Perfect!

And then there is bias
  • 2/10/2014 9:12:14 AM


I love this post - these are terrific points, and all should be taken into account!

May I also add that we should examine methods for signs of plain old ordinary bias? For example, I've seen a number of posts recently about a data science salary survey. Most of the posts push the value of certain skills, based on the survey results.

A quick review of the survey report reveals that all the respondents were attendees of one particular conference. That conference focuses on certain specific skills, and certain types of applications. It's by no means a representative sample of the analytics community, or even the "data science" community as a whole. It's a biased sample, and it's obvious that nobody even tried to eliminate bias when planning the survey.

Alas, that's a common problem.

Re: Respectfully pushing back
  • 2/9/2014 11:39:09 PM

@bkbeverly, was there a specific event that flicked on the light bulb?

Re: Respectfully pushing back
  • 2/8/2014 7:48:17 PM

Broadway, I used to be one of those persons! In an academic environment, that is the way of life. It feeds the ego when you think you have won a round of one-upsmanship. But over the years, I learned that it is better to make colleagues than enemies. In fact, "there are no permanent friends and there are no permanent enemies, but there are permanent interests". The person you intellectually attack today may be a needed friend tomorrow. And since all of us have strengths and weaknesses, you really gain nothing by seeking to show up someone because what goes around comes around. So over the years, I found it more constructive to question the range of options rather than attack a person. Oh I am quite tempted sometimes when someone else makes a first strike at me; the natural response is to hit back. And even then I have to quickly assess if the statement is personal or professional. Cannot say that I always respond like a grown up. But I do try not to initiate conflict. So yes, I try these days to take a softer approach, but oh no, that is not how I started out.

Re: Respectfully pushing back
  • 2/8/2014 6:22:15 PM

@bkbeverly, granted I was in grad school nearly 15 years ago and it was for a liberal arts degree, but back then students and teachers didn't question -- they always accused and attacked. Why is this relevant? Etc. etc. perhaps it was the culture of that school.

Page 1 / 3   >   >>