Situation: You have read a report that describes the objective, methods, results, and significance of an analytic inquiry. The author’s objective is not 100% clear to you, but you attribute that to the fact that highly intelligent people communicate at a pitch that only Mensa members can hear.
The literature review references doctoral dissertations written in German. Every fifth sentence uses a Latin phrase (like Ceteris paribus -- all things being equal). There is enough linear notation to require a degree in quantum physics to understand. The explained variance in the regression model is less than 3%, but the author frames the results in language that suggests the results are ground-breaking.
There are even several charts and graphs that tell a story in pictures, but you are too ashamed to tell someone that they do not make sense to you. When you Google the author, you find that he or she hosts YouTube tutorials and authors several blogs.
Question: As a professional analyst, should you readily accept the contents of the report?
Answer: Not on your life! As an analytics professional, especially if you are just getting started in the field, it is important to know that everything that is presented or published is not always correct or unchallengeable. One of the skills you will need to be successful in analytics is the ability to respectfully question and push back on statements that do not make sense.
The analytics profession bears the burden of producing results. When the results are more mind-numbing than awe-inspiring (i.e., studies show that being alive is the largest risk factor of experiencing death), authors can be tempted to "put lipstick on the pig" to justify the project's funding. Here are some embellishment techniques that you should look for in the analytic report:
- Attempting to establish a big truth with a small sample size. Nine out of ten doctors may recommend a particular glucose drug for diabetics, but how many doctors (in the US, Brazil, China -- location is important) does this constitute? It could be nine out of ten doctors in the same HMO group in Rhode Island or Wyoming.
- Distorting details of tests. Stating that rats who ate chocolate developed cancer does not state the proportion of chocolate consumed. In some of the animal studies, the test subjects consume twice their body weight of a substance for a period of time, and then develop a morbid condition. But any human who consumes twice his/her body weight of anything will also develop a morbid condition.
- Understating or misstating units of measurement. One bad habit I have seen is when an author uses charts/graphs from other studies as points of comparison, but the units of measurement are not the same. This becomes more problematic when trying to overlay two graphs that are scaled differently.
- Presenting misleading comparisons. This is one of my favorites: "Software product A is faster than software product B." The question to ask here is, do the two products execute in the same manner? Some products write to disk while others write to RAM. If you purchase the RAM-based product without also buying more RAM, the speed differential may be non-existent.
- Changing the hypothesis to fit the results. This is one is self-explanatory. It's unethical and only done by those who feel that it is in their best interests to do so.
In short, part of being an analytics professional is assessing the work of other analysts. While we seek to leverage our curiosity and tenacity for learning, a healthy dose of skepticism is in order. We are a self-governing body and have to keep each other accountable in advancing the profession. Respectfully pushing back helps to push us forward.