Many people make it through statistics class without a clear understanding of the common process used for all statistical hypothesis tests. In fact, it's not unusual to complete the class without realizing there is a common process. That's not the students' fault -- most professors don't emphasize it.
Executives need accurate, relevant information that is presented simply, quickly, and clearly. They need answers to their questions, and they need to have confidence, not just in the data's message, but also in the messenger.
What happens to analysts who don't know the process and don't use it to clarify their thinking? Most often, executives just don't find those analysts persuasive. Their presentations sound too esoteric, and their answers to questions do not satisfy.
Here's a confession: I love data analysts who are full of hot air. It is so easy to disarm these windbags. All I have to do is smile sweetly, look them straight in the eye, and ask a little question -- What were your assumptions here? -- or make a simple point -- The method you've used was designed for ratio measures, but this metric is ordinal -- and it's all over. Everybody in the room understands. Don't let that happen to you. The five-step process common to every statistical hypothesis test will make your work bulletproof.
- State a null and alternative hypothesis. If this doesn't seem familiar, or you are not sure of how it's done, read any introductory statistics textbook. It's in all of them.
- State the assumptions that are reasonable for the application. These are conditions we believe to be true: simple random sampling, normally distributed population, and so on.
- Select a test type. The type of test to use depends on what you are measuring and the assumptions you can reasonably make. Each test has its own assumptions, and you may have to comb the books carefully to find them. Make sure the assumptions fit your situation. If they don't, you've selected the wrong test.
- Calculate a p value. This can be done for any type of statistical hypothesis test. The p value is interpreted the same way for all tests. Very low p values imply strong evidence against a null hypothesis. High p values represent weaker evidence against the null hypothesis -- or no such evidence at all.
- State your conclusion in a simple, plain-language sentence. This one is the downfall of many a mathematical genius. State the results in terms any business person can understand, such as "Our tests show no evidence that the test coupon will outperform the coupon we are using today." Avoid sentences like "The p value was 0.12, so we failed to reject the null hypothesis."
Do you use these five steps in your work? Please share your experience.