Comments
Prediction, Explanation and the November Surprise
View Comments: Newest First | Oldest First | Threaded View
Re: Surprised?
  • 1/14/2017 10:29:52 AM
NO RATINGS

Understanding emotions might well be a clue to getting a better understanding of what factors to consider in prediction. But, the complexity and scope of that might just be something we won't have a good handle on for some years to come.

Re: Surprised?
  • 12/19/2016 10:26:25 PM
NO RATINGS

Odds only give the probability of an outcome. And unfortunately for gamblers and fortune tellers, those odds are not going to predict an out come. And we have surely learned to not rely of pundits and partisan commentators to predict the future as well. With so many complex variables, it's a wonder that we don't mis-predict more often.

Re: Surprised?
  • 12/5/2016 9:53:08 PM
NO RATINGS

..

DBdebunker writes


Your comment is essentially a POST-HOC explanation of the failure to predict which, as I argue in the piece is much easier than developing a theoretical one from which to derive the prediction IN ADVANCE. 


 

Kinda brings to mind the phrase "Hindsight is 20/20 ..."

..

Re: Surprised?
  • 12/5/2016 5:47:28 PM
NO RATINGS

To reiterate, the point is not this specific election, or even elections in general. I only used the election as a vehicle for stressing the distinction between prediction and explanation and the non-scientific tendencies in data science by skipping  validation.

Your comment is essentially a POST-HOC explanation of the failure to predict which, as I argue in the piece is much easier than developing a theoretical one from which to derive the prediction IN ADVANCE.

Another way to say it is that there can be prediction without explanation that cannot be considered scientific.

 

Re: Surprised?
  • 12/5/2016 5:03:49 PM
NO RATINGS

..

I don't think the Big Polling Fail in this election was a failure of some elaborate prediction theory but a basic failure of data input validity. How the voting population felt about key economic issues, policy issues (like immigration), etc. certainly consituted underlying factors, but polling/surveying basically tries to gauge what the vote outcome is gonna be from how the surveyed population indicate they intend to vote (or which way they're "leaning". etc.).

There's a fundamental assumption that, by and large, how people say they're gonna vote is pretty indicative of how they're actually gonna vote. In this electoral contest, there were some reliability problems with that key assumption.

Also. as some have suggested in other A2 comment threads, a major problem was that pollsters apparently tended to focus excessively on big-city urban voters and to underestimate the importance of a lot of small-city, small-town, and rural voters.

Some observers and savvy pundits have also noted that apparently there was a considerable dichotomy in motivation. So people might have told pollsters that they favored Clinton, and intended to vote for her ... but when time came to actually go out, stand in line, and make the effort, a lot of them just stayed on the couch, in the office or other workplace, or went back to sleep.

There's a brier patch of other factors that likely played a role (e.g., the obstacles to voting erected by many GOP-controlled state governments). On the whole, a lot of factors could not be assessed by conventional polling, and no matter how sophisticated the analytics of the results, these factors generally remained below the polling radar. And if your survey population is skewed to start ... it's a recipe for an analytics train wreck.

..

Re: Surprised?
  • 12/5/2016 4:15:24 PM
NO RATINGS

That's not what my post is about.

Rather, it is about "data science" trying to get away without theorization preceding prediction, i.e., predictions are derived and tested from theory and data analysis is used to test them. Upside down and backwards: theory is inferred from the prediction.

Data mining is OK for insight into theorization, but then predictions must tested on different data, which is not done. That is not scientific.

 

 

 

Surprised?
  • 12/5/2016 3:58:53 PM
NO RATINGS

In the first place, Nate Silver's final prediction was 70% Clinton, 30% Trump.

Most observers seem to treat this as if it was 99% Clinton, 1% Trump. They are overly surprised and this betrays that they simply don't understand what 30% means.

In response to your topic, the prediction method that provides some basis, some explanation for the conclusion gives more information. If we understand more about why people are voting a certain way, we can ask additional questions to better examine the 'why'. If it's emotion, for example; how strong are those emotions? How are those emotions changing with time?



INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +