Comments
View Comments: Newest First | Oldest First | Threaded View
kq4ym
User Rank
Data Doctor
Re: Surprised?
kq4ym   1/14/2017 10:29:52 AM
NO RATINGS
Understanding emotions might well be a clue to getting a better understanding of what factors to consider in prediction. But, the complexity and scope of that might just be something we won't have a good handle on for some years to come.

kq4ym
User Rank
Data Doctor
Re: Surprised?
kq4ym   12/19/2016 10:26:25 PM
NO RATINGS
Odds only give the probability of an outcome. And unfortunately for gamblers and fortune tellers, those odds are not going to predict an out come. And we have surely learned to not rely of pundits and partisan commentators to predict the future as well. With so many complex variables, it's a wonder that we don't mis-predict more often.

Lyndon_Henry
User Rank
Blogger
Re: Surprised?
Lyndon_Henry   12/5/2016 9:53:08 PM
NO RATINGS
..

DBdebunker writes

Your comment is essentially a POST-HOC explanation of the failure to predict which, as I argue in the piece is much easier than developing a theoretical one from which to derive the prediction IN ADVANCE. 



 

Kinda brings to mind the phrase "Hindsight is 20/20 ..."

..

dbdebunker
User Rank
Data Doctor
Re: Surprised?
dbdebunker   12/5/2016 5:47:28 PM
NO RATINGS
To reiterate, the point is not this specific election, or even elections in general. I only used the election as a vehicle for stressing the distinction between prediction and explanation and the non-scientific tendencies in data science by skipping  validation.

Your comment is essentially a POST-HOC explanation of the failure to predict which, as I argue in the piece is much easier than developing a theoretical one from which to derive the prediction IN ADVANCE.

Another way to say it is that there can be prediction without explanation that cannot be considered scientific.

 

Lyndon_Henry
User Rank
Blogger
Re: Surprised?
Lyndon_Henry   12/5/2016 5:03:49 PM
NO RATINGS
..

I don't think the Big Polling Fail in this election was a failure of some elaborate prediction theory but a basic failure of data input validity. How the voting population felt about key economic issues, policy issues (like immigration), etc. certainly consituted underlying factors, but polling/surveying basically tries to gauge what the vote outcome is gonna be from how the surveyed population indicate they intend to vote (or which way they're "leaning". etc.).

There's a fundamental assumption that, by and large, how people say they're gonna vote is pretty indicative of how they're actually gonna vote. In this electoral contest, there were some reliability problems with that key assumption.

Also. as some have suggested in other A2 comment threads, a major problem was that pollsters apparently tended to focus excessively on big-city urban voters and to underestimate the importance of a lot of small-city, small-town, and rural voters.

Some observers and savvy pundits have also noted that apparently there was a considerable dichotomy in motivation. So people might have told pollsters that they favored Clinton, and intended to vote for her ... but when time came to actually go out, stand in line, and make the effort, a lot of them just stayed on the couch, in the office or other workplace, or went back to sleep.

There's a brier patch of other factors that likely played a role (e.g., the obstacles to voting erected by many GOP-controlled state governments). On the whole, a lot of factors could not be assessed by conventional polling, and no matter how sophisticated the analytics of the results, these factors generally remained below the polling radar. And if your survey population is skewed to start ... it's a recipe for an analytics train wreck.

..

dbdebunker
User Rank
Data Doctor
Re: Surprised?
dbdebunker   12/5/2016 4:15:24 PM
NO RATINGS
That's not what my post is about.

Rather, it is about "data science" trying to get away without theorization preceding prediction, i.e., predictions are derived and tested from theory and data analysis is used to test them. Upside down and backwards: theory is inferred from the prediction.

Data mining is OK for insight into theorization, but then predictions must tested on different data, which is not done. That is not scientific.

 

 

 

PredictableChaos
User Rank
Prospector
Surprised?
PredictableChaos   12/5/2016 3:58:53 PM
NO RATINGS
In the first place, Nate Silver's final prediction was 70% Clinton, 30% Trump.

Most observers seem to treat this as if it was 99% Clinton, 1% Trump. They are overly surprised and this betrays that they simply don't understand what 30% means.

In response to your topic, the prediction method that provides some basis, some explanation for the conclusion gives more information. If we understand more about why people are voting a certain way, we can ask additional questions to better examine the 'why'. If it's emotion, for example; how strong are those emotions? How are those emotions changing with time?



Latest Blogs
It may be impossible to predict the perfect bracket, but these academics have managed to predict perfectly the "at large" bids that were included in the March Madness NCAA college basketball tournament this year and with 96% accuracy over the last 6 years.
The role of the data scientist is changing before our eyes, both in the necessary skillsets and positioning within the company. A2 Radio looks at how the data science role is evolving.
Everybody knows how important analytics is to remaining competitive. Where does your company and industry stand in terms of advanced analytics maturity?
The volume of astronomical data continues to expand at an explosive rate. The application of analytics to so much data offers new opportunities and even career options to space and data enthusiasts.
It's time for the annual college basketball tournament known as March Madness. Did you use analytics to inform your bracket choices?
Radio Show
A2 Conversations
UPCOMING
James M. Connolly
Evolution of the Data Scientist Role


3/23/2017   REGISTER   0
UPCOMING
Jessica Davis
Monetize Your Data: Turning Insights Into Action


3/29/2017   REGISTER   0
ARCHIVE
Jessica Davis
Data Analysts in Training: Meeting Tomorrow's Demand


3/8/2017  LISTEN   63
ARCHIVE
Jessica Davis
Our Bodies, Our Data: Medical Records For Sale


2/21/2017  LISTEN   63
ARCHIVE
Jessica Davis
Energy Analytics: Using Data to Find Savings


2/14/2017  LISTEN   44
ARCHIVE
Jessica Davis
Sharpen Your Analytics & Data Management Strategy


2/8/2017  LISTEN   74
ARCHIVE
Jessica Davis
Analytics: Make the Most of Data's Potential in 2017


1/19/2017  LISTEN   19
ARCHIVE
Jessica Davis
A2 Radio: Can You Trust Your Data?


12/20/2016  LISTEN   70
ARCHIVE
James M. Connolly
Retail Analytics: See Where Style Meets Statistics


12/6/2016  LISTEN   53
ARCHIVE
James M. Connolly
Why the IoT Matters to Your Business


11/29/2016  LISTEN   45
ARCHIVE
James M. Connolly
Will Data and Humans Become Friends in 2017?


11/22/2016  LISTEN   40
ARCHIVE
James M. Connolly
We Can Build Smarter Cities


10/20/2016  LISTEN   31
Information Resources
Quick Poll
Quick Poll
About Us  |  Contact Us  |  Help  |  Register  |  Twitter  |  Facebook  |  RSS