Facing Down Danger & Other Big-Data Challenges


One of the great things about my work is that I get to travel and meet really smart people the world over. On Election Day, for example, I happened to be attending an analytics event and had the opportunity to do plenty of offline conversing on who would win.

Just imagine a bunch of quants sitting around a lunch table talking about all the various, possible election outcomes, their related probabilities, the evils of the Electoral College, the possibility of an electoral tie, and on and on. Aside from the obvious danger of waking up nose-deep in a garden salad, that's a truly fascinating conversation, right? Well, for me, not so much.

Why? Because who would win had been obvious to me for quite some time. The data were pretty clear.

Unfortunately for me, I hadn't quite gotten around to starting that blog I've been threatening to write for the last couple of years, and so I hadn't published my erudite, quantitatively sound, and ultimately correct prediction. Some guy named Nate Silver did.

Nate who?

That was my reaction, anyway. Apparently I've somehow managed to avoid hearing about the predictive wunderkind who took the political prognostication realm by storm back in 2008. Fortunately, the event's host had the foresight to give each attendee a copy of Silver's new book, The Signal and the Noise: Why So Many Predictions Fail -- But Some Don't. It had been waiting for me on my place setting that very morning. That was great, because I love books, but too bad, because I just don't read them anymore. Don't get me wrong -- I'm a voracious reader -- but the only time I read something that's printed on actual paper is during that time on the airplane when I have to turn off my e-reader between the ground and 10,000 feet. The only notable exception is the output of one tree-killing colleague of mine who insists on printing absolutely everything. You know who you are.

Anyway, not wanting to spring for the e-reader version with a perfectly good -- if somewhat anachronistic -- actual book in my hand, I decided to wade into the pulp. It turns out that Silver worked as an economic consultant for a big accounting firm, got bored with that (unimaginable, I know), wrote some software to predict baseball player performance la sabermetrics, and then founded a political blog, now on The New York Times, called FiveThirtyEight. It was on that blog that he correctly predicted the outcome of the 2012 presidential election in all 50 states. Four years ago, he only got 49 out of 50 correct (darn that pesky Indiana). It might be easy to argue that 2008 was a fluke, but repeated success is somewhat more difficult to discount.

Because of that success, Silver has received a lot of attention. The question I ask is not how he did it, but rather why his success seems to be such a singularity (setting aside the fact that I wasn't in the game yet). Why aren't lots of other people coming up with the similar predictions and similar rates of accuracy? Perhaps others are, but there are certainly a lot of people trying who aren't, and most of them, sadly for us, seem to have found employment as TV talking heads. I believe one possible answer to my question is foundational to the big-data challenge, and it's embodied in my new favorite quote. From the introduction to Silver's book:

We face danger whenever information growth outpaces our understanding of how to process it.

Does this problem sound familiar to anyone?

Mark Pitts, Data Scientist & Healthcare Executive

Mark Pitts is a data scientist and healthcare executive with more than 25 years of experience solving business problems with technology and analytics. He started programming at the age of 13 – writing his first program on paper because he didn't yet have a computer – and hasn't stopped since. Over the years, he's garnered advanced education and expertise in computing science and business domains, and has applied his multidisciplinary skillset in leading real-world implementations of enterprise resource planning, financial and business intelligence systems, and multimillion-dollar, greenfield development projects to solve enterprise-scale business challenges.

He ultimately progressed from the IT shop to the business, driving the financial performance of healthcare organizations in areas including managed care contracting, provider compensation, payment integrity, forecasting, clinical quality, medical billing, receivables management, and analytics. His innovative work has been recognized with a variety of awards, and his creations support benefits measured in billions of dollars.

In May 2013, Pitts will complete an additional graduate program at Texas A&M University, receiving a Master of Science in statistics with a dual emphasis in applied statistics and biostatistics. He undertook these studies with the recognition that advances in computing technology, the explosion of the electronically interconnected world, and advances in machine learning would combine to change the game, especially in healthcare. He has a passion for writing and public speaking, with a track record of highly rated appearances in a variety of venues, from business and executive conferences to technical and analytics conferences. He has been interviewed, quoted, and featured in a variety of print and online publications, and is currently developing a course designed to introduce business people to the power of data visualization and analytics to solve everyday business problems.

He is also developing a Website for advanced analytics and is incubating a book project that will gain more momentum post-graduation. Pitts is currently employed by a Fortune 25 company, and he lives in Minneapolis with his lovely (and patient) wife, their two teenagers, and four remarkably spoiled dogs. He is also still glowing – and is somewhat harder to live with – after Harvard Business Review declared that data scientist is the sexiest job of the 21st century. You can follow Pitts on Twitter @DatalyticSci.

Here at SGF: Lovin' San Fran & Talking Up HPA

Watch for me at SAS Global Forum, and join me for my 2:00 p.m. PT session on high-performance analytics.

Navy Uses Cellphone App to Fight... 'For Good'

With an Android app, cellphone data, and analytics, the Navy hopes to better understand, monitor, and react during a crisis like severe flooding, earthquakes, and disease outbreaks.


Re: Silver
  • 12/3/2012 7:07:43 PM
NO RATINGS

..

Responding to a recommendation for another political analytics site, Beth writes


Another good one. I'll be looking out for 'em in 2016.


 

Don't forget the midterm elections in 2016.  These contests are becoming just about as important as the presdiential elections — perhaps even more so.

 

Re: Silver
  • 12/1/2012 2:02:27 AM
NO RATINGS

@Bryan_Engle, Thanks for your comment.  I find that one of the chief obstacles to driving adoption of analytics-based practices is that most business people don't understand what is possible.  One of the key roles I play is educating people about the possibilities.  The good news is that with all the buzz around analytics, people are eager to listen these days.

Re: Silver
  • 11/30/2012 5:42:49 PM
NO RATINGS

True, leadership is still a small minority. As more enterprise organizations begin to invest in big data, adapting and creating data driven cultures should be a key strategic involvment in day-to-day decision-making. 

Re: Silver
  • 11/30/2012 12:56:43 PM
NO RATINGS

@Bryan_Engle. Aha! Another good one. I'll be looking out for 'em in 2016.

Re: Does this problem sound familiar to anyone?
  • 11/30/2012 11:30:40 AM
NO RATINGS

@hospice, thanks for your comment. I also believe we will figure it out, and I think we will make great impact that may one day be taken for granted. Mother Earth didn't figure out how to feed us on her own. Tremendous efforts in science and agricultural engineering went into creating the food supply that supports our population. There are many great stories, innovations, and unsung heroes in that history. In the same way, it will take the focused effort and remarkable creativity to tame the data deluge, but the rewards will be great.

Re: Silver
  • 11/30/2012 11:16:37 AM
NO RATINGS

May also want to check out the Princeton Election Consortium, which Deadspin (not your usual source for analytics) calls "criminally underrated."   Agree with your point though - why aren't we using this type of methodology used more in other areas.  We hear a lot about data driven decision making, but seems that we (businesspeople in general) have a hard time moving away from using our expertise/gut to make decisions. 

Does this problem sound familiar to anyone?
  • 11/30/2012 7:33:13 AM
NO RATINGS

"We face danger whenever information growth outpaces our understanding of how to process it."

It reminds me the Malthusian catastrophe that predicted that "the power of population would be superior to the power of the earth to produce subsistence for man". But was Thomas Malthus' observation correct? The earth has ever since learnt how to porduce subsistence to its growing population. I think we will be able to do the same with regard to big-data challenges.

Re: Silver
  • 11/29/2012 3:27:34 PM
NO RATINGS

Ah, callmebob -- I hadn't seen these. Thanks for sharing.

Re: Silver
  • 11/29/2012 2:24:03 PM
NO RATINGS

Beth - More important are Silver's frequent stops on Comedy Central with Jon Stewart and Stephen Colbert. He's everything one imagines a geeky quant should look and act like. Sweet.

Re: Silver
  • 11/29/2012 1:59:18 PM
NO RATINGS

Mark -- certainly having the Times affiliation is great exposure. I think others of the spot-on presidential predictors are well enough known in their own domains, academics, for instance. -- like Election Analytics.

Page 1 / 2   >   >>
INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +