Keeping Your Cool With Big-Data Analytics

As big-data surges around us, you might fear losing your analytical footing. New data types, larger data sets, and speedier time to results can challenge even the most seasoned professionals. This big-data phenomenon is big stuff.

But fear not, says Bill Franks, author of Taming the Big Data Tidal Wave. If you've got a grounded perspective on your company's business needs and understand how to make data work in meeting those, then you should do just fine in this new world of big-data. Big-data fits right into the general analytical trends that have been unfolding over the last decade or two, he explained during our e-chat with him yesterday.

"There really are some commonalities," said Franks, pointing to one major one: the perpetual struggle with the data at hand. "It is always too big and tough to analyze."

But, somehow, the analytics work gets done. The industry cranks out new tools, and analysts get more adept at handling the new requirements. But even as the technology advances and approaches evolve, you can always count on the persistence of a few underlying analytical principles, Franks said. None of this changes with the advent of big-data.

He pointed to his own career's worth of experience as an example. "As an analytical professional, I have always wanted to get all the data I could in order to address a given problem. I now have to add big-data to the mix. It may require some extra work in some cases, but the goal is still to extract meaningful insights from it."

And, as always, what's most important is an understanding of what you will do with big-data to drive value, Franks added.

"The fact it is big or unstructured really doesn't matter when it comes to deciding if you need to use the data and what value it will drive. It only matters to the extent that it impacts what tools and techniques you may have to use. But the important decision is [whether] the data has value or not."

One approach in determining value is to identify a business problem and then brainstorm on whether a given data source can help address it. "If you find a match, do some experimentation," Franks recommended.

Another approach is to explore data proactively and experiment to see what analytics it can drive. The idea behind such work, captured in an "innovation center," is to let the experimentation drive the requirement, he said. "You don't have it figured out up front, so you experiment as a starting point."

Also remember that big-data doesn't translate to "big in scope."

Some projects may be small in scope yet require a lot of data, he noted, citing a retailer that used big-data to identify people who browsed products but didn't buy. "That required processing through a lot of data, but the actual analytics and mechanics were simple. They got a huge ROI. Starting small makes a lot of sense."

Have you gotten started with a big-data project yet, big or small? Share on the message boards below.

Beth Schultz, Editor in Chief

Beth Schultz has more than two decades of experience as an IT writer and editor.  Most recently, she brought her expertise to bear writing thought-provoking editorial and marketing materials on a variety of technology topics for leading IT publications and industry players.  Previously, she oversaw multimedia content development, writing and editing for special feature packages at Network World. In particular, she focused on advanced IT technology and its impact on business users and in so doing became a thought leader on the revolutionary changes remaking the corporate datacenter and enterprise IT architecture. Beth has a keen ability to identify business and technology trends, developing expertise through in-depth analysis and early adopter case studies. Over the years, she has earned more than a dozen national and regional editorial excellence awards for special issues from American Business Media, American Society of Business Press Editors,, and others.

Midmarket Companies: Bring on the Big Data

The "big" in big data is no reflection of the size of the organization embracing its potential.

Push Yourself to New Analytical Discoveries

Take inspiration from Christopher Columbus as you pursue your analytical journeys.

Re: Experimentation with Big Data
  • 5/21/2012 3:47:28 PM

Hi MDMConsult -- I'm curious, would you say this is more important these days, with big-data all around us, or simply more of the same (but with  more data at our disposal)?

Experimentation with Big Data
  • 5/21/2012 3:39:02 PM

@Beth This is very true, Experimentation with Big Data is good. How to solidify these concepts is important in the new work environments with Big Data. "Test and Learn"

Re: Keeping Your Cool With Big-Data Analytics
  • 5/21/2012 10:17:36 AM

SaneIT, I absolutely agree. Companies today need to allow for experimentation and exploration of big-data, internal and external. The everyday analytics projects need to get done, of course, but companies that don't allow for innovative use of their analytics capabilities and data aren't thinking competitively enough.

Re: What if value cannot be derived
  • 5/21/2012 9:52:22 AM

@Altaf  All the raw data processing to make it woth for analysis would take place in step 2.

Re: Keeping Your Cool With Big-Data Analytics
  • 5/21/2012 8:14:24 AM

"Another approach is to explore data proactively and experiment to see what analytics it can drive."  This makes a lot of sense, rather than getting overwhelmed by a big data set playing with it to see what you can get out is a good strategy. Even if you can't use it for your current project you'll know what you can pull from it and it might reveal some data that doesn't replace up but supports your other data.


Re: What if value cannot be derived
  • 5/21/2012 1:11:17 AM

It might be difficult at the beginning but by digging deep into analytics especially when you dig into raw data it is interesting

Re: What if value cannot be derived
  • 5/19/2012 11:55:18 PM

@ Anish

Thanks for enlisting the analysis' steps. I agree that proper database management techniques can solve the problem of having allocate considerable time and effort to identify what data is all about. Though, still data can be in a raw form if it is extracted from an external source. Fine tuning will be required from step 1 to find out its value as the extractor has no control over the form of data kept by the source. 

Re: What if value cannot be derived
  • 5/19/2012 3:32:39 PM

Hi Atlaf- The problem you addressed is a data insufficiency issue. I think an Analyst should categorize his analysis into three steps:

1. Specification gathering and data requirement

2. Data Audit and Preparation

3. Modeling and Visualization

The optimization of first two steps using data source configuration (mapping with third party sources) and proper database management techniques (cleaning and validation) using ETL tools can solve the problem to a major extent.

Data is the NEED not an OVERHEAD
  • 5/19/2012 3:25:29 PM

Thanks for sharing this insight with us Beth. I totally second Franks on the importance of insight driven analytics. The objective for any analysis is usually predefined and calls for data requirements, big or small, which drive the process as the first step towards providing business recommendations. Big data also sometimes offers scope of approaching the problem much more creatively and innovate by exploiting the extra information available from the data.


What if value cannot be derived
  • 5/19/2012 11:50:59 AM

Its true that allocating value to big data can help determine whether advanced techniques/tools need to be applied and considerable time can be devoted to make the data usable for producing analytics which can further derive value to business. However, in many cases, the data is not even in a condition where the expected value of it can be predicted. First the data has to be aligned in a structured way and then it can be estimated that what value it can have if analytics can be applied to it.