Putting Analytics Fragmentation Into Perspective

As an analyst working with newer forms of communication and emerging media, I've seen a proliferation of analytics platforms aimed at these areas. I don't get it. Why, I wonder, do we need new analytics platforms when plenty of excellent ones are already available?

Could it be that, as new data formats arise and we face the constant need to come up with new analytics capabilities, we're falling behind? Instead of finding a more persistent and long-term solution to the data fragmentation dilemma, are we generating more and more one-off solutions?

The problem lies in the fact that most of today's analytics platforms are "static," as Ravi Kalakota, a managing director of the global professional services firm Alvarez & Marshal, discusses in this blog post.

The dominant design of such platforms is dependent on a specific set of questions and dimensions. However, as businesses and individuals are evolving, they face a mushrooming cloud of fragmented information they don't know how to use, along with a shortage of qualified personnel to handle it. Their in-house or third-party platforms aren't flexible enough to meet their needs, so they buy or develop different analytics platforms -- platforms that often overlap in data and purpose (particularly in the social media space). In particular, platforms fail to allow ad hoc exploration using real-time data feeds. If they do allow for such exploration, the process is time consuming and costly.

Beginning in 2010, we've seen many acquisitions in the social analytics and cloud data space, but there hasn't been much in the way of creating a more dynamic approach to analytics platforms, a la the analytics-as-a-service concept discussed in this Sand Hill blog.

In the emerging media space where I'm most focused (through client work and the social media measurement and social media for the arts courses I teach at University of California-Irvine Extension and at Rutgers University, respectively), I've come across three constructs that explain the analytics fragmentation we face and suggest possible solutions we can explore.

  1. Dark data: This is the data organizations generate but don't understand how to optimize or use, so they aren't drawing much insight from it. As Wired Magazine wrote several years ago, "Freeing up dark data could represent one of the biggest boons to research in decades, fueling advances in genetics, neuroscience, and biotech." But unless we have questions to apply against the data, it may make no difference what the data is. Rather than focusing on the data, maybe our focus really belongs on the nature of our questions and how our data can help answer them. Dark data might be worth exploring.
  2. Dark social: This aspect of dark data is related to social media activity that isn't easily traceable to the actual origin point, as the Atlantic recently discussed. Web analytics is limited in what it can capture, and it has to be instrumented deliberately to capture much of anything specific, which leaves out much of social data activity. In addition, the privacy settings in Facebook, LinkedIn, most forums, and some photo-sharing and blogging sites make sharing social data across analytics applications difficult. Attempts to solve the problem have simply generated more analytics fragmentation.
  3. Ultraviolet data: I coined the term "UV data" in 2010, when I was among the first analysts to recognize a problem that analytics platforms weren't designed to solve. That problem: Businesses aren't equipped to capture much of the data they need, but data is present nonetheless, and it is waiting to be collected. It's quite possible the uncollected data is more valuable than the collected data. I've devised an audit process for discovering UV data, and I am figuring out how best to collect and organize it.

No doubt, all the forms of dark data, dark social, and UV data boil down to the same thing -- what we don't know might be as important as what we do know, if not more. It also explains why many analytics platform fail to drive the needed insights: They're based on static dimensions and haven't been built to answer the questions that we must increasingly solve in order to succeed.

Marshall Sponder, Web Analytics and SEO/SEM Specialist

Marshall Sponder is a Web analytics and SEO/SEM specialist with expertise in market research, social media, networking, and public relations. As both an in-house team leader and consultant, he has used sophisticated analysis to optimize the social media marketing efforts of companies and brands including IBM, Monster, Porter Novelli, WCG, Gillette, Pfizer, Warner Brothers, Laughing Cow, The New York Times, and Havana Central. Sponder is a board member emeritus at the Web Analytics Association, a member of the Search Engine Marketing Professionals Organization (SEMPO), and a member of the Certified Institute of Public Relations Social Media Measurement Study Group (CIPR).

Putting Analytics Fragmentation Into Perspective

When the data we don't know is as important as the data we do, our analytics platform are all but guaranteed to fail us.

3 Musts You'd Find in an Ideal Analytics Platform

Segmentation, multichannel integration, and intelligent dashboard reporting are vital capabilities, yet many business analytics solutions fall short.

Re: Dark data, looming questions
  • 12/3/2012 8:34:25 PM

I like the terminology and will love to float the term around and see people's reaction. 

The thing with many old systems are they are not flexible, scalable and tend to be products of departments silos , with one or two purposes that need to become more connected. Hence, giving employees access to the whole brain and not just one part.

Re: Dark data, looming questions
  • 12/1/2012 12:19:15 AM

I think the right question would be contextually correct to the data you have to work with on one hand and your core needs, on the other other.     I think it takes some perspective (ha, the title of my post!) in order to come up with the best questions, and I think some give and take, which a team of trusted advisors, and perhaps one or two new people who can introduce new perspectives, might be the best way to go.

I often worry that by trying to come up with the questions myself, without really hashing them out in group discussion, a too near sighted formulation comes up that ends up missing the point, and miss the insights - the very thing that leads to Dark Data, perhaps Dark Social, and certainly UV Data.

Re: Dark data, looming questions
  • 11/30/2012 9:37:11 PM

Agreed. I know people will naturally want to know what Makes a question right or not - and that could be a post all by its self..

Re: Dark data, looming questions
  • 11/30/2012 5:53:25 PM

We have a lot of information in front of us constantly, but unless we have a reason (need) and wonder if the data has a solution for us, we may never do anything. People are need based - until we have a need (which generates a series of questions) to understand and solve a problem (often one that is thrust upon us) we may never compose the questions that lead us to examine the data in forint of us. That's exactly the situation I posited we had at my Rutgers University class that I teach on Social Media for The Arts. The first temptation was to give students the fundamentals of self promotion so they could take the Art and Humanities trading they get and apply it in Social Media. But from my perspective ... Simply getting someone to apply what they know on a new medium (dark social in this context) is not as useful as if we taught them more important questions to ask in the first place. That required exposing a new framework and that leads to new questions, which in turn leads us to examine the data around us, figure out if it might be useful, and then wonder how and what.

Dark data, looming questions
  • 11/30/2012 5:37:05 PM

Hi Marshall, I'm trying to get my mind around this point you make about dark data: "But unless we have questions to apply against the data, it may make no difference what the data is. Rather than focusing on the data, maybe our focus really belongs on the nature of our questions and how our data can help answer them." Don't you have to know the type of data you have to know the type of questions you can answer? Unless you're also including external data in this dark data category?

Re: Freeing up dark data...
  • 11/30/2012 5:22:19 PM

Yes, and I am glad you enjoyed the article.

Freeing up dark data...
  • 11/30/2012 3:00:16 PM

Dark data either way is a benefit. Whether data is determined as "useless" it can easily be deleted and written off as "no value." Integrating dark data with existing data can prove to be significant data for business intelligence further adding good value to measuring. It is a good point in this article and to assess data accordingly