As an analyst working with newer forms of communication and emerging media, I've seen a proliferation of analytics platforms aimed at these areas. I don't get it. Why, I wonder, do we need new analytics platforms when plenty of excellent ones are already available?
Could it be that, as new data formats arise and we face the constant need to come up with new analytics capabilities, we're falling behind? Instead of finding a more persistent and long-term solution to the data fragmentation dilemma, are we generating more and more one-off solutions?
The problem lies in the fact that most of today's analytics platforms are "static," as Ravi Kalakota, a managing director of the global professional services firm Alvarez & Marshal, discusses in this blog post.
The dominant design of such platforms is dependent on a specific set of questions and dimensions. However, as businesses and individuals are evolving, they face a mushrooming cloud of fragmented information they don't know how to use, along with a shortage of qualified personnel to handle it. Their in-house or third-party platforms aren't flexible enough to meet their needs, so they buy or develop different analytics platforms -- platforms that often overlap in data and purpose (particularly in the social media space). In particular, platforms fail to allow ad hoc exploration using real-time data feeds. If they do allow for such exploration, the process is time consuming and costly.
Beginning in 2010, we've seen many acquisitions in the social analytics and cloud data space, but there hasn't been much in the way of creating a more dynamic approach to analytics platforms, a la the analytics-as-a-service concept discussed in this Sand Hill blog.
Dark data: This is the data organizations generate but don't understand how to optimize or use, so they aren't drawing much insight from it. As Wired Magazine wrote several years ago, "Freeing up dark data could represent one of the biggest boons to research in decades, fueling advances in genetics, neuroscience, and biotech." But unless we have questions to apply against the data, it may make no difference what the data is. Rather than focusing on the data, maybe our focus really belongs on the nature of our questions and how our data can help answer them. Dark data might be worth exploring.
Dark social: This aspect of dark data is related to social media activity that isn't easily traceable to the actual origin point, as the Atlantic recently discussed. Web analytics is limited in what it can capture, and it has to be instrumented deliberately to capture much of anything specific, which leaves out much of social data activity. In addition, the privacy settings in Facebook, LinkedIn, most forums, and some photo-sharing and blogging sites make sharing social data across analytics applications difficult. Attempts to solve the problem have simply generated more analytics fragmentation.
Ultraviolet data: I coined the term "UV data" in 2010, when I was among the first analysts to recognize a problem that analytics platforms weren't designed to solve. That problem: Businesses aren't equipped to capture much of the data they need, but data is present nonetheless, and it is waiting to be collected. It's quite possible the uncollected data is more valuable than the collected data. I've devised an audit process for discovering UV data, and I am figuring out how best to collect and organize it.
No doubt, all the forms of dark data, dark social, and UV data boil down to the same thing -- what we don't know might be as important as what we do know, if not more. It also explains why many analytics platform fail to drive the needed insights: They're based on static dimensions and haven't been built to answer the questions that we must increasingly solve in order to succeed.
I like the terminology and will love to float the term around and see people's reaction.
The thing with many old systems are they are not flexible, scalable and tend to be products of departments silos , with one or two purposes that need to become more connected. Hence, giving employees access to the whole brain and not just one part.
I think the right question would be contextually correct to the data you have to work with on one hand and your core needs, on the other other. I think it takes some perspective (ha, the title of my post!) in order to come up with the best questions, and I think some give and take, which a team of trusted advisors, and perhaps one or two new people who can introduce new perspectives, might be the best way to go.
I often worry that by trying to come up with the questions myself, without really hashing them out in group discussion, a too near sighted formulation comes up that ends up missing the point, and miss the insights - the very thing that leads to Dark Data, perhaps Dark Social, and certainly UV Data.
We have a lot of information in front of us constantly, but unless we have a reason (need) and wonder if the data has a solution for us, we may never do anything.
People are need based - until we have a need (which generates a series of questions) to understand and solve a problem (often one that is thrust upon us) we may never compose the questions that lead us to examine the data in forint of us.
That's exactly the situation I posited we had at my Rutgers University class that I teach on Social Media for The Arts. The first temptation was to give students the fundamentals of self promotion so they could take the Art and Humanities trading they get and apply it in Social Media.
But from my perspective ... Simply getting someone to apply what they know on a new medium (dark social in this context) is not as useful as if we taught them more important questions to ask in the first place.
That required exposing a new framework and that leads to new questions, which in turn leads us to examine the data around us, figure out if it might be useful, and then wonder how and what.
Hi Marshall, I'm trying to get my mind around this point you make about dark data: "But unless we have questions to apply against the data, it may make no difference what the data is. Rather than focusing on the data, maybe our focus really belongs on the nature of our questions and how our data can help answer them." Don't you have to know the type of data you have to know the type of questions you can answer? Unless you're also including external data in this dark data category?
Dark data either way is a benefit. Whether data is determined as "useless" it can easily be deleted and written off as "no value." Integrating dark data with existing data can prove to be significant data for business intelligence further adding good value to measuring. It is a good point in this article and to assess data accordingly
2015 Visual Analytics Interactive RoadshowSAS(r) experts are coming to a city near you in a series of live, interactive workshops focused on SAS Visual Analytics, including how to prepare your data for VA, the integration of VA with Office Analytics and a Visual Statistics demo.
January 22: King of Prussia, PA
February 24: Austin, TX
March 26: Redwood City, CA
April 22: NYC, NY (1st of 2 stops)
May 13: Seattle, WA
June 18: Minneapolis, MN
July 21: Rockville, MD
August 18: Chicago, IL
September 24: Irvine, CA
October 9: Cary, NC (during SAS Championship)
October 21: NYC, NY (2nd of 2 stops)
November 17: Orlando, FL
December 8: Atlanta, GA
LEADERS FROM THE BUSINESS AND IT COMMUNITIES DUEL OVER CRITICAL TECHNOLOGY ISSUES
The Current Discussion
Visual Analytics: Who Carries the Onus? The Issue: Data visualization is an up-and-coming technology for businesses that want to deliver analytical results in a visual way, enabling analysts the ability to spot patterns more easily and business users to absorb the insight at a glance and better understand what questions to ask of the data. But does it make more sense to train everybody to handle the visualization mandate or bring on visualization expertise? Our experts are divided on the question. The Speakers: Hyoun Park, Principal Analyst, Nucleus Research; Jonathan Schwabish, US Economist & Data Visualizer
The hospitality industry gathers massive amounts of customer data, and mining that data effectively can yield tremendous results in terms of improved CRM, better-targeted marketing spend, and more efficient back-end processes. Roger Ares, vice president of analytics at Hyatt Corp., discusses the ways he and his staff use big data.
Charged with keeping track of travel assets, including employees, iJET International relies on data management best-practices and advanced analytics to keep its clients in the know on current and potential world events affecting travel, Rich Murnane, Director of Enterprise Data Operations & Data Architect, told All Analytics in an interview from the 2014 SAS Global Forum Executive Conference.
Jason Dorsey, chief strategy officer for the Center for Generational Kinetics and keynote speaker at last month's SAS Global Forum 2014, describes how Gen Y professionals are enhancing the makeup of multigenerational analytics organizations.
From analytics talent development to the power of visual analytics, All Analytics found a variety of common themes circulating throughout the exhibition floor and session discussions at the 2014 SAS Global Forum and SAS Global Forum Executive Conference events held last month in Washington, DC.
Talking with All Analytics live from the 2014 SAS Global Forum Executive Conference, Eric Helmer, senior manager of campaign design and execution for T-Mobile, discussed the importance of customer data -- starting internally -- in devising the mobile operator's marketing plans.
The big-data analytics market can be a confusing place. Among the vendors vying for your dollars are traditional database management providers, Hadoop startup services, and IT giants. In this video, All Analytics editors Beth Schultz and Michael Steinhart sit down in a Google+ Hangout on Air with Doug Henschen, executive editor of InformationWeek. Henschen discusses use cases for big-data analytics, purchase considerations, and his recent roundup of the top 16 big-data analytics platforms.
At the National Retail Federation BIG Show last month, All Analytics executive editor Michael Steinhart noted a host of solutions for tracking and analyzing customer activity in retail stores. From Bluetooth beacons to RFID tags to NFC connections to video analytics, retailers must find the right combination of tools to help optimize the shopper experience, streamline operations, and boost revenues.
The days when historical shipment trends and gut feelings were enough to forecast retail demand accurately are long over. SAS chief industry consultant Charles Chase outlines the benefits of pulling real-time sales information from point-of-sale and product scanner systems, then flowing that data into dynamic forecasting tools from SAS.
With today's advanced visual analytics tools, you can stream data into memory for real-time processing, provide users the ability to explore and manipulate the data, and bring your data to life for the business.
Dynamic data visualizations let analysts and business users interact with the data, changing variables or drilling down into data points, and see results in a flash. Advance your use of data visualization with tools that support features like auto-charting, explanatory pop-ups, and mobile sharing.
No doubt your enterprise is amassing loads of data for fact-based decision-making. Hand in hand with all that data comes big computational requirements. Can traditional IT infrastructure handle the increasing number and complexity of your analytical work? Probably not, which is why you need a backend rethink. Big data calls for a high-performance analytics infrastructure, as Fern Halper, a partner at the IT consulting and research firm, Hurwitz & Associates, discusses here.
Redbox's bright-red DVD kiosks are all but ubiquitous these days, located in more than 28,000 spots across the country. Jayson Tipp, Redbox VP of Analytics and CRM, provides an insider's look at how the company has accomplished its phenomenal nine-year growth.
InterContinental Hotels Group (IHG), a seven-brand global hotelier, has woven analytics into the fabric of its operations. David Schmitt, director of performance strategy and planning, shares IHG's analytics story and his lessons learned.