With so much data in the world, it’s time to figure out how much of it is unstructured -- that which a human needs to look at in order to understand it best -- and what to do about it.
IT research firm IDC has estimated 7.9 zettabytes of digital data in the world by 2015, and I think the biggest chunk of it will come from social media generated on mobile platforms and driven from email. Intel estimates that at least 2.5 billion people will be online by 2015, generating more and more data and requiring more resources for storing and processing the data, as reported in InformationWeek. Such outlooks have led evangelist analysts to gush over unstructured data's potential; Google's Avinash Kaushik, for example, publicly claimed to have “orgasms over big, unstructured data.”
How to structure unstructured data
Many of us are really only getting started with unstructured data, looking for ways to get started and trying to figure out how best to handle it all. Actually, we need to ask ourselves if we should even bother to try working with it, as many previous attempts to add structure robotically have been disappointing to say the least, and fail at least much of the time. After all, dealing with and automating processes around structured data is tough enough!
Here I've put together a few things you can do with or relative to unstructured data:
Distribute the data in the cloud -- just store more of it and hope you can see useful patterns in the data with advanced big-data analytics and predictive analytics platforms.
Develop more powerful analytics engines to analyze the data, most of which will be in the cloud, in real time
Merge as much data as you can into large data files, a lesson learned by Team Obama in preparing for the 2012 election recently; merging several different databases and cleaning the data made developing predictions and gleaning insights easier.
Clean the data -- this assumes unstructured data is dirty, or not useful for analysis in its current state. You can purge duplicate information, ensure consistency in the naming of entities, and empty and sparse datasets, for example. Consider checking out Saleforce Data.com's Social Key, which ties customer data records to social media accounts and online content by those accounts. Perhaps Salesforce is on to something here; the cost of cleaning data might be shared, as the data a company is able to clean for its own use also goes back (the part that can be shared) to the overall Data.com repository in Salesforce’s cloud.
Working with unstructured data won't be easy -- but it will be necessary.
What advice do you have for working with unstructured data? Share below.
webmetricsguru, - One big challenge i see is analytics of sentiment oriented data even though the area has grown a great deal in the course of this year -- new apps and all. This kind of data at least for now may still need to be reviewed closer because its difficult to automate it in one given pattern without locking out new sentiments that are outside of that original set. However as intelligent systems learn this data, it will cut down what we have to look at even in sentiment analytics.
If I understand you correctly, that's the bane of the PR / Marcom industry - that you can actually look at bunch of verbatim (maybe that's ok for 10-30) but what happens when you have thousands or more?
I think a discussion on just what cleaning data is and how to to best do it would be good for AllAnalytics.com personally. I'd like to see what we come up with, and I bet a lot of others would too.
"If we can find the essential information or pattern, we might not need to look at most of it"
I see. I suppose that those hand-written patterns can just be domain specific and will be difficult to generalize. I agree that extracting the most useful patterns might be enough in most cases, as it difficult to think of all possible patterns. One of drawback of such model is that human patterns are often low-recall, even if precision is high.
It is true that some of the points you mentioned are debatable - like "Distribute the data in the cloud". But they are valid points to take into account when dealing with unstructured data. To the question how to clean unstructured data? I think that it depends on the shape and the model that has been defined.
What I meant is that currently, people usually end up needing to look at the data to understand it (because it is un structured information) and attempts to use software to understand it, in my opinion, won't work, at least not today.
What you can do, I think, and maybe our friends here can confirm or argue this, is cut down on what we have to look at.
If we can find the essential information or pattern, we might not need to look at most of it - and hopefully the software created can help surface that information, and maybe that's the best we can hope for (big data hype or not).
At any rate, this is an interesting discussion and I don't have all the answers - but I am wondering just what they are.
"I define it as something a human needs to look at to fully process"
I still don't get it. Do you mean that there is the need for human intervention to figure out whether the data is unstructured or not? Won't that be time consuming and practically impossible for human to go through all the instances of the data due to it size? Maybe it is not what you mean?
"So I guess, we need to find better search and organizing processes."
I think that is what the cleaning and storage processes are all about. Some data can fit in many categories depending on the search parameters. This may complicate the storage process as the same data will be duplicated - sometimes unnecessary.
Diego Klabjan, chair of the INFORMS University Analytics Program Committee and program director for Northwestern University's Master of Science in Analytics program, gives his advice for figuring out where to get an advanced analytics degree.
What Works: Open Source Analytics Software International Institute for Analytics WebinarOn Wednesday, Sept. 24, join IIA CEO and Co-Founder Jack Phillips, along with featured guest Gary Spakes, as we explore the five modernization stages that analytics hardware/software have experienced. We will discuss the considerations when calculating total cost of ownership of the analytics ecosystem.
2014 VA Interactive Roadshow -- Cary, NCThe 2014 VA Interactive Roadshow will feature SAS® Data Management and SAS® Visual Analytics experts covering topics like prepping data for VA and VA integration with SAS® Office Analytics. This year's events will keep presentations at a minimum and focus on giving attendees hands-on exposure to the latest version of VA.
Essential Practice Skills for Analytics Professionals Drawing on best practices from the field, this INFORMS course helps analytics professionals add value from beginning to end: listening to clients, framing the central problem, scoping a project, defining metrics for success, creating a work plan, assembling data and expert sources, selecting modeling approaches, validating and verifying analytical results, communicating and presenting results to clients, driving organizational change, and assessing impact.
Analytics 2014 The Analytics 2014 Conference is a two-day, educational event for anyone who is serious about analytics. This annual event brings together hundreds of professionals, industry experts and leading researchers in the field of analytics. All Analytics members save $500 on conference fees by using promo code ACAA.
Premier Business Leadership Series 2014 The Premier Business Leadership Series is an exclusive event for senior executives and decision makers that focuses on solving the current issues that affect governments and businesses globally. The Series is a unique learning and networking experience focused on the most innovative leadership strategies and analytic solutions for competing in todayâ€™s global economy.
2014 VA Interactive Roadshow -- BostonThe 2014 VA Interactive Roadshow will feature SAS® Data Management and SAS® Visual Analytics experts covering topics like prepping data for VA and VA integration with SAS® Office Analytics. This year's events will keep presentations at a minimum and focus on giving attendees hands-on exposure to the latest version of VA.
Data Exploration & Visualization Get hands-on training that focuses on the critical steps in the process of analyzing data: accessing and extracting data, cleaning and preparing data, exploring and visualizing data. This INFORMS course will use several of the most popular software tools intensively, and provide an overview of the range of software options.
Foundations of Modern Predictive Analytics In this INFORMS course, learn about modern predictive analytics, the science of discovering and exploiting complex data relationships. This course will give participants hands-on practice in handling real data types, real business problems and practical methods for delivering business-useful results.
2014 VA Interactive Roadshow -- AtlantaThe 2014 VA Interactive Roadshow will feature SAS® Data Management and SAS® Visual Analytics experts covering topics like prepping data for VA and VA integration with SAS® Office Analytics. This year's events will keep presentations at a minimum and focus on giving attendees hands-on exposure to the latest version of VA.
LEADERS FROM THE BUSINESS AND IT COMMUNITIES DUEL OVER CRITICAL TECHNOLOGY ISSUES
The Current Discussion
Visual Analytics: Who Carries the Onus? The Issue: Data visualization is an up-and-coming technology for businesses that want to deliver analytical results in a visual way, enabling analysts the ability to spot patterns more easily and business users to absorb the insight at a glance and better understand what questions to ask of the data. But does it make more sense to train everybody to handle the visualization mandate or bring on visualization expertise? Our experts are divided on the question. The Speakers: Hyoun Park, Principal Analyst, Nucleus Research; Jonathan Schwabish, US Economist & Data Visualizer
The hospitality industry gathers massive amounts of customer data, and mining that data effectively can yield tremendous results in terms of improved CRM, better-targeted marketing spend, and more efficient back-end processes. Roger Ares, vice president of analytics at Hyatt Corp., discusses the ways he and his staff use big data.
Charged with keeping track of travel assets, including employees, iJET International relies on data management best-practices and advanced analytics to keep its clients in the know on current and potential world events affecting travel, Rich Murnane, Director of Enterprise Data Operations & Data Architect, told All Analytics in an interview from the 2014 SAS Global Forum Executive Conference.
Jason Dorsey, chief strategy officer for the Center for Generational Kinetics and keynote speaker at last month's SAS Global Forum 2014, describes how Gen Y professionals are enhancing the makeup of multigenerational analytics organizations.
From analytics talent development to the power of visual analytics, All Analytics found a variety of common themes circulating throughout the exhibition floor and session discussions at the 2014 SAS Global Forum and SAS Global Forum Executive Conference events held last month in Washington, DC.
Talking with All Analytics live from the 2014 SAS Global Forum Executive Conference, Eric Helmer, senior manager of campaign design and execution for T-Mobile, discussed the importance of customer data -- starting internally -- in devising the mobile operator's marketing plans.
The big-data analytics market can be a confusing place. Among the vendors vying for your dollars are traditional database management providers, Hadoop startup services, and IT giants. In this video, All Analytics editors Beth Schultz and Michael Steinhart sit down in a Google+ Hangout on Air with Doug Henschen, executive editor of InformationWeek. Henschen discusses use cases for big-data analytics, purchase considerations, and his recent roundup of the top 16 big-data analytics platforms.
At the National Retail Federation BIG Show last month, All Analytics executive editor Michael Steinhart noted a host of solutions for tracking and analyzing customer activity in retail stores. From Bluetooth beacons to RFID tags to NFC connections to video analytics, retailers must find the right combination of tools to help optimize the shopper experience, streamline operations, and boost revenues.
The days when historical shipment trends and gut feelings were enough to forecast retail demand accurately are long over. SAS chief industry consultant Charles Chase outlines the benefits of pulling real-time sales information from point-of-sale and product scanner systems, then flowing that data into dynamic forecasting tools from SAS.
With today's advanced visual analytics tools, you can stream data into memory for real-time processing, provide users the ability to explore and manipulate the data, and bring your data to life for the business.
Dynamic data visualizations let analysts and business users interact with the data, changing variables or drilling down into data points, and see results in a flash. Advance your use of data visualization with tools that support features like auto-charting, explanatory pop-ups, and mobile sharing.
No doubt your enterprise is amassing loads of data for fact-based decision-making. Hand in hand with all that data comes big computational requirements. Can traditional IT infrastructure handle the increasing number and complexity of your analytical work? Probably not, which is why you need a backend rethink. Big data calls for a high-performance analytics infrastructure, as Fern Halper, a partner at the IT consulting and research firm, Hurwitz & Associates, discusses here.
Redbox's bright-red DVD kiosks are all but ubiquitous these days, located in more than 28,000 spots across the country. Jayson Tipp, Redbox VP of Analytics and CRM, provides an insider's look at how the company has accomplished its phenomenal nine-year growth.
InterContinental Hotels Group (IHG), a seven-brand global hotelier, has woven analytics into the fabric of its operations. David Schmitt, director of performance strategy and planning, shares IHG's analytics story and his lessons learned.