Well Jeanne, I think we're just about out of time. I want to thank you for sharing your fantastic insight. I also want to thank all of our participants, vocal or not. And, I'd like to invite folks to take an MIT SMR survey on how they're dealing with the big-data deluge. You can do that here
If we have an app that can help us address very specific questions from massive amounts of unstructured data, we have something that could be valuable. I've been imagining more general data mining efforts.
This is more the kind of "killer app" thing I've had in mind. The app developer would need to look over samples of the data, detect possible patterns for value, agree on a strategy with the organization, and then write an app that would yield valuable results. Hopefully, the results would not just add to the growing mountain of junk data...
Top companies know how to determine what has business value. They demand a strong business case for every project and they check at every milepost whether that business value case is still solid and then they check afterwards if they got the value. It helps them decide what is and isn't worth pursuing. This is a learning process that will be valuable in the analytics space.
MicroStrategy has been doing a lot of work in the Social Analytics space. Their "Wisdom" app allow you to mine Facebook data and analyze social trends. They have some interesting examples, such as determining marketing opportunities based on "Likes" across many people, pages, and products.
@BethSchultz, there's room for competitors to leapfrog the top firms just by implementing a more comprehensive data function company-wide. There's no limit on tech advances and just one misstep can change the leaderboard. Do RIM & Nokia have the best analytics in their industry? I think not.
I think the companies that are findings lots of value in analytics have lots of transaction data, because it affords so many opportunities for analysis. So i expect that retailers like Wal-Mart, Amazon and Target are ahead of most companies in most industries.
Nice example Lyndon. But different from some of the data mining that we were talking about, I think. If we have an app that can help us address very specific questions from massive amounts of unstructured data, we have something that could be valuable. I've been imagining more general data mining efforts.
Jeanne, a lot of companies seem to be spending lots of effort in understanding what's being said on the social networks about them and their brands, so a particular type of unstructured data. Given your description of sacred data and what you've said about unstructured data, I have to ask how you feel about trying to glean sentiments and what-not from social data ....
I think the top companies in every sector have gone well beyond simple analytics and obvious transactional reporting. They'll have dozens of analytical models covering all aspects of their business and will have initiatives to analyze new data sets, regardless of their (un)structure, source or size.
Beth asks about my experience with old data. Well, yes. Transit agency historical records from the founding of the agency. Needed to solve problems, answer critics. Unfortunately, almost entirely paper (yuk). But absolutely essential in answering key questions 15-20 years later. Hard to sort through, some of it still in my brain. Nice killer app would be great, one that could sift through boxes of paper...
But, the same thing could happen with Digital Age records. You know, all those docs entered in WordStar and Lotus 1-2-3?
There has been a lot of discussion of structured versus unstructured data, and the consensus is out on the value of optimizing toward heavy undirected analysis of unstructured data. So what does it mean to get better at structuring data? That seems far more subtle and dynamically proactive than simply defining tags /labels / object models extensible to future data sets. What will real progress look like there? Will it be tied more closely to enterprise needs?
Analytics models ARE strategic assets!. A few weeks ago the NYTimes had a great article about Target using their vast data to pre-market to pregnant women...some of whom hadn't even acknowledge publicly that they were pregnant yet. That's a powerful capability, and one that Target was most interested in protecting.
Sacred data is the transaction data that helps you know your customers, products, suppliers, and that has obvious value when analyzed. I think if we start there instead of looking for big, glitzy things, we'll learn how to drive value and then we can get a bit fancier.
@PredictableChaos, I think that "strong IT person" who knows analytics needs to be embedded in the business and report to the business. I've seen a lot of talent locked up on the IT side that could be generating a ton more business value if they worked directly for the front-line business leaders.
@egerter - For data analytics to work requires a strong IT person together with someone from the buisiness side who is willing to work at explaining and re-explaining what reports, what information would be most helpful. Where i've seen this work, there is a lot of back and forth to get to the summary reports that are actually helpful.
I questions whether storing data until we figure out how to use it means we are storing data that will be irrelevant by the time we analyze it. Someone once told me that NASA hadn't kept some old records on how they build early rockets--they thought that was a problem. I can't imagine we lost anything valuable there.
Jeanne, it sourt of sounds like you're suggesting organizations implement a strategy for evaluating and storing their data. Good idea. But a lot of the time, the data are pouring in, and it's maybe years before somebody looks are figures out how it should have been organized.
@Beth, I would try to get value from data analytics and accompanying technologies by identifying one business process that my company thought was critical to long-term success. Then i'd ask waht data it depends on, and then i'd get good operational data, and then i'd get analysts of all kinds to understand what is going on.
IT needs data curators, people who understand how to capture, store, document and disseminate data sets. Getting these data sets in front of business analysts is key to getting value from them. Having IT do analysis themselves is a losing scenario; they don't know the business as it stands today and their analytics tend toward simple counts and correlations. The business side (at the department level) has the field of vision to understand current challenges and will need data (properly analyzed) to support their budget requests.
We have been thinking that things like semantic webs etc were going to solve the lack of business focus and discipline for a long time. I can't imagine an app that simply "finds" the value in all our data that is lying around. We still need to know what we're trying to do.
Jeanne, given all the problems IT/enterprise data architects/analysts, etc. have in dealing with the data companies have today, what's your advice for how to approach the data of tomorrow -- the more complex and unstructured data increasingly gathered. Do they start from scratch? Bring in new technologies like Hadoop? Or build off of what they have in place today?
Oh, OK. So i think there's a gradual building of capabilities. Companies aren't likely to use predictive analytics well until they have good "habits" around business processes and customer relationships. You can analyze anything but whether you get good output or can make changes based on that output assumes some level of organizational maturity around business processes.
I keep thinking there must be some kind of opportunity for analytics whizkids to think up killer apps to sift through all that "just in case" data...
Amazing what you can glean from data, even decades old, if you have some purpose and identify the pattern for extracting value...
Jeanne, well, I can't find James' "official" definition of a DMS but in essence he's talking about a next-generation information management system that combines and applies predictive analytics, optimization and intelligent business rules to help companies automate certain types of decisions.
I have studied enterprise architecture for a long time and i noted the huge potential for business value from business process standardization. But many companies don't get nearly as much value as they think they will after they put in their ERP or CRM because they don't reassign responsibilities. So often the problem is that they never clean up or use the data that these systems make possible. So i've started studying data.
Safe, meaning they dispose of old data on a regular basis due to regulatory or legal concerns, or limit access to data due to concerns about theft or disclosure. Companies that can navigate these concerns and keep their historical record intact have a higher potential for learning and competitive advantage.
I'm trying to imagine a formalized data management system. There are some decision rules that we can build into automated systems--and it's good to keep people simply as oversight for when quirky things happen. Then there are people decisions that rely on rules, but i'm not sure they are formalized. Am i understanding you?
@egerter: re your point, "companies that value "safe" over innovation are risking being blind-sided by more technology-adept competitors that do keep all of their data and find ways to extract value from it" -- I'd argue that they risk being blindsided period, even by those companies that aren't keeping all their data!
Ah, the ideal analytics employee is a number of people. For the Big Data issues like engineering control systems or supply chains, get experts who understand statistics, logic, and biological systems. You also need data architects, and domain experts. It takes a team. They all need communications skills.
I don't think we "accidentally" get value fromd data that happens to be around. We get value from knowing what customer intimacy or efficiency or whatever goals will make a difference and then finding specific data that helps us understand what we've got.
The thing about data is that we tend to think the big opportunity is data mining and analytics. I really think it's more operational than that. Get data into the hands of people who make daily operating decisions, arm them with good business rules, and you can see a big business impact.
@Lyndon, I can tell you from experience in a public sector position that a good bit of data is routinely deleted. I can also tell you volumes of it are still stashed away - often in paper form.
@Jeanne, do many companies still have vast archives of paper documents?
Jeanne, in the article you talk about IT being a good data steward and that business needs to take a leadership position here. Where does the analytics team — sometimes positioned within IT, sometimes in the business, and sometimes independent — fit into all of this?
Re: differences between public and private sector ... Public sector agencies are under more public and political pressure to reveal organizational info. They tend to get into more trouble if they trash Emails from 25 years ago, than a private business. So, just wondering about aspects like that...
I think @danmeier and @egerter are spot on. And cheaper storage will just make it worse, as big-data companies keep promising that you will discover the long-lost secret to instant customer satisfaction if you just have enough data and ask the right question. Kind of astrology.
Working for a financial industry, data retention policies have been very clear. So it has been easy to decide when to indefinitely store, 7 yrs, 10yrs, etc. However, in other I worked for "just in case" was killing system performance if not creating liability.
Difficult for an IT department somewhat divorced from the business side to make the determination that a particular data set is not "worth keeping". With cheap storage (1 PB = $100K), it's easier to just store everything and maybe someone someday will find something of value.
IT doesn't have the business knowledge to decide what data is relevant, and the business users don't have the time or inclination to analyze what they really need. The result: "store everything just in case." They figure it out as they go along. The problem is that the business users never go back to review the data they're storing that they've found they don't need.
We are starting to see the creation of data management departments but their mission isn't yet clear. I would recommend starting with a workflow department. First figure out who is using what data for what and then you can manage the data and the flows.
Yes, i think people increasingly believe that storing too much creates a legal risk, but getting rid of anything that a company is legally required to keep is also an issue. Most lawyers know what is necessary; i don't think most people in their companies do.
A "just in case" item would be extensive transaction records--just in case we want to analyze who is buying what (10 years ago) or massive amounts of email because we can't figure out legally whether we're better off with too much or too little.
@Jeanne, when you say IT has "little interest in taking the time or risk to identify data that is expendable" that smacks of the age-old problem of having a disconnect between IT and the business. Does the arrival of "big-data" exacerbate that problem, in your opinion?
That seems to be a selling point for the cloud vendors Jeanne - cheaper IT than do-it-yourself. That should lead to a value return to the companies, but maybe it really is just the first inning, and no one is really sure how the game will play out. But they still have to play it.
The management practice most often mentioned was just better tiering and archiving. There is little interest in taking the time or risk to identify data that is expendable. So IT units are just trying to minimize the cost without losing anything.
Jeanne -- that is a big issue -- meaning, saving data "just in case" -- especially as the volumes keep swelling so. I can't imagine coming to an answer is easy to do. What did some of the folks you studied have to say about how to come to a resolution?
I love the question about the cloud impact. I don't actually think the cloud will have an impact on the value question. I think that companies manage and use data well--or not. The cloud is another tool for storing and accessing. Would you expect an impact, Rodney?
Mark your calendars for an e-chat here on Thursday, June 28, at 4 p.m. ET. We'll be talking with Jeanne Ross, director of MIT Sloan School of Management's Center for Information Systems Research, on how to find value in the information explosion.
At the CISR, Jeanne directs and conducts academic research that targets the challenges of senior-level executives at the center's more than 80 global sponsor companies. Besides her work at the CISR, Jeanne also teaches in MIT Sloan's Transforming Your Business Through IT Executive Education Course.
Join in the conversation!
A2: ALL ANALYTICS ACADEMYVIEW CLASSES +
There are no easy answers when it comes to data privacy, but there are some best practices, some hard rules that can't be violated, some technical considerations that help, and some ways to understand the right and wrong of managing data. This series will guide you and your organization around some of the pitfalls you could face on the path to success.
ANALYTICS IN ACTION
INFOGRAPHICSVIEW ALL +
- by James M. Connolly