With the holiday shopping season in full swing, a lot of little elves are closely watching how Wal-Mart, the biggest of all big-box retailers, has priced favorite "Dear Santa" items and other goodies. But have you ever stopped to wonder what Wal-Mart might have on its Christmas wish list?
I'll give you a hint: big-data. No, make that ginormous-data.
If Wal-Mart were to compile a Christmas wish list, the world's largest product repository would likely top it off. The company is intent on building such a database itself, so I'd venture to guess it wouldn't pass up a sprinkling of magic from the sugarplum fairies.
This would be no ordinary product repository, either (as if you could possibly have thought otherwise). Wal-Mart would like it to include detailed information on literally every product in the known world, Digvjay Lamba, a distinguished architect with Walmart Labs, told attendees at the Predictive Analytics Innovation Summit held by The Innovation Enterprise this month in Chicago. What's the product called? What retailers are selling it? How much are they charging? What are people saying about it in the social sphere? These are all questions Wal-Mart would like to answer from this single information source.
For perspective, Lamba took us, not forward to Christmas, but back to Halloween, the second-largest commercial holiday, with $10 billion in gross merchandise sales. "With 9 billion pieces of candy and 100 million pumpkins sold each year, it's a massive, massive festival," he said. "But it's also a big festival for big-data." Consider these stats he shared: 1 billion transactions and 1 billion visits to Halloween-related Websites every year, 100 million customers, 100 million tweets, 20 million check-ins, 1 million YouTube videos, 50 million images on Instagram, and 1 billion pages on Google if you search for Halloween 2012. "As time has gone along, the amount of data on Halloween has grown exponentially."
In the pre-big-data world of, say, 2007, a retailer like Wal-Mart would prepare for the Halloween shopping season by looking at historical costume sales and doing some market trend analysis. "All the data we were using was our own data, our own sales data, and some survey data." That has obviously changed. "We've got all this extra data outside of Wal-Mart telling us what's going on. We've got people tweeting and uploading videos -- and these can help us drive more insights on what we should really be selling for Halloween and when we should start selling."
Answering the key question "Can we predict what is going to sell this year?" takes that massive product repository plus a combination of domain expertise and data science smarts, says Lamba, who joined Wal-Mart last year when it acquired Kosmix, a search and social media analytics platform provider (Wal-Mart's own sugarplum fairy, perhaps). It's all about what Walmart Labs calls its Social Genome Platform -- a system for coming up with unexpected insights. The platform comprises five essential nodes, each of which is a taxonomy in itself: products, people, locations, events, and interests.
We build these taxonomies, and then when any data comes in -- transactions, social status updates, images and videos, blogs and Web updates, check-ins and locations -- we mark them with what they're talking about on this taxonomy. So we take this unstructured data, and we annotate it so we know what it's really talking about in terms of these taxonomies that we've built so we can give structure to the data.
Then comes the most interesting part, using the tagged, annotated data (whether that is transaction data or social data) to build a dashboard for product managers and business users so they, not the data scientists, can come up with the insights and ideas. From a dashboard, for example, a business manager could see that zombie parties are trending in San Francisco, while handmade costumes are popular in New York City. "The fact that event themes can drive what we sell in different areas doesn't have to be something that the data scientist has to come up with. The business owners can drill down into these verticals across different dimensions and see what's out there."
Walmart Labs' sample (and simplistic) Social Genome Platform dashboard.
Walmart Labs is well on its way to making this sort of scenario a reality. But Lamba says it's not easy. Besides the technology, of course, there are people issues. Not every domain expert wants to or should be working with big-data. But identifying those who do, and can, means more power to the business -- and that's good no matter what holiday season is upon us.
@Beth: When ten years ago I met the concept of faceted navigation in Marti Hearst's papers, I have been impressed by the similarity of faceted navigation with classical multidimensional OLAP approach, as analyzed in many accademic papers (for example, www.siam.org/proceedings/datamining/2009/dm09_103_zhangd.pdf or nadav.harel.org.il/papers/p33-ben-yitzhak.pdf ). But I have always been puzzled by the difficulty for this concept to gain ground in the Enterprise, where only numerical and structured data were analized in OLAP cubes, while faceted search was, at best, used by information architects to navigate intranet pages; missing completely, apart few experiences, the potentiality of analyze contextually numerical and textual informations. Maybe now is the right time, as the WalMart project suggests, when social media data are approaching the volume of traditional numerical data. Recent examples, http://iswc2011.semanticweb.org/fileadmin/iswc/Papers/Research_Paper/12/70310001.pdf, testify the interest in the concept, but the approach is still quite traditional. I haven't personal experiences of large scale projects geared toward real-time facets extraction from social streaming, but I suppose that good canditates for an implementation stack could be Kafka, Storm, Lucene and Bobo.
@Andrea Scarso, thanks for jumping into the conversation. Let me ask, where have you seen "faceted search meets social big data" on a smaller scale? (I'm not implying it hasn't been, just curious if you can point us to some cool examples.)
Agreed, the niche suppliers are going to be on very shaky ground. I know someone who has done business with WalMart in the past, it's been tough to get on the shelves in the first place for a long time. I can only see that getting more difficult when they are looking at trending products more closely.
I also think small businesses that have managed to gain shelf space at WalMart will have a potentially shorter window of opportunity to prove themselves in sales. I can see some losing shelfspace because Wal Mart determined that the vendor's product did not meet sales targets. I am not fully familiar with the process, but that's my first immediate thought. I am open to other thoughts on this.
@Callmebob, agreed, although WalMart has been better at collecting data for a long time this new platform seems to be turning them toward competing more closely with small specialty shops that stock items becasue customers asked for them or it's what the operator is "into". This lets WalMart be as flexible as the smaller shops which was one of their last saving graces.
If small retailers didn't have enough to worry about, WalMart Labs with their Social Genome Platform can dominate them on the big data analysis front. Their dashboard gives their business managers the ability to whittle down their merchandising assumptions and increase their home run capabilities. The small fry retailer won't have that luxury and find it easier to be a trend follower if they can identify it. And following WalMart does not constitute a winning business formula.
Diego Klabjan, chair of the INFORMS University Analytics Program Committee and program director for Northwestern University's Master of Science in Analytics program, gives his advice for figuring out where to get an advanced analytics degree.
2014 VA Interactive Roadshow -- Cary, NCThe 2014 VA Interactive Roadshow will feature SASŪ Data Management and SASŪ Visual Analytics experts covering topics like prepping data for VA and VA integration with SASŪ Office Analytics. This year's events will keep presentations at a minimum and focus on giving attendees hands-on exposure to the latest version of VA.
Essential Practice Skills for Analytics Professionals Drawing on best practices from the field, this INFORMS course helps analytics professionals add value from beginning to end: listening to clients, framing the central problem, scoping a project, defining metrics for success, creating a work plan, assembling data and expert sources, selecting modeling approaches, validating and verifying analytical results, communicating and presenting results to clients, driving organizational change, and assessing impact.
Analytics 2014 The Analytics 2014 Conference is a two-day, educational event for anyone who is serious about analytics. This annual event brings together hundreds of professionals, industry experts and leading researchers in the field of analytics. All Analytics members save $500 on conference fees by using promo code ACAA.
Premier Business Leadership Series 2014 The Premier Business Leadership Series is an exclusive event for senior executives and decision makers that focuses on solving the current issues that affect governments and businesses globally. The Series is a unique learning and networking experience focused on the most innovative leadership strategies and analytic solutions for competing in todayâs global economy.
2014 VA Interactive Roadshow -- BostonThe 2014 VA Interactive Roadshow will feature SASŪ Data Management and SASŪ Visual Analytics experts covering topics like prepping data for VA and VA integration with SASŪ Office Analytics. This year's events will keep presentations at a minimum and focus on giving attendees hands-on exposure to the latest version of VA.
Data Exploration & Visualization Get hands-on training that focuses on the critical steps in the process of analyzing data: accessing and extracting data, cleaning and preparing data, exploring and visualizing data. This INFORMS course will use several of the most popular software tools intensively, and provide an overview of the range of software options.
Foundations of Modern Predictive Analytics In this INFORMS course, learn about modern predictive analytics, the science of discovering and exploiting complex data relationships. This course will give participants hands-on practice in handling real data types, real business problems and practical methods for delivering business-useful results.
2014 VA Interactive Roadshow -- AtlantaThe 2014 VA Interactive Roadshow will feature SASŪ Data Management and SASŪ Visual Analytics experts covering topics like prepping data for VA and VA integration with SASŪ Office Analytics. This year's events will keep presentations at a minimum and focus on giving attendees hands-on exposure to the latest version of VA.
LEADERS FROM THE BUSINESS AND IT COMMUNITIES DUEL OVER CRITICAL TECHNOLOGY ISSUES
The Current Discussion
Visual Analytics: Who Carries the Onus? The Issue: Data visualization is an up-and-coming technology for businesses that want to deliver analytical results in a visual way, enabling analysts the ability to spot patterns more easily and business users to absorb the insight at a glance and better understand what questions to ask of the data. But does it make more sense to train everybody to handle the visualization mandate or bring on visualization expertise? Our experts are divided on the question. The Speakers: Hyoun Park, Principal Analyst, Nucleus Research; Jonathan Schwabish, US Economist & Data Visualizer
The hospitality industry gathers massive amounts of customer data, and mining that data effectively can yield tremendous results in terms of improved CRM, better-targeted marketing spend, and more efficient back-end processes. Roger Ares, vice president of analytics at Hyatt Corp., discusses the ways he and his staff use big data.
Charged with keeping track of travel assets, including employees, iJET International relies on data management best-practices and advanced analytics to keep its clients in the know on current and potential world events affecting travel, Rich Murnane, Director of Enterprise Data Operations & Data Architect, told All Analytics in an interview from the 2014 SAS Global Forum Executive Conference.
Jason Dorsey, chief strategy officer for the Center for Generational Kinetics and keynote speaker at last month's SAS Global Forum 2014, describes how Gen Y professionals are enhancing the makeup of multigenerational analytics organizations.
From analytics talent development to the power of visual analytics, All Analytics found a variety of common themes circulating throughout the exhibition floor and session discussions at the 2014 SAS Global Forum and SAS Global Forum Executive Conference events held last month in Washington, DC.
Talking with All Analytics live from the 2014 SAS Global Forum Executive Conference, Eric Helmer, senior manager of campaign design and execution for T-Mobile, discussed the importance of customer data -- starting internally -- in devising the mobile operator's marketing plans.
The big-data analytics market can be a confusing place. Among the vendors vying for your dollars are traditional database management providers, Hadoop startup services, and IT giants. In this video, All Analytics editors Beth Schultz and Michael Steinhart sit down in a Google+ Hangout on Air with Doug Henschen, executive editor of InformationWeek. Henschen discusses use cases for big-data analytics, purchase considerations, and his recent roundup of the top 16 big-data analytics platforms.
At the National Retail Federation BIG Show last month, All Analytics executive editor Michael Steinhart noted a host of solutions for tracking and analyzing customer activity in retail stores. From Bluetooth beacons to RFID tags to NFC connections to video analytics, retailers must find the right combination of tools to help optimize the shopper experience, streamline operations, and boost revenues.
The days when historical shipment trends and gut feelings were enough to forecast retail demand accurately are long over. SAS chief industry consultant Charles Chase outlines the benefits of pulling real-time sales information from point-of-sale and product scanner systems, then flowing that data into dynamic forecasting tools from SAS.
With today's advanced visual analytics tools, you can stream data into memory for real-time processing, provide users the ability to explore and manipulate the data, and bring your data to life for the business.
Dynamic data visualizations let analysts and business users interact with the data, changing variables or drilling down into data points, and see results in a flash. Advance your use of data visualization with tools that support features like auto-charting, explanatory pop-ups, and mobile sharing.
No doubt your enterprise is amassing loads of data for fact-based decision-making. Hand in hand with all that data comes big computational requirements. Can traditional IT infrastructure handle the increasing number and complexity of your analytical work? Probably not, which is why you need a backend rethink. Big data calls for a high-performance analytics infrastructure, as Fern Halper, a partner at the IT consulting and research firm, Hurwitz & Associates, discusses here.
Redbox's bright-red DVD kiosks are all but ubiquitous these days, located in more than 28,000 spots across the country. Jayson Tipp, Redbox VP of Analytics and CRM, provides an insider's look at how the company has accomplished its phenomenal nine-year growth.
InterContinental Hotels Group (IHG), a seven-brand global hotelier, has woven analytics into the fabric of its operations. David Schmitt, director of performance strategy and planning, shares IHG's analytics story and his lessons learned.