Wal-Mart Dreams Up Insight Innovation


With the holiday shopping season in full swing, a lot of little elves are closely watching how Wal-Mart, the biggest of all big-box retailers, has priced favorite "Dear Santa" items and other goodies. But have you ever stopped to wonder what Wal-Mart might have on its Christmas wish list?

I'll give you a hint: big-data. No, make that ginormous-data.

If Wal-Mart were to compile a Christmas wish list, the world's largest product repository would likely top it off. The company is intent on building such a database itself, so I'd venture to guess it wouldn't pass up a sprinkling of magic from the sugarplum fairies.

This would be no ordinary product repository, either (as if you could possibly have thought otherwise). Wal-Mart would like it to include detailed information on literally every product in the known world, Digvjay Lamba, a distinguished architect with Walmart Labs, told attendees at the Predictive Analytics Innovation Summit held by The Innovation Enterprise this month in Chicago. What's the product called? What retailers are selling it? How much are they charging? What are people saying about it in the social sphere? These are all questions Wal-Mart would like to answer from this single information source.

For perspective, Lamba took us, not forward to Christmas, but back to Halloween, the second-largest commercial holiday, with $10 billion in gross merchandise sales. "With 9 billion pieces of candy and 100 million pumpkins sold each year, it's a massive, massive festival," he said. "But it's also a big festival for big-data." Consider these stats he shared: 1 billion transactions and 1 billion visits to Halloween-related Websites every year, 100 million customers, 100 million tweets, 20 million check-ins, 1 million YouTube videos, 50 million images on Instagram, and 1 billion pages on Google if you search for Halloween 2012. "As time has gone along, the amount of data on Halloween has grown exponentially."

In the pre-big-data world of, say, 2007, a retailer like Wal-Mart would prepare for the Halloween shopping season by looking at historical costume sales and doing some market trend analysis. "All the data we were using was our own data, our own sales data, and some survey data." That has obviously changed. "We've got all this extra data outside of Wal-Mart telling us what's going on. We've got people tweeting and uploading videos -- and these can help us drive more insights on what we should really be selling for Halloween… and when we should start selling."

Answering the key question "Can we predict what is going to sell this year?" takes that massive product repository plus a combination of domain expertise and data science smarts, says Lamba, who joined Wal-Mart last year when it acquired Kosmix, a search and social media analytics platform provider (Wal-Mart's own sugarplum fairy, perhaps). It's all about what Walmart Labs calls its Social Genome Platform -- a system for coming up with unexpected insights. The platform comprises five essential nodes, each of which is a taxonomy in itself: products, people, locations, events, and interests.

We build these taxonomies, and then when any data comes in -- transactions, social status updates, images and videos, blogs and Web updates, check-ins and locations -- we mark them with what they're talking about on this taxonomy. So we take this unstructured data, and we annotate it so we know what it's really talking about in terms of these taxonomies that we've built so we can give structure to the data.

Then comes the most interesting part, using the tagged, annotated data (whether that is transaction data or social data) to build a dashboard for product managers and business users so they, not the data scientists, can come up with the insights and ideas. From a dashboard, for example, a business manager could see that zombie parties are trending in San Francisco, while handmade costumes are popular in New York City. "The fact that event themes can drive what we sell in different areas doesn't have to be something that the data scientist has to come up with. The business owners can drill down into these verticals across different dimensions and see what's out there."

Walmart Labs' sample (and simplistic) Social Genome Platform dashboard.
Walmart Labs' sample (and simplistic) Social Genome Platform dashboard.

Walmart Labs is well on its way to making this sort of scenario a reality. But Lamba says it's not easy. Besides the technology, of course, there are people issues. Not every domain expert wants to or should be working with big-data. But identifying those who do, and can, means more power to the business -- and that's good no matter what holiday season is upon us.

Beth Schultz, Editor in Chief

Beth Schultz has more than two decades of experience as an IT writer and editor.  Most recently, she brought her expertise to bear writing thought-provoking editorial and marketing materials on a variety of technology topics for leading IT publications and industry players.  Previously, she oversaw multimedia content development, writing and editing for special feature packages at Network World. In particular, she focused on advanced IT technology and its impact on business users and in so doing became a thought leader on the revolutionary changes remaking the corporate datacenter and enterprise IT architecture. Beth has a keen ability to identify business and technology trends, developing expertise through in-depth analysis and early adopter case studies. Over the years, she has earned more than a dozen national and regional editorial excellence awards for special issues from American Business Media, American Society of Business Press Editors, Folio.net, and others.

Midmarket Companies: Bring on the Big Data

The "big" in big data is no reflection of the size of the organization embracing its potential.

Push Yourself to New Analytical Discoveries

Take inspiration from Christopher Columbus as you pursue your analytical journeys.


Re: Faceted search
  • 11/28/2012 8:25:51 PM
NO RATINGS

@Beth: When ten years ago I met the concept of faceted navigation in Marti Hearst's papers, I have been impressed by the similarity of faceted navigation with classical multidimensional OLAP approach, as analyzed in many accademic papers (for example, www.siam.org/proceedings/datamining/2009/dm09_103_zhangd.pdf or nadav.harel.org.il/papers/p33-ben-yitzhak.pdf ).
But I have always been puzzled by the difficulty for this concept to gain ground in the Enterprise, where only numerical and structured data were analized in OLAP cubes, while faceted search was, at best, used by information architects to navigate intranet pages; missing completely, apart few experiences, the potentiality of analyze contextually numerical and textual informations.
Maybe now is the right time, as the WalMart project suggests, when social media data are approaching the volume of traditional numerical data.
Recent examples, http://iswc2011.semanticweb.org/fileadmin/iswc/Papers/Research_Paper/12/70310001.pdf, testify the interest in the concept, but the approach is still quite traditional.
I haven't personal experiences of large scale projects geared toward real-time facets extraction from social streaming, but I suppose that good canditates for an implementation stack could be Kafka, Storm, Lucene and Bobo.

Re: Faceted search
  • 11/28/2012 12:52:01 PM
NO RATINGS

@Andrea Scarso, thanks for jumping into the conversation. Let me ask, where have you seen "faceted search meets social big data" on a smaller scale? (I'm not implying it hasn't been, just curious if you can point us to some cool examples.)

 

Re: Trendsetter or follower?
  • 11/28/2012 7:26:34 AM
NO RATINGS

Agreed, the niche suppliers are going to be on very shaky ground.  I know someone who has done business with WalMart in the past, it's been tough to get on the shelves in the first place for a long time.  I can only see that getting more difficult when they are looking at trending products more closely.

Faceted search
  • 11/27/2012 6:05:56 PM
NO RATINGS

Faceted search meets social big data.

Interesting idea. Not new, but never executed at Wal-Mart scale.

Re: Trendsetter or follower?
  • 11/27/2012 1:03:48 PM
NO RATINGS

I also think small businesses that have managed to gain shelf space at WalMart will have a potentially shorter window of opportunity to prove themselves in sales.  I can see some losing shelfspace because Wal Mart determined that the vendor's product did not meet sales targets.  I am not fully familiar with the process, but that's my first immediate thought. I am open to other thoughts on this.

Re: Trendsetter or follower?
  • 11/27/2012 8:48:13 AM
NO RATINGS

@Callmebob, agreed, although WalMart has been better at collecting data for a long time this new platform seems to be turning them toward competing more closely with small specialty shops that stock items becasue customers asked for them or it's what the operator is "into".  This lets WalMart be as flexible as the smaller shops which was one of their last saving graces.  

Re: Trendsetter or follower?
  • 11/26/2012 3:33:47 PM
NO RATINGS

@callmebob--I do admit the word "monopolistic" comes to mind sometimes when I hear of some of the advanced big-data analytics work retailers and others are doing. 

Re: Trendsetter or follower?
  • 11/26/2012 3:28:28 PM
NO RATINGS

@Beth - Absolutely. But with it, the platform makes the competitive hill against WalMart even steeper.

Re: Trendsetter or follower?
  • 11/26/2012 2:28:25 PM
NO RATINGS

@callmebob, true. But wouldn't you say that even without the Social Genome Platform Wal-Mart already dominates the small fry retailer on data analytics anyways?

Trendsetter or follower?
  • 11/26/2012 2:22:44 PM
NO RATINGS

If small retailers didn't have enough to worry about, WalMart Labs with their Social Genome Platform can dominate them on the big data analysis front. Their dashboard gives their business managers the ability to whittle down their merchandising assumptions and increase their home run capabilities. The small fry retailer won't have that luxury and find it easier to be a trend follower if they can identify it. And following WalMart does not constitute a winning business formula.

INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +