Why Recommendation Engines Still Aren't Accurate


Recommendation engines are deeply embedded in American culture. Anyone who shops online, subscribes to a streaming media service, conducts on online search, or uses social media is encouraged to do something -- buy this, click on that, listen to this song, watch that movie. Sometimes the recommendations are accurate. Sometimes they're not.

For example, Google's search engine thinks I'm male. Netflix thinks I might enjoy comedies aimed at college-age men. Amazon's recommendations can be strange if not laughable.

"On some level, these algorithms are amazing, but the types of errors that can be made are foolish," said Patrick Wolfe, professor of statistics and honorary professor of computer cience at IEEE Signal Processing Society's Big Data Special Interest Group. "It probably wouldn't take a lot of data to teach Google you're not male."

Credit: Pixabay
Credit: Pixabay

Context is Everything

Google and Facebook continue to serve up ads for the fringed boots I bought at Macy's two weeks ago. They're also recommending the infant and toddler car seats I bought as a gift in the same time frame. Clearly, the recommendations lack appropriate context.

"One of the reasons I think recommendation engines aren't as accurate as they could be is that much of machine learning is about making predictions -- predicting the weather or whether a document is about politics or not," said Thorsten Joachims, a professor in the Department of Computer Science and in the Department of Information Science at Cornell University. "A recommendation engine has to be smart about the actions it takes."

Pandora serves up more music that I like than dislike, although my genre preferences, Native American flute music and smooth jazz, represent dramatically smaller universes of choices than classical music or rock and roll. Pandora doesn't tell me anything about local musicians or the local music scene. Granted, that would be an extremely difficult undertaking given the number of cities that consider their local music scene an integral part of their culture -- Chicago, Las Vegas, and Ithaca, New York, for example.

Ithaca College Associate Professor Doug Turnbull and Cornell's Joachims are tackling that very problem. With the help of their students, they developed MegsRadio, a music app that recommends local musicians and local music events. The app is specific to Ithaca's local music scene, although others could benefit from the work if they chose to do something similar in another city, Turnbull said.

I tried the app and created a smooth jazz station. The playlist included some of favorite smooth jazz artists and some very talented artists (presumably from Ithaca) with whom I was unfamiliar.

Solving the Problem Pragmatically

Turnbull and Joachims are using counterfactual machine learning to compensate for the fact that they lack a user base large enough to train the algorithms. Counterfactual learning considers the recommendations an algorithm made, users' historical choices, the accuracy of the recommendations made by the algorithm, and what would happen if a different algorithm were applied in the same circumstance.

"Counterfactual learning is saying that if we don't have new users but we do have what the old algorithm predicted and the probability of the thing it predicted and given what the user historically picked in the past, would this new algorithm be better?" said Turnbull. "It's like an alternate parallel universe."

By comparison, popular search engines, shopping sites, and streaming media companies use A/B testing on a small percentage of users to determine whether one algorithm performs better than another. If the new one performs better than the existing one, the new new algorithm is eventually rolled out to all users. Counterfactual learning enables offline research. Users can be brought in later for validation purposes.

"It's really making these systems much more like an agent that thinks about what the effects of its actions will be so that it can learn from past mistakes," said Cornell's Joachims.

What's Your Take on Recommendation Engines?

As more analytics become prescriptive, recommendation engines will find their way into more use cases. Are recommendation engines helping or hindering you? What improvements would you like to see? We'd love to continue the discussion in the comments section below.

Lisa Morgan, Freelance Writer

Lisa Morgan is a freelance writer who covers big data and BI for InformationWeek. She has contributed articles, reports, and other types of content to various publications and sites ranging from SD Times to the Economist Intelligent Unit. Frequent areas of coverage include big data, mobility, enterprise software, the cloud, software development, and emerging cultural issues affecting the C-suite.

How Analytics Are Changing Frequent Flyer Programs

Frequent flyer programs have evolved over the years. So has the amount of data available. Here's a look at where we stand today.

Why Advanced Analytics Is in Your Future

Companies doing analytics 20 years ago enjoyed a competitive advantage. Today, if you are not doing analytics, you are falling behind.


Re: Another vacuum?
  • 2/5/2017 9:11:11 PM
NO RATINGS

..

Jim writes


There's no excuse for that data gap. If Amazon and others sell you a product their marketing department should know to take you off the promotion list (except for related products), and if you return a product, maybe a note saying, "Sorry it didn't work out" would be a nice touch.

Unfortunately, a sales lead to marketers is like a bowl of popcorn. They just keep reaching for one handful after another. 


I fail to understand why these large online retailers don't just pay the bucks to overhaul their recommendation algorithms.

Instead of badgering you to re-purchase that product you just bought, wouldn't it pay off a lot more to have ads saying "Here are some accessories you might consider to go with your new purchase ..."

 

 

Re: Another vacuum?
  • 1/7/2017 4:52:19 PM
NO RATINGS

Yes, it does seems that so much of online advertising is just visual spam.  I wonder how many of us have trained our brains to just ignore the ads. Or maybe the marketers have found that what we think we're ignoring is really getting to the brain anyway as a sort of subliminal branding we'll pick up later when we actually want to buy something?

Re: Another vacuum?
  • 1/6/2017 9:14:54 AM
NO RATINGS

@Joe. The key difference is that banner ads and supermarket fliers never pretended to be well targeted at individuals. Facebook, Amazon, Google and others supposedly are "leveraging" data that they have gleaned from us. Old school results when they claim to be using new school techniques.

Re: Another vacuum?
  • 1/5/2017 7:13:08 PM
NO RATINGS

@Jim: As I think about it more, that's a really apt analogy, I think.  Digital is a must, and terribly targeted Facebook ads are just the new digital-age equivalent of those take-one flyers in those bins.

Of course, before Facebook ads, the digital-age equivalent, say, 10 years ago (and even more recently, yet) was old-school banner ads.

Re: Another vacuum?
  • 1/5/2017 8:34:05 AM
NO RATINGS

@Lyndon. I agree that it is annoying. In fact, I have had a couple of experiences lately where I had horrible experiences with online purchases -- one product was never delivered, a Christmas gift was too late, and one tech product was junk and simply went in the trash.

In each case I voiced my displeasure with the company and still kept getting promotions for the same product from the same suppliers, and in the case of the trashed tech product I was getting requests to help other buyers solve their tech problems.

There's no excuse for that data gap. If Amazon and others sell you a product their marketing department should know to take you off the promotion list (except for related products), and if you return a product, maybe a note saying, "Sorry it didn't work out" would be a nice touch.

Unfortunately, a sales lead to marketers is like a bowl of popcorn. They just keep reaching for one handful after another.

Re: Another vacuum?
  • 1/4/2017 9:00:41 PM
NO RATINGS

..

Jim writes


Crosschecking two lists is part of it when it comes to recommendations. If I browse looking for a certain item on Amazon and I buy it from Amazon, that site shouldn't keep recommending it. However, they might recommend related products or accessories.


 

Getting continuously and repeatedly barraged with ads for the exact same products I just bought from the same site is extremely irritating. On the other hand, recommendations for related products, accessories, etc. can be helpful.

Jim also writes


However, if I started my search on Google or another site and I bought from Amazon, Google and the various sponsors that it serves up won't know that I completed my purchase. My search history for shoes or a book via Google lives on and on and on.


 

Typically I shop on Amazon and I still get the ads from the same products, on Amazon, that I just bought. That's bad enough. However, if my browsing on other sites tends to trigger such ads, then something is also seriously wrong from a marketing standpoint. Why implement marketing algorithms that just serve to irritate the potential customer and cannot possibly lead to a purchase of the advertised product (which has already been purchased)? They need to fix this ...

..

Re: Another vacuum?
  • 1/4/2017 8:42:35 AM
NO RATINGS

@Lyndon. Crosschecking two lists is part of it when it comes to recommendations. If I browse looking for a certain item on Amazon and I buy it from Amazon, that site shouldn't keep recommending it. However, they might recommend related products or accessories.

However, if I started my search on Google or another site and I bought from Amazon, Google and the various sponsors that it serves up won't know that I completed my purchase. My search history for shoes or a book via Google lives on and on and on.

 

Re: Another vacuum?
  • 1/4/2017 8:30:48 AM
NO RATINGS

@Joe. I agree, Facebook doesn't seem to know what to do with the information that it gleans from our activity. However, something is working for them. Maybe they aren't charging sponsors a whole lot for an individual impression, so if the sponsor gets even a few clicks they are happy. FB might be touting "personalization" when it's really no more personal than the supermarket fliers that we take directly from the mailbox to the recycling bin. Just enough people use those to make the sponsor's investment worthwhile.

Re: Another vacuum?
  • 1/4/2017 5:35:42 AM
NO RATINGS

@T Sweeney: I've long suspected that, while Facebook may have a bunch of information about me, they are for the most part clueless as to what to do with it.

I base this on the fact that I've repeatedly gotten ads for woodworking and motorcycle repair courses.

Not exactly my cup of tea.

(So too with many other sites, which constantly hound me with ads to help me find a lawyer in my area.  Clearly they know that I use the words "lawyer" and "attorney" and various legal terms a lot in my profiles and correspondences, and are trying to do something with that information.  What they've missed?  That the reason I use those terms is because I'm a lawyer.)

Re: Or: ...Have Become Less Accurate
  • 1/4/2017 5:32:50 AM
NO RATINGS

@T Sweeney: I suspect you're right.  It's just extra obvious in the case of Netflix because their recommendation engine used to be so on the mark and so obviously catered to users' personal tastes.

Now?  It's just this "New for You" garbage -- presented in a far more oppressive and restrictive UI than formerly existed (under the guise of being "mobile friendly") -- that Netflix wants you to watch.

Page 1 / 5   >   >>
INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +