Recommender systems and diversity

Knowledge@Wharton recently published an article, "Reinforcing the Blockbuster Nature of Media: The Impact of Online Recommenders". The article discusses research work by Kartik Hosanagar and Dan Fleder at Wharton on whether recommender systems improve diversity of sales and help people discover items that otherwise might be buried in the long tail.

An excerpt from the article:

Recommenders -- perhaps the best known is Amazon's -- tend to drive consumers to concentrate their purchases among popular items rather than allow them to explore and buy whatever piques their curiosity, the two scholars suggest.

Hosanagar and Fleder argue that online recommenders "reinforce the blockbuster nature of media." And they warn that, by deploying standard designs, online retailers may be recreating the very phenomenon -- circumscribed media purchasing choices -- that some of them have bragged about helping consumers escape.

I am briefly quoted in the article arguing for a somewhat milder conclusion, saying:

Linden, reached via email, declares himself untroubled by Hosanagar and Fleder's findings. "Recommendation algorithms easily can be tuned to favor the back catalog -- the long tail -- as Netflix does," he argues. Netflix, the online DVD purveyor, consciously highlights obscure titles in designing its recommender.

Linden also argues that, in the absence of online recommenders, consumers would turn to even cruder tools, like traditional bestseller lists. "You have to ask what content would otherwise be in place of the recommendations and whether that content would have greater diversity," he says.

Hosangar and Fleder have two papers detailing their work, a very long paper titled "Blockbuster Culture's Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity" and a shorter ACM article, "Recommender systems and their impact on sales diversity".

The papers are an interesting read. What I found most surprising about their work was that, in their simulations, a recommender algorithm that did compensate for bestseller bias (called r4 in their paper) still reduced diversity. Although I had questions I had about their simulation model (which I already have discussed with Dan), I think their work should serve as an additional caution to those working on recommender systems to be concerned the impact choices in the algorithm can have on level of diversity, especially if one of the business goals of the recommendations is to drive movement in the back catalog.

Please also see comments on this research work from recommender researcher and U of Michigan Professor Paul Resnick, particularly his thoughts on the simulation framework used.