The algorithms using data to suggest your next purchase

Recommendation engines help Amazon to suggest books you might like, Linked­In to flag up jobs you could apply for and Facebook to suggest friends to contact. They also make proposals for the 125m hours of films and television that users of online streaming service Netflix watch every day.

As Carlos Gomez Uribe, Netflix’s vice-president of product innovation, and Neil Hunt, chief product officer, wrote this year in an academic paper: “Humans are surprisingly bad at choosing between many options, quickly getting overwhelmed and choosing ‘none of the above’ or making poor choices.”

They reckon that algorithmically generated recommendations account for about 80 per cent of hours streamed to Netflix subscribers, with the rest coming via search. By their estimates, recommendation engines that pique viewers’ interest and engage them with the service save the company about $1bn a year lost through customer churn. Netflix’s 2015 annual revenues were $6.8bn.

Companies selling products or services online have been using such techniques for a while, whether off-the-peg programs or tailor-made versions. What has changed is the science behind the recommendations, the sheer volume and variety of data fed into the engines and the sophistication of the algorithms used to analyse them. This means users are receiving more helpful suggestions and therefore buying more, says Sean Owen, director of data science at US-based data software company Cloudera.

“A sales uplift in the region of 20 per cent to 25 per cent is what a company like Amazon might be aiming for from its recommender systems,” he says. “That’s pretty consistent with what I hear from other ecommerce companies that do this well.”

Recommendation algorithms broadly fall into two categories. One is content-based filtering, driven by the attributes of items. If you view a comedy starring Seth Rogen, a content-based recommender will propose other films in which the actor appears.

The other kind, collaborative-based filtering, analyses user behaviour. It analyses data about users’ activities and engagement with items to cluster them into groups. If you buying a carrying crate for a puppy, you may soon receive a recommendation for a particular book on dog training or a certain harness because other people who bought the crate also bought that book or harness.

“Collaborative filtering tends to do a better job of presenting new ideas to the customers,” says Mr Owen. However, collaborative filtering also requires a great deal more data.

Ted Dunning, chief application architect at data software company MapR, advises any companies just starting to use recommendation techniques to avoid wasting time on implementing lots of complex algorithms before they understand what a few simple ones reveal. They should also be ready to maintain a quest for ever-better data to feed into the recommendation engine.

The approach a company takes will depend on the kind of item being sold. UK-based start-up Artfinder, an online art marketplace, introduced a recommendation engine this year and, on the basis that any artwork is unique, chose a collaborative filtering approach. It felt that suggestions based on similar items would not be as effective as basing them on algorithmic observations of how different groups of buyers responded to works by the same artist or belonging to similar genres.

Zalando, the Germany-based online fashion retailer, with annual sales of €3bn, uses both approaches, says Mikio Braun, who heads its recommendation and search operations. With collaborative filtering “we can derive similarities between items that might not be observed solely through product data,” he says. Such data used in content-based filtering will not reveal that customers think a certain top goes well with a certain skirt, but collaborative filtering, based on customer behaviour, can reveal they make a good outfit.

Overall, while companies may seek greater insights into online customers’ preferences, they should remember this may not always be welcome. As Jason Gordon, a consumer analytics expert at professional services firm Deloitte, says: “There’s a fine balance to be struck between relevance and creepiness.”