When political pollsters get it wrong

From left: Freddie Sayers, Stephan Shakespeare and Anthony Wells at YouGov’s London offices

I’m sitting at a table in a glass-walled room at the headquarters of the market research company, YouGov. Facing me are Stephan Shakespeare, chief executive, Anthony Wells, head of political polling, and Freddie Sayers, editor-in-chief.

We’re talking about the general election. We all know what happened in the election. Consumers preferred the blue brand by a significant margin. But the pollsters had calculated that consumers would like the red brand just as much.

I’m here because I’d like to know what went wrong with the market research. Shakespeare, Wells and Sayers would also like to know what went wrong. As yet, they don’t know. They are looking into it. They’re a little sheepish but, nevertheless, cheerful and welcoming.

“We have to be slightly nervous about what went wrong and how to fix it, because that is an ongoing investigation,” says Shakespeare, an imposing 58-year-old with a bold striped shirt and slicked-back silver hair. This year, he was named in Debrett’s as one of the 20 most influential people in politics, even though his company is mostly concerned with consumer brands. Santander, the Prudential and the online food ordering company Just Eat are among its clients.

Shakespeare has the air of a man who is used to being right. After all, he was right in 2001, 2005, and 2010. “Obviously we got it wrong,” he says, of the most recent election. “There’s no arguing about that. We have to take it on the chin. But we got everything right until this year. This is the first time that we’ve been wrong.”

Of course, Shakespeare, Wells and Sayers might never know for sure why they went wrong. To poll an election, they take a sample of 1,700 people, scientifically selected to represent the population as a whole. “Scientists,” says Shakespeare, “that’s how we see ourselves.”

Wells, 37, who has the air of an academic, explains how they have perfected their sampling method, using statistical models. “Social class, age, gender, newspaper readership, which we do essentially to make sure the respondents aren’t too sophisticated, not just Guardian readers, but people who read the Star, the Sun and the Mirror.”

Having collected the sample, they ask questions, receive answers to these questions and make a projection. Five years ago, they were within a whisker. But not this time. Sayers, 33, who looks and sounds like a character from Made in Chelsea, and whose job it is to control YouGov’s enormous databank, says: “The number that we came out with was 34 per cent Labour, 34 per cent Conservative. The actual numbers were more like 31 and 37.” And somewhere inside those figures, inside that discrepancy, lurks a mystery, a story about the British political consumer that nobody, not even these experts, can understand.

Scientifically selected people, representing the population as a whole, said they were going to behave in a certain way. But the population as a whole behaved in a slightly different way. And this tells us something very important about the whole mindset of market research, an industry which, according to the Market Research Society, is worth around £2.5bn a year in Britain. Something is wrong with it. In this case, perhaps the sample wasn’t scientific enough. Or maybe the sampled consumers were not telling the truth about their real intentions. Or, most intriguingly, they might not have known their real intentions, because, a lot of the time, people’s real intentions do not exist in their conscious minds at all.

Ever since George Gallup started trying to call elections in America in the 1930s, market research has been about asking people questions. But what if asking people questions doesn’t work? In his post-election blog, Peter Kellner, the president of YouGov, tried to sum up why he believed that people in voting booths had reached for the blue brand in greater numbers than had been predicted. “They are ‘shy Tories’,” wrote Kellner, “not because they are unwilling to admit their choice of party to a stranger but because they really would like to support someone else but, faced with a ballot paper in the privacy of the polling booth, simply can’t.” In other words, these particular political consumers would have liked to choose the red brand but, upon serious consideration, they didn’t think the red brand was quite good enough and so, driven by logic, they reluctantly chose the blue brand.

Market research consultant Philip Graves is reading me Kellner’s blog over the phone. “They really would like to support someone else. But,” says Graves, reading Kellner’s words with precise diction, “faced with a ballot paper in the privacy of the polling booth, simply can’t.”

Graves says: “I’ve never met Peter Kellner, and it may be he is the most perceptive person in the universe. But I have a sneaking suspicion he has derived this knowledge from focus groups and surveys.” In other words, by asking people questions.

Graves has a very strong opinion about what’s wrong with the traditional methods of market research. He thinks they don’t work. He thinks that asking people how they think they’ll behave in the future, or why they behaved as they did in the past, is a hopeless way of actually predicting their behaviour, which means focus groups and surveys are misleading and dysfunctional. To Graves, the whole world of market research is about to change. And we’ll all look back at the way it was and shake our heads at how naive we used to be.

In his book Consumer.ology: The Truth about Consumers and the Psychology of Shopping, Graves writes: “The fundamental tenet of market research is that you can ask people questions and that what they tell you in response will be true. And yet . . . this is a largely baseless belief.” He says there is a simple, clear, obvious reason for this. It is that: “The unconscious mind is the real driver of consumer behaviour.” In other words, there’s no point in asking people what they want, or what they’re going to do, because they don’t know themselves.


Graves, 47, lives in a large Victorian villa outside Cambridge. There are electric gates and a gravel drive. As I arrive, his lawn is being trimmed by a robot mower. Inside the house, we sit down at his kitchen table. For the next five hours, he explains the central idea that underpins his work with clients such as the BBC, ITV, Virgin Media, Whirlpool and HSBC. The idea takes a while to get used to. It is that, while we think we know our own minds, we don’t. In fact, most of the time, we walk around like automatons.

Market research, Graves tells me, is based on “a fundamental belief in the power of asking people questions to understand what they think. And that is a profoundly flawed view. There are some things we can find out. But there really is not very much. And if you look at all the failed product launches, all the bad ideas and all the ideas that were slammed in research but ended up being successful, there is a pretty consistent picture that emerges. Asking people what they think . . . you really might as well be flipping a coin.”

In 1985, The Coca-Cola Company replaced its regular brand with a newer version that consumers had preferred in taste tests. In other words, when asked, consumers thought they liked New Coke, which was sweeter. But, later, when the new product became a reality, it bombed. Researchers had thought that consuming Coke was largely about the taste, something you could ask people about.

But Coke is about all sorts of things, many of them unavailable to the conscious mind: the effect of the logo on your emotions, the memories of summer evenings when you were a teenager, the shape of the bottle, the music from the ads.

A complex network of associations you have built up over the years. In other words, the brand. And the point about brands is that they don’t really exist in our conscious minds. Mostly, they exist somewhere out of reach of our conscious minds, a place we don’t have easy access to: what Graves calls the “adaptive unconscious”.

Graves gives me more examples. Using their conscious minds to respond to a Google survey, people expressed a desire for the company to provide more search results. But when Google tripled the number of results, its traffic actually declined.

Again, in their conscious minds, consumers of fast food wanted McDonald’s and Kentucky Fried Chicken to serve a range of healthier meals. But when McDonald’s launched the McLean burger, and KFC came up with the low-fat KFC Skinless, the products were not successful. Time and time again, says Graves, our conscious minds tell us what we want. But, without knowing it, we want something else.

“We are lousy witnesses to ourselves,” he says. “If you look at people’s inability to be empirical . . . they’ll go out and buy another exercise machine. They’ll join another gym. They’ll buy another smoothie maker. They won’t look in their cupboard and think, ‘I’ve got 16 household appliances that I don’t use.’ And then there are short-term contradictions. I watch people walk out of a pound store and go into a Starbucks. Clearly they’re meeting different needs, satisfying different psychological drives. But it makes no sense in any kind of narrative about people doing things for good reasons.”

But if you can’t ask people what makes them tick, what can you do? You can observe them, says Graves. One of his current projects is a new type of Innocent smoothie, with a mix of fruit and vegetable juices. If you want to know what sort of smoothie people want, don’t ask them about smoothies. Watch them as they walk around the supermarket. Track their sightlines. Notice how they work the shelves. Build a picture of what they do without knowing why they are doing it. Build a database.


Until comparatively recently, market research existed in a world of sparse data. People would ask you a few questions in a shopping mall. Or make a call to a randomly selected number and ask for a few minutes of your time. Questions were asked. Answers were given. Boxes were ticked. Yes, you drink Carling Black Label. At least you think you do. Yes, you like Spain as a holiday destination. That was how market researchers operated. They needed to project an air of confidence.

But now we live in a world of “big data”. Our behaviour is tracked and monitored and quantified through our gadgets and appliances — not just our phones, tablets and laptops but our TVs, our cars and the magnetic strips on our credit cards. Soon, our fridges, cookers, washing machines and even our toilets will be storing and relaying data about us. This is a world in which Google could predict the path of the 2009 US flu pandemic just by correlating billions of data points. Crucially, Google doesn’t ask questions. It is an observer. Like Philip Graves with his smoothies. The people at YouGov know this, of course. They are becoming observers of big data themselves. They have been building their own databank, which tracks the behaviour of 10,000 people who have agreed to let YouGov put cookies — spyware, if you like — into their devices.

While Google, Facebook and Amazon create their banks of data by offering free services and observing the behaviour of the people who use them, YouGov must pay for the privilege: its fee is £5. “And that means we can be much more Google-like,” says Shakespeare. He calls his new data set “the Cube” and the main point about it is that it is not a conscious entity. You can’t ask it questions. Instead, it generates correlations between one piece of data and another.

“That’s what’s exciting,” says Sayers. “What’s new is that these are very powerful computers. Beforehand, a researcher would have to design a long, complex survey and some poor person would have to have to go through it and analyse it. A computer can scan across huge numbers of data points that an individual can never look at. It gives you the answers to the questions before you even know what the questions are.”

Soon, perhaps, people like Shakespeare, Wells and Sayers won’t need to ask people questions about insurance, or what makes them use an app to order a pizza, or who they will be voting for.

“This gets a bit nerdy, and I’m sure there is a limit to this,” says Shakespeare, “but the more information you’ve got and the more sophisticated you can be with analysing it — I think that’s going to be the edge of one company over another.”

For a moment, he ponders the brave new world of market research, then says: “We don’t want to get to a creepy level.”

Photograph: Anna Huix


Letter in response to this article:

YouGov was wrong about the Scottish referendum / From Neil Woodcock

Copyright The Financial Times Limited 2017. All rights reserved. You may share using our article tools. Please don't cut articles from FT.com and redistribute by email or post to the web.