Don Putnam, designer of investment products since the 1970s, points to the most fundamental of lessons learnt from last year’s global financial implosion: markets are “defined by their participants as much as they are by their mechanics” and it is people’s motivations that ultimately count.

“The car is important but the driver is crucial – and a panic among the drivers, if you will, makes the technologies of roadways irrelevant,” says Mr Putnam, who runs Grail Partners, a boutique Boston investment bank.

To some, that conclusion may seem self-evident. But to accept it sweeps away assumptions that for half a century formed the foundations of the financial industry. The reigning theory, often referred to in shorthand as “efficient markets”, is deeply embedded in the way that markets operate. The regulations for pension funds and banks both ultimately hinge on these assumptions. So does much law on securities fraud.

It is central to business schools’ curriculum and is part of the Chartered Financial Analyst qualification that acts as a gateway to the investment profession. Fund managers run their businesses by comparing their performance against benchmark indices, another idea from efficient markets. The products based on derivatives that grew notorious for their role in the market crash all stemmed directly from efficient markets theory.

If the theory needs to be abandoned, the effect on investing will be profound. More important still is what will come to replace it. Efficient markets borrowed from mathematics but that is now widely regarded as an oversimplified and often downright misleading theory that fostered the cavalier confidence leading to the crash.

PENSIONS

Options to restore confidence in retirement

Pensions were already in crisis before markets tumbled last year. But that crash, which shook the assumptions on which pension investing had been based, for the last generation, has made the existing crisis problems much worse.

At issue was whether the long-running debate had been between final-salary or “defined benefit” pensions – where employers companies guarantee to pay a proportion of final salary as a pension and themselves take on the investment risk – themselves – had to give way to and “defined contribution” plans, in which the individual saver takes the risk. Defined contribution plans steadily took over as companies and governments could no longer afford the traditional guarantees.

But it was also increasingly obvious that leaving investment decisions up to individuals was a bad idea. The UK’s personal pensions scandal in the late 1980s, where many savers with excellent guaranteed pensions were tricked by salesmen into switching into inferior versions without guarantees, showed the dangers.

Experience in the US has also been telling. When savers were offered a range of choices, they tended to leave the bulk of their savings in cash or in company stock. Neither is seen as desirable.

Pension providers must also contend with the collapse in savers’ confidence following last year’s events. So a model for pensions in the future will revolve around improving, or reinventing, defined contribution plans. The new model may look quite like the old.

One predominant idea is to offer savers a clear-cut default option that will give them a reasonable trade-off of risk and return, while allowing them the opportunity to be more adventurous if they want. The problem now becomes to design a good default option, given that many ideas about the optimal way to allocate assets took a knock during the last year’s crisis.

Another possibility is may be to offer some guarantees. As a reward for a certain number of years of saving, savers might get could be given irrevocable bonuses. This would diminish their fear of periodic market crashes, even if it would not come close to the total guarantees of the defined benefits era.

Academics are now ransacking a range of other disciplines in the quest for a better understanding. That search has ranged from evolutionary biology through behavioural psychology to thermal dynamics and chaos theory. None is likely to deliver answers as clean and simple as those that came from efficient markets.

“We’ll get a new theory,” says Paul Trickett, European head of Watson Wyatt, one of the largest consultants to pension funds. “This is a godsend for academic study. That’s great. Long live the people who can turn their minds to this thing. The challenge is really positive. But I don’t think that what we see means we can throw away the existing theory.”

Modern portfolio theory, developed over the past 50 years in academia – particularly at the University of Chicago and Massachusetts Institute of Technology – has all been based on the common premise that market prices at all times attempt rationally to incorporate all known information.

From this came the ideas that drove countless investment decisions: that the pattern of returns in financial markets follows the normal “bell curve” distribution often observed in natural sciences, that risk can be defined by the extent to which securities prices vary around their mean, and that observing how they have moved in relation to each other in the past allows a precise and measurable trade-off between risk and return.

Among the most important corollaries to flow from such notions: it became possible to put a precise price on derivatives, markets would be impossible to beat other than by luck (because they are efficiently priced, they would instead follow a “random walk” in reacting to each new piece of information), and equities would outperform other assets in the long term (because they carried a higher risk and would therefore generate a better return). Everyone knew that panics and bubbles recurred periodically and that markets were not always fully rational. But the argument was that these could be treated by rational investors as an opportunity to profit.

In spite of the weight that was put on the theory, it has long been known to have problems. First, market returns do not follow a “bell curve”. Instead, extreme events happen far more often than a “normal” distribution would imply. Benoit Mandelbrot, the mathematician who invented fractal geometry, proved this more than 40 years ago, when efficient markets theory was in its infancy, and has continued to criticise the established theory ever since.

If stocks really followed a bell curve, he observed, then a swing of more than 7 per cent in a day for the Dow Jones industrial average should happen once every 300,000 years. In fact there were 48 such days during the 20th century. “Truly, a calamitous era that insists on flaunting all predictions. Or, perhaps, our assumptions are wrong,” he concluded.

Another obvious weakness in efficient markets is the assumption that investors always make their decisions rationally. Virtually everyone knows this is not true. Indeed, over the past two decades a new discipline of behavioural economics had started to substitute findings from psychology for the assumption of rationality. But last year’s events inflicted fresh damage in the critical area of asset allocation – how to divide up an investment portfolio among broad asset classes such as stocks, bonds and commodities.

According to modern portfolio theory, there is an “efficient frontier” in which different assets can be mixed to maximise return for a given level of risk. This rests on the correlation among the assets – the extent to which they tend to move in line with each other – and their risk, which is defined as the amount they tend to vary or move away from their long-term average. This is common sense – add an asset to a portfolio that will move up when the others move down and you have made it less risky.

But it relies on these correlations being static. Investing using this formula – and borrowing money to do so – can be a recipe for disaster if the correlations change. And that is what happened last year. According to Jeremy Siegel of the University of Pennsylvania’s Wharton School, whose book Stocks for the Long Run was influential in persuading asset allocators to put a big weighting towards equities, says: “The most serious attack on efficient markets is the change in correlation of asset classes under extreme conditions.”

He points out that oil and equities traditionally move in opposite directions – higher oil prices would damage the stock market. Thus buying oil would provide a hedge against lower stock prices. But for the past two years oil and stocks have been closely correlated, meaning that adding oil to a stock portfolio in effect makes the same bet twice.

By the time the stock market hit bottom in March, even the price of gold – normally a “safe haven” that rises when equities fall – was declining. This calls into question the notion of gains from diversification. As Mr Siegel says, it might be possible to adapt the theory to account for changing correlations in times of high stress – but this would not only be complex but a “very different framework than what we teach in finance classes today”.

Attempts are also under way to adapt the theory using “career risk”. This corrects for the behaviour of fund managers, who are determined to avoid looking bad compared with their peers and will thus tend to move like a herd, into and out of the market. As they are doing so to safeguard their jobs, which is a rational motive, this maintains the possibility that they are behaving rationally.

Efficient markets theorists had added various “patches” as they found anomalies. For example, small companies, or those whose shares sell for a low multiple of their earnings, tend to outperform – and there is a “momentum effect” in which stocks that have been performing well continue to do so for a while.

Critics say all these efforts are a waste of time. Jeremy Grantham of Grantham Mayo Otterloo, a Boston-based fund manager, and a long-time critic of the efficient markets hypothesis, says academics will “patch theories until they finally collapse under the weight of all the patches”.

Mr Grantham suggests that the entire efficient markets framework should be jettisoned. Economists keep the idea, he says, because they are “incapable of dealing with theories that don’t give answers to two decimal points, and the only kind of theory that can deliver that is one based in complete rationality”. Mr Mandelbrot is also scathing. He says: “Such ad hoc fixes are medieval. They work around, rather than build from and explain, the contradictory evidence.”

The problem then becomes finding an alternative theory. Markets could be modelled using the the physics of wind and turbulence, says Mr Mandelbrot. But he admits that the mathematics is “frankly forbidding”, research has “barely begun” and reliable applications are “distant”.

Another new theory borrows from Charles Darwin. Andrew Lo of MIT has produced arguably the best-known idea, the adaptive markets hypothesis, by borrowing from evolutionary biology. His idea is that markets are “adaptive” and evolve over time. For long periods they will be stable – which explains how they appear to be efficient for much of the time – but this will be punctuated by periods of crisis when some species die out and others replace them.

He admits that his model does not have the same degree of mathematical rigour as the classic efficient markets approach, but says economists have been suffering from “physics envy” – wishing their models had the same mathematical precision.

While academics look for deeper models of markets that work, investors are left with a toolkit they have had since business school that suddenly seems not to work. It is unsettling. “I look at this process as being, in a sense, halfway done,” says Mr Putnam of Grail. “We’ve learnt a great deal but we haven’t learnt how to integrate and synthesise that into a way to go forward. So the process, the paralysis, the trauma, has really taken hold right at the moment.”