For investors who bought technology funds during the internet boom, only to see their value halve when the bubble burst, studying “behavioural finance”, the analysis of irrational investor behaviour, could pay big dividends.

Behavioural finance contrasts with the traditional argument that markets are “efficient” that is, share prices reflect publicly available information. According to this traditional school of thought, it is difficult to outwit the market as this means getting information before everyone else.

But a growing number of academics are starting to disagree with this long-held view. They insist that markets are not efficient because of the psychological flaws of the average investor. They say private investors are particularly prone to mental biases which can lead them to lose money unnecessarily.

They preach the behavioural finance argument that share and bond prices can, at times, bear little relationship to the underlying fundamentals because investors are sometimes irrational.

This academic work is influencing City analysts, too. James Montier, an equity analyst at Dresdner Kleinwort Wasserstein, an investment bank, says in a research note that to understand markets, you must understand the psychology behind them.

If you support this contrarian view, you can already buy offshore funds which aim to make money by applying these behavioural finance ideas. And, ABN Amro Asset Management, a fund manager, plans to launch a European fund available to UK investors within months.

Behavioural finance experts say their philosophy can offer critical insights for private investors, who frequently fail to diversify their portfolios, trade too often or do not sell shares which have fallen.

Investors do these things for a number of psychological reasons. But often it is simply owing to irrational over-confidence, say behavioural finance advocates.

Partly because of rising awareness of the impact irrational investors have, the influence of behavioural finance is starting to enter the mainstream, academics argue. Many active fund managers implement the insights of behavioural finance in their decisions, when they try to exploit inefficiencies in markets.

Some academics point to the rapid rise of the hedge fund industry as evidence of irrational investor behaviour. “You could view the entire hedge fund industry as a response to behavioural finance, because it's there to exploit [market] inefficiencies,” says Chris Malloy, an assistant professor of finance at the London Business School.

ABN Amro Asset Management, a fund manager, has already launched a few funds designed to generate superior returns by applying the principles of behavioural finance. It has €500m (£347m) under management in these funds.

So far, however, ABN Amro's retail funds have underperformed, suggesting that it is difficult to second guess irrational investor behaviour. The fund manager says its Ratio Invest fund has underperformed the MSCI Europe index by 0.9 of a percentage point since its launch in April 1999. Underperformance is also seen in another fund; ABN Amro's Japanese behavioural finance fund fell by 20.8 per cent between March 2001 and January 2005, says Standard & Poor's, a data company. That was worse than the average for the offshore Japanese equity sector, which dropped by 15.6 cent.

One consultant suggests privately that funds that try to generate returns by applying the principles of behavioural finance incur such big trading costs that they perform badly.

But ABN Amro Asset Management says it does not trade unless it expects the benefits to outweigh the costs. It says the Japan fund had a “difficult time” because fixed costs had a disproportionate impact on the small portfolio. These charges have since been capped.

Some academics argue that frequent trading of stocks and shares is one of the biggest faults of irrational investors as their over-confidence leads them to trade too often in the belief that the benefits of the deal will outweigh trading costs.

Professor Nick Barberis, a professor of finance at Yale School of Management, says: “Don't trade too often. Retail investors would do a lot better if they traded less. Their trading takes a lot away from their net returns because of transaction costs [the money you pay to stockbrokers or fund managers to buy or sell shares and funds].”

A different psychological bias leads people to be reluctant to sell shares which fall. They want to avoid feeling regret which they would experience if they crystallise a loss. Research by Terence Odean of the University of California in Davis found that investors are 70 per cent more likely to sell winning shares than losing stocks. And the shares which investors keep, tend to underperform, while those they sell later outperform.

This research highlights what academics call “loss aversion”: the pain we feel when we lose money outweighs the pleasure we feel from an increase.

“For psychological reasons, people tend to hang on to prior losers. But this only worsens their portfolio performance because prior losers tend to keep losing,” says Professor Barberis, whose research has focussed on behavioural finance.

He adds that psychological factors stop investors from diversifying their portfolios enough. For example, they often invest only in domestic stock markets, or invest too heavily in their employers' shares although they already depend on employers for salaries and in many cases pensions as well.

Yet none of these trends necessarily means that it is easy to take money from irrational investors. The private investors, stockbrokers and fund managers who might hope to generate returns from irrational distortions can be prey to the same biases as anyone else.

One problem for fans of behavioural finance could crop up if they think share prices are overvalued or undervalued. They may believe other investors have traded irrationally but they cannot know how long that irrationality will last. Chris Malloy of London Business School says he would like to have sold US technology shares he did not own in March 2000, before buying them back at much lower prices and generating big profits. But if technology share prices had continued to rise, he would have lost out.

“I'd like to have shorted the Nasdaq in spring 2000, but there's a danger that everyone else does the opposite.

“The markets can be irrational longer than you can remain solvent,” he adds, citing the renowned economist John Maynard Keynes.

If you bought technology funds just before the information technology bubble burst, you will have already learned some expensive lessons.

Adopting some of the rules of behavioural finance could help you avoid such pain again. But being able to spot irrational investor behaviour does not necessarily mean that you can profit from it.

The old rules of keeping your portfolio well diversified and not betting too much of your money on the herd's favourite sector still applies.

Get alerts on Personal Finance when a new story is published

Copyright The Financial Times Limited 2018. All rights reserved.

Comments have not been enabled for this article.