How can we tell if stocks are too expensive? It is arguably the hardest question for long-term investors, but in recent years it had appeared that there was an answer.

As regular readers will know, the cyclically adjusted price/earnings ratio (Cape) signalled the biggest market turning points over the last century, and also correctly showed that the rally after the 2000 dotcom crash was not to be trusted (it ended in the great financial crisis). This measure compares prices with the average of earnings for the previous decade, smoothing out any manipulation of earnings, and accounting for the tendency of p/e ratios to shift according to the profit cycle.

Cape has two problems, however. One is that it has twice in the last decade failed to tell investors that stocks were cheap, ahead of rallies that lasted four years, in which stocks more than doubled. For fund managers and investors, this shakes their confidence. Cape might be right that stocks are historically expensive, they say, but you would still be better off buying them.

A second problem is that it currently suggests stocks are expensive – even though many shorter-term metrics suggest they are cheap. According to Yale University’s Robert Shiller, who popularised the measure, US stocks now sell for 23.8 times their 10-year earnings, compared with a long-term average of 16.5. This implies that returns for the next decade will be far below average. That is inconvenient for the big industry that is based on selling people stocks.

As a result, there is a concerted attempt to prove that Cape cannot be trusted. This might lead to useful fine-tuning of an important measure. But it could also delude investors into ignoring an important signal that stocks are overvalued.

Recently, Merrill Lynch’s Savita Subramanian produced a note looking at 16 different measures of the US stock market. All bar Cape suggested that stocks were either cheap or fair value. She said: “The most recent profits recession was the worst we have seen and was exacerbated by a high leverage ratio which has since been dramatically reduced. Assuming that this scenario is going to repeat itself is, we think, overly pessimistic.”

By Professor Shiller’s own data, which take into account inflation, the 92 per cent fall in real earnings over the 21 months from June 2007 was indeed the greatest fall over such a short period in US history.

However, earnings then enjoyed an equally historic rebound. From their nadir in March 2009 they rose twelvefold over the next three years.

How comparable are earnings over huge sweeps of history? The companies in the index change over time, as do accounting conventions, and tax. In 1900, US corporation tax was zero: 50 years later it was 40 per cent. This would affect the multiple that people would pay for companies’ earnings.

Volatility has been stoked by changing accounting standards, and by the practice of paying managers with stock options. This gives them an incentive to “throw out the kitchen sink” in quarters when earnings are bad, and overstate losses, so that subsequent profits will look all the better. A chunk of those losses came from a handful of financial groups – but they had all enjoyed overinflated profits for several years leading up to the crisis.

Look at the 10-year rolling average of real earnings and the volatility is put into context. The latest earnings dip no longer looks exceptional, or even very unusual – although it is the worst since the second world war. The bounce is also not unusual. Metrics based on one-year earnings during much of this period would have been useless: Cape makes far more sense.

It cannot be used well for timing, as once bubbles have become established they can reach levels of true insanity (as in 2000) before bursting. If interest rates are low and companies are disgorging cash at a record rate, as at present, stocks can stay expensive for a long time.

But still, why is Cape stubbornly saying stocks are expensive, when the p/e compared with last year’s earnings is below average?

Prof Shiller’s reply: “It diverges so much because earnings have shot up. The sense is that they are unsustainable. They grew at about 2 per cent per year over the last 150 years. When they suddenly shoot up, in the middle of a financial crisis, you wonder.”

He also defends Cape’s simplicity. “It does rely on earnings which are accountants’ efforts to summarise the data,” he admits, but many problems should average themselves out over 10 years. And while 10 years is an arbitrary measure, so is a one-year period. Investors are more likely to pay attention to a measure if it is simple and makes intuitive sense, like Cape.

The more criticisms are aimed at it, the more the waters are muddied. Is it perfect? No measure is, or can be. But it still appears to be better than anything else available. It would be wise to heed the signal it sends, inconvenient though that might be.

Copyright The Financial Times Limited 2018. All rights reserved.