For the second time in a month – on capital gains tax and now on non-domiciled residents – Alistair Darling, chancellor, has beaten an embarrassing retreat after a fierce offensive led by wealthy individuals in the City. One U-turn was unfortunate; two suggests serious incompetence. Both operations were badly botched. But the generous tax treatment of non-doms is still a scandal requiring action.

Ordinary taxpayers are angry that rich people can use loopholes and expensive lawyers to avoid paying tax at up to 40 per cent like the rest of us. For a millionaire, 40 per cent is not a penal rate.

Super-rich oligarchs from Russia and India have used the system to buy expensive property with a tax obligation limited to £2,000 or so of council tax. Many have their homes in offshore companies to avoid stamp duty on sales.

For less exotic non-doms, there are several well-established tax avoidance dodges. One is the use of dual contracts for work in the UK and overseas with the overseas portion not taxed unless remitted. UK capital gains can also be moved offshore into overseas trusts since non-doms are not subject to the same tax avoidance rules as domiciled residents. Inheritance tax is voluntary for non-doms.

Gordon Brown, prime minister, promised to take action when in opposition. Then, after five years as chancellor, he argued in 2002 that current rules “do not reflect the reality of today’s more integrated world”. He argued that “it is generally accepted as fair that those with a long-term connection to a country owe a special oblig-ation to support the social structures of the state”. He was right. Mr Brown then dithered for five years, under pressure from Labour’s big donors.

The numbers (112,000 in 2004-05) have grown as long-standing residents realised that a tenuous overseas connection – an overseas parent – can save many thousands in tax. The government reluctantly acknowledged that it was losing £1bn-£1.5bn in tax revenue to non-doms.

By autumn last year there was mounting pressure for action even in the financial community. My Treasury team commissioned and published a YouGov poll showing that 70 per cent of the public wanted non-dom tax status cut back. The Conservatives saw their chance and put forward an annual poll tax of £25,000 to raise the £3.5bn they needed for a pre-election pledge to abolish inheritance tax. The government, in pre-election mode, essentially copied the Conservative policy, with a £30,000 fee for non-doms of seven years residency or more.

The poll taxes proposed by Mr Darling and George Osborne, Tory shadow chancellor, will not work. The fee is prohibitive for large numbers of non-doms with modest income or gains. Even for a banker on temporary secondment to the UK the charge is the equivalent of £50,000 a year extra in UK taxed income, which many employers will balk at paying. But, for Roman Abramovich and Lakshmi Mittal, the charge is the cost of a small party.

The City has mounted a counter-offensive. They have a point about some details added by the Treasury; for example, on whether liability for capital gains should apply to gains accruing before non-dom status. There are genuine problems for Americans falling foul of the double tax agreement.

But the apocalyptic predictions of mass exodus will not wash. Where will the tax refugees flee to? The established tax havens like Monaco or the Bahamas are not major international financial centres. Nor is Ireland, which offers non-dom status. Some cantons in Switzerland have long offered a favourable tax regime and will have already attracted those more interested in skiing than share dealing. Big centres competing with London – New York and Frankfurt, for example – recognise no such animal as a non-domiciled res-ident and any non-dom taking refuge in the US or Germany would pay full tax on their global income and gains.

In my view, the government should drop the Darling-Osborne poll tax on non-doms. It should, instead, restrict non-dom status to those who are not yet fully settled here – a seven-year cut-off seems reasonable. Short-term expatriates would not be affected. Others could plan their future with certainty. But a Labour government cannot run away from the issue altogether without losing all credibility and self respect. If it wants 60m Britons to take it seriously on tax avoidance and fairness in the tax system it must make non-doms pay up or pack up.