Adapt: Why Success Always Starts With Failure, by Tim Harford, Little, Brown, RRP£20, 320 pages

Man occupies an awkward midpoint between thoughtless insect and all-wise demigod. We like to imagine that we understand the complex, far-reaching consequences of our decisions but most of the time we just muddle through. Tim Harford’s terrific new book Adapt urges us to understand and profit from our muddling. Humanity has done and will do great things, not because we’re always right, but because our randomness generates plenty of gold amid the dross.

Harford, who writes for this newspaper, begins with an experiment in which the psychologist Philip Tetlock asked experts from many fields “to make specific, quantifiable forecasts – answering 27,450 of his questions between them – and then waited to see whether their forecasts came true”. The experts did better than random undergraduates, “but by any objective measure, they didn’t do well”. Harford pulls corporate chieftains into his march of folly by discussing the decline of once seemingly invincible businesses such as US Steel and the fact that “10 per cent of American companies disappear every year”. Yet even though the world’s complexity can overwhelm our limited brainpower, we are often saved by an evolutionary process through which “the market fumbles its way to success, as successful ideas take off and less successful ones die out”.

That perspective leads Harford to favour decentralised authority, not top-down command and control. He illustrates the problems of centralised bureaucracy with a discussion of the US military in Iraq before General David Petraeus assumed overall leadership of operations in 2007. In Harford’s telling, the Pentagon had tried to impose a uniformity that ignored local knowledge and crowded out both dissent and innovation. He understandably admires the military men, including Petraeus himself, who defied the centralisers and tried something different.

I share Harford’s enthusiasm for decentralisation and have long thought that attracting smart people and getting out of their way is the best strategy for academic management and urban economic development. Yet I wonder whether Adapt slightly undersells old-fashioned hierarchies. There is always a trade-off between the innovation and local knowledge created by decentralisation and the losses in co-ordination and control that this entails.

The military has a tough problem: getting people to do the extremely dangerous job of fighting. Soldiers often prefer to avoid battle, so they need to be prodded by officers, who in turn need to be prodded by their superiors. Our armies were built to get ordinary people to show up for warfare and that meant disciplined hierarchies.

One way to understand the military situation in Iraq is that the US Army had the correct, high level of centralisation for fighting a war, but that proved to be far too much centralisation for administering Iraq. Perhaps it is also true that many corporations today have the right level of centralisation for an era dominated by logistics and scale, but too much for an era dominated by innovation and creativity. Perhaps the right corporate models for today are examples of successful decentralisation such as Whole Foods and Google, the latter of which gives its engineers one-fifth of their time for their own projects.

These decentralised entities prosper by allowing for experimentation – and Harford spends 75 pages making the case for trying new things and testing them rigorously. He tells the story of the Spitfire – the superstar fighter plane of the Battle of Britain – which was built despite the conventional wisdom that “bombers could not be stopped” because one civil servant “decided to bypass the regular commissioning process and order the new plane as ‘a most interesting experiment’”. Harford argues compellingly that Air Commodore Henry Cave-Browne-Cave’s success reflected his willingness to try new things, not any ability to predict which things would succeed

Evaluation is the necessary counterpart of experimentation. If we are going to try new things, we need to rigorously measure whether they work. Harford earnestly advocates randomised trials pretty much everywhere. He correctly praises the “Randomistas” of development economics who are busy testing everything from anti-malaria interventions to placing cameras in classrooms to catch out teachers who fail to show up to work.

Harford’s picture of human folly and random success poses challenges for public policy, and he offers advice in the areas of innovation, climate change and financial market regulation. Within innovation policy, he sees the value of patents but also their downsides, particularly their stifling of competition. He likes the model of encouraging researchers to compete for rewards tied to a specific achievement, such as the 18th-century prize to solve the longitude problem or the modern Archon X Prize for Genomics.

His environmental views will be far more controversial. He has fun with a fictional well-meaning Green who tries to reduce his carbon emissions, but only manages to do more damage by eating local food, which requires the help of energy-intensive hothouses. He criticises ham-handed regulations such as the UK’s “Merton Rule”, which mandates the level of renewable energy capacity in new buildings. Like many economists, he favours a carbon tax that would prod the market into figuring out its own ways to reduce emissions.

Adapt explains the financial crisis as the result of overly interlocked systems in which failures can have catastrophic consequences. Failure will always be there, so wise regulations should “try to simplify and to decouple these high-risk systems as much as is feasible, to encourage whistleblowers to identify latent errors waiting to strike, and – sadly – to stand prepared for the worst”. This is all true, but since no one really knows how to “decouple” our financial systems, the risk continues.

Harford is a gifted writer whose prose courses swiftly and pleasurably. He has assembled a powerful combination of anecdotes and data to make a serious point: companies, governments and people must recognise the limits of their wisdom and embrace the muddling of mankind.

Edward Glaeser is a professor of economics at Harvard University and author of ‘Triumph of the City’ (Macmillan)

Copyright The Financial Times Limited 2017. All rights reserved. You may share using our article tools. Please don't copy articles from and redistribute by email or post to the web.