There is a curious irony in information technology. The main role of an IT department is to understand and automate the intricate processes that keep an organisation working. And yet IT departments are, historically, not good at understanding their own processes. Indeed, the old cliche about the poor state of the shoes of the cobbler’s children is often mentioned when talking about IT development processes.

The well-publicised delays and failures of large-scale government projects are only the tip of the iceberg. Every organisation, no matter how adept, has a story to tell about an IT project that was either late or abandoned. The recent suspension of work on the multimillion pound generic clearing system for the London finance markets is another high-profile example of the failure of an important IT project. According to a report in Computer Weekly, the clearing house LCH.Clearnet, which was formed by a merger between the London Clearing House and its French counterpart Clearnet in 2003, called in Accenture last month to try to rescue a three-year project, which was supposed to go live in October 2004.

Despite several decades of experience and many improvements in software design and production techniques, IT development is still difficult. Better control over IT development and a long-term business view of the role of IT are currently seen as the most promising solutions. Business alignment, good project planning, achievable goals, realistic deadlines and meaningful measurement of results all contribute to what is being called IT governance.

IT governance has risen up the business agenda, partly because of paranoia about the wider impact of corporate governance regulations, such as Sarbanes-Oxley. But it is not only regulation that is driving change. There is also a growing frustration among senior executives that IT goes wrong more often than it goes right.

“Compliance is a main driver because it is putting IT projects under the spotlight. But at the same time, business wants to see a return on its investment in IT. The business is saying it cannot keep spending without seeing improvements,” says Andrew Barstow, a partner in the technology risk group at Ernst Young.

The scale of many development programmes adds to the problem: “Some of these developments cost tens of millions and even large organisations are realising that they do not have the experience and skills to manage such large projects.” He goes on to say that business and IT must work more closely together: “There is no such thing as a pure IT programme – it is about business change and IT is just one of the things you need to consider.”

Recent research by Accenture, based on interviews with 300 executives in the UK and Ireland, identified two main concerns over IT development projects. Andrew Morlet, head of the strategic IT effectiveness group at Accenture, explains: “First, there was a strong consensus that IT spending is going up and the alignment between business and IT is improving. But second, there is still a belief that IT is not delivering against investment.”

He goes on to say that good IT governance is the key to continued improvements in the effectiveness of IT: “When you see that good governance and the right processes are in place, you see better alignment between business and IT. The best companies see it this way. The trick is in how you manage business as usual while, at the same time, improving productivity and getting a smoother flow of development.”

Mr Barstow also sees a significant change in the management culture of IT, which is helping to improve governance over IT development: “We are seeing more professional management in IT now. The first wave of professionally trained managers are reaching positions of responsibility and they have the business acumen to see why the business needs to get real value out of IT investment.”

The growing tendency to put large-scale development programmes in the charge of a senior business executive rather than an IT technologist is, he says, also helping to bring IT development under proper control.

Accenture’s survey also found that the more successful organisations spend less on IT overall – but a higher proportion of their IT budget is genuine investment in new systems. “Generally speaking, around 60-65 per cent of IT cost is ‘non-discretionary’ – to keep the business going. It is how you spend the rest that matters,” notes Mr Morlet.

Many of the qualities needed for proper IT governance in development are, essentially, basic common sense. Project planning and system design must be inclusive to ensure that the development is always tied to business goals. Budgets and deadlines must be realistic. And every part of the project must be closely monitored by a representative “steering committee”.

“The skills you require will vary by project. But the project governance committee should always include the IT management and a business sponsor. They need to track events as they take place and ensure the resources and skills are available,” explains Mr Morlet.

Automation has a strong role to play and a wide range of software “tools” is now available to track the progress of a development. Even the humble spreadsheet has a role to play – and is often the first step to a more advanced project management regime: “In a simple project, you could use an Excel spreadsheet,” says Mr Barstow, “as long as you have ways to recognise priorities and understand the impact of development programmes on each other. In the end, tools are good – but only to help manage the process.”

Vendors of specialist tools acknowledge that their products should be used in a broader context. “IT governance is a combination of software tools and a change of attitude by IT,” says Ayman Gabarin, EMEA vice-president of IT governance at Compuware, a supplier of software tools.

“IT has traditionally been an ‘order taker’,” he continues. “The business requests a change or a new application and IT responds with a quotation. But now IT has to be more proactive. IT governance is about doing things better and treating the business as premium customers.”

Mr Gabarin says tools such as Compuware’s Changepoint can help IT departments improve their performance by monitoring change programmes more effectively. “The tools enable IT to see what if scenarios, so that the impact of delays is visible. And [managers] can also see what resources are being used and what skills are available.”

Automated tools can help in other ways. Roger Gilheany, UK IT governance business development director at Mercury Interactive, another supplier of software tools, says tools can help the IT department present its case to the board more effectively: “IT has got to show the value that it delivers. If it has a proper governance framework supported by automated tools, it gives [the department] visibility at board level.”

He goes on to say, however, that automated tools should not stifle the natural – and important – creative talents of the IT department: “You have to bring some form of order to the development side and tools can help to do that. But you have to keep that creative environment as well.”

A combination of cultural change and advanced tools are the “good shoes” that enable IT departments to reduce the possibility of failure. The time for them to take a walk into a more effective role as business enabler is long overdue.