Ian Campbell of Nucleus Research

One might expect Ian Campbell to be gleeful at last month’s $5.85bn takeover of Siebel Systems by Oracle, its enterprise IT rival.

Three years ago last month, Mr Campbell’s company, Massachusetts-based Nucleus Research, published a highly critical study of the use of Siebel software.

It found more than 61 per cent of “reference customers” it interviewed had not achieved a positive return on investment after more than two years, even though they were being promoted as success stories on Siebel’s website.

This, it said, contradicted the company’s claims about the positive ROI average customers were achieving.

Tom Siebel [the founder] was apoplectic. “It made for some good phone calls,” says Mr Campbell.

In fact Mr Campbell owes Mr Siebel a debt of gratitude. The latter’s angry public reaction and a similar response at Germany’s SAP following another study of March 2003 helped put Nucleus on the technology consultancy map.

Mr Campbell, a former research manager at IDC, the IT analysts, founded Nucleus in 2000, convinced of the need for quantitative assessment of investments, to complement the prevalent opinion-based approach.

“Everything we do has some sort of number at its core,” he says. Its main tool, the ROI assessment, is a “great, impartial measuring tool, if done correctly.”

In an IT context, the rigour and consistency of an ROI assessment has a particular resonance. “Some people will write whatever you want,” says Mr Campbell. And an inaccurate case study could result in a vendor losing its credibility with customers.

Also, he says, the strategic and market-related analysis from the main IT research companies such as Gartner and Forrester Research tends to be general, but ROI assessments have to be done on an individual basis to have any value.

Three years after the spats with Siebel and SAP, relationships have improved, he says. “Everyone now understands that we’re not out to get them, and we’re not going to lie to them.”

In a sign the industry accepts the need for impartial ROI studies, Nucleus works not only for customers but also for vendors, including Microsoft, Siebel and Oracle.

The respect Nucleus has won reflects the “rules of engagement” it established when it took on Siebel. “We follow a certain path. No vendor can say ‘Make sure the ROI is 200 per cent’,” says Mr Campbell.

Even so, Nucleus’s work can still give customers and vendors pause for thought. Its study of customers of Salesforce.com, the fast-growing customer relationship management (CRM) service, supplied on demand over the internet, showed that a third would switch if they could find a better and cheaper alternative.

“We quoted one customer who said it would take 30 minutes to switch to another provider. It’s not that Salesforce.com are doing poorly, but it needs to realise its tenuous hold on customers.”

In general, Nucleus’s ROI studies suggest hosted CRM services such as Salesforce.com work better for smaller customers. As companies get bigger, and want to build on top of their CRM and integrate it with other systems, they should be thinking whether to bring CRM in-house.

Nucleus has also looked at the ROI case for Venda, the hosted e-commerce service founded by UK serial entrepreneur Dan Wagner. “Does it make sense for anybody to create their own e-commerce system and support it, when they can outsource to Dan and pay a few thousand a month?” he asks.

He is much less enthusiastic about the ROI case for two of the most-hyped fads: radio frequency identification, and Voiceover Internet Protocol.

“I don’t think [tagging items with] RFID is a compelling ROI story for people other than a Wal-Mart,” he says. “When Wal-Mart [the world’s biggest retailer] moves from barcodes to tags it has positive ROI because of the high volumes, but very few companies fall into that category. Does Prada want an RFID tag on every one of its shoes?”

VoIP is “wonderful”, he says, but questions whether there is an ROI case for it. “Yes, if you don’t have anything right now, you’d buy VoIP. But if you already have something, does it make sense to yank it out? Not yet.”

Meanwhile, Mr Campbell refuses to gloat over Siebel’s takeover. “After our report there was a strong change in the way the company interacted with analysts, the press and customers,” he says. “I think they took what we said to heart.”