Wanted: statistical soothsayers. The job title “data scientist” has emerged in recent years to encapsulate the unique mix of statistical, software engineering, and business analytics expertise required to find the predictive insights companies are all looking for.

As healthcare, retail, advertising, and finance companies have amassed larger and larger troves of data, the number of experts who are able to manage and mine it has diminished proportionately.

“There’s simply not enough of them,” said Dan Vesset, an analyst with IDC. “And not everyone can afford them.”

The field is so new and the education cycle so slow, it will be several years before schools can graduate enough candidates with the right skills to meet demand.

But even for the self-made data scientists working today, the numbers skills alone are not enough. Data scientists must also have extraordinary communication skills and a keen understanding of business goals in order to ask the right questions of the data, said DJ Patil, a data scientist-in-residence at Greylock, the venture capital firm.

“When the CEO has a gut feeling about a business trend, filtering that through the data, then explaining results that defy the boss’s instinct requires both a superior understanding of the data and a delicate sense of tact,” Mr Vesset said. The two rarely come in one person.

Even accomplished data scientists face an internal struggle.

Josh Wills, senior director of data science at Cloudera, said the difficulty was melding the two very different mindsets of software engineering and statistics.

“The aspect of a data scientist that is a hacker is relentlessly optimistic. They believe they can do anything with a computer,” he said. “But a good statistician is in many ways a pessimist who is very concerned about what I can really say I know from this data or what’s a valid inference. It’s a schizophrenic mix.”

The value of data scientists is immense. In the financial world, and the digital advertising business, a good data scientist can crunch data and tweak algorithms that bring in millions of dollars of revenue for a company. Companies such as Google and Facebook are in bidding wars over such candidates.

In areas such as healthcare, hospitals cannot compete with Silicon Valley or Wall Street on salary. They must rely on the promise of preventing illness and saving lives to attract data scientists, said Steve Fihn, director of analytics at the US Veterans Health Administration. “They’re just happy they’re not making CDOs,” he said.

Copyright The Financial Times Limited 2024. All rights reserved.
Reuse this content (opens in new window) CommentsJump to comments section

Follow the topics in this article

Comments