Number crunchers seek patterns in data

Applying brute computing power to the task of predicting future movements in financial markets is a dream that has been around almost as long as computers themselves. With a combination of big data and the latest advanced algorithms, according to some experts, that moment may finally be close.

Behind this confidence lies a belief that if you only knew how to look hard enough, patterns would appear in seemingly random markets.

“It’s not completely random and unpredictable: there are regularities in financial markets,” says Tom Mitchell, head of machine learning at Carnegie Mellon University, home to one of the US’s most respected computer science departments.

Mr Mitchell is one of those now applying cutting edge science to making trading profits – in his case through Meta Alpha, a start-up that sells its services to specialist commodity traders.

The firm’s system draws on “thousands and thousands of variables” covering a “vast array of financial market and macroeconomic data,” says Victor Long, chief executive. The algorithms used to crunch this ocean of data in search of predictable patterns are enhanced through machine learning, through which they are automatically fine-tuned based on results.

Included in the streams of data tapped by the algorithms are unstructured sources drawn from the web. An important indicator for Apple’s stock, for instance, might be “what people are saying on their blogs about the iPad mini”, said Mr Mitchell.

It is through “never-ending learning systems” like these, which become smarter the more information they are able to crunch, that the real benefits of big data will be felt, says Mr Mitchell.

In one example of machine learning, algorithms written by CMU have been crawling the web continuously for nearly three years, seeking to parse progressively more of the information as they get better at “understanding” it. “It gets a little bit better every day. It’s a system we never plan to turn off,” he says.