Big data is the term used to describe the huge volumes of data generated by traditional business activities and from new sources such as social media. Typical big data includes information from store point-of-sale terminals, bank ATMs, Facebook posts, and YouTube videos.
Companies use sophisticated software to analyse this data looking for hidden patterns, trends or other insights that they can use to better tailor their products and services to customers, anticipate demand or improve performance.
Haven’t companies been doing this for years?
Companies and other organisations, including governments, have for many years been collecting, and slowly sifting through so-called structured data. These are data, like a sales ledger, that are already well-organised and therefore relatively easy to process.
But more recently, there has been an explosion in the amount of ‘unstructured’ data like video clips or facebook posts. Their lack of an identifiable structure makes them much more difficult to analyse, but these information promise to provide the most valuable insights for a company. For example, Facebook posts could tell brands what consumers think of their products.
Is big data all that its cracked up to be?
That depends. Like most hot new technology trends, big data is going through a ‘hype cycle’. Initially people expect far too much, then disillusionment sets in, but eventually, the technology begins deliver real benefits as it matures.
Big data has been heavily overhyped and as a result, some companies have invested in the technology without a clear idea of how to use it or how it fits into their business strategies.
On the other hand, some companies, particularly early adopters in retail and financial services, are already reporting significant results from big data projects.
Who will make money out of big data?
Some companies have already used big data to disrupt existing industry models, but the main beneficiaries are likely to be the manufacturers of the computer systems and software that do the heavy-lifting analytical work.
These include established vendors like IBM, Oracle and SAP, together with a raft of startups including Alpine Data Labs, Splunk, and SumoLogic.
Other likely winners are companies that collect and supply the data – there is already a multibillion-dollar data broker industry.
Should I be worried about the privacy implications of Big Data?
Much of the most valuable big data is information about people and the digital trail we all leave when we go online.
By making connections between disparate snippets of information, big data can reveal far more about us than we ever intended. Inevitably that means the collection and use of big data is front and centre of the debate over privacy and the use of personal data.
In some senses however, the privacy genie is already out of the bottle. As a report on big data commissioned by the Obama administration last year noted: “the impact of Big Data has the potential to be as profound as the development of the Internet itself,” and that impact is likely to be felt by all of us.