September 27, 2013 8:00 pm
Not long ago, organisations convened focus groups to assess customer interest. Today, executives can ask their data specialists to find those insights in social sentiment, sales numbers, web site behavior, sensor data, and more.
In the next few years, the ability to find and assess trends, turn insight into foresight, to tap the behavior of multiple audiences, and to optimise decisions will move into the hands of the executives, their teams, and even individuals.
One of the recommendations that we will be making at the upcoming Gartner Symposium/ITxpo is that in the meantime, organisations must evaluate big data technologies with an eye on what’s possible and practical for seizing opportunities in the future and today.
Investment in analytics and the information management infrastructure to support it has been a top chief information officer investment priority for the past six years. Yet according to a recent survey, only 8 per cent of respondents say they have deployed a “big data” project to production.
Many big data deployments are still in the knowledge gathering, strategy and piloting phases and the overwhelming majority of big data initiatives are processing traditional data sources, like transactions or log data rather than a variety of data sources, such as social, email, voice, machine or sensor data. As organisations prepare for big data initiatives, they should consider the following key trends of investment driving growth.
Analytics will be more pervasive. It’s not just about big data – it’s about ubiquitous data. Analytics and insights from analytics will move out of the hands of a select few specialists to be more pervasively accessible to non-traditional business intelligence (BI) users, customers, and even for personal use. Most current tools require users to know the data they want to analyse, know the questions they want to ask in advance and also possess specialised skills to initiate queries, build analytics and mathematical models and build visualisations using the tools. These skill sets are beyond those possessed by most business users.
But this is changing as business users increasingly demand consumer-like capabilities that allow them to easily find causal relationships in data and allow them to use that as a basis to more precisely predict outcomes and prescribe the best action or decision to take (often in real time) to drive the greatest business value without specialised skills.
Examples of scenarios on the horizon include routing a caller to the best call center agent based on the caller’s voice sentiment, interaction history, social behavior and influence and demographics. To achieve the best outcome, the call center agent is automatically sent an optimised script, offers and treatment recommendations for that specific caller. Or, carpets equipped with sensors that monitor and analyse senior citizens’ activities for dangerous abnormalities, which are then delivered via mobile devices with prescribed intervention or remedial actions for healthcare professionals and /or care givers.
These types of scenarios will become more mainstream over the next two to three years with technologies that give business users human friendly and intuitive visual interaction (for example, users would be able to initiate queries and analysis using natural language voice or text questions as inputs instead of having to access BI tools) and data exploration and discovery tools with guided recommendations for finding patterns in data and for conducting more advanced types of analysis.
This will be achieved by embedding and encapsulating complex analytics from users, surfacing recommendations for optimal courses of action at the point of decision (increasingly on a mobile device), and incorporating the user’s context (i.e. location, intention, sentiment, past behavior and network). In addition, social and collaboration capabilities integrated with analytics will be increasingly important investment areas making it easier to share, discuss and socialise results and to provide a mechanism for making transparent, high quality decisions.
Much like Amazon users are presented with a “people who bought this item, also bought this one” recommendation, analytics users will be presented with similar guided analysis based on the social profiles and decision history of other decision makers and their previous interactions.
Analytics will be more precise. Organisations are increasingly investing in capabilities that enable them to discover more precise patterns and micro predictions based on diverse data - increasingly in real time.
This will require investments in advanced analytics for more precisely predicting likely outcomes with high productivity (iterating and refining many more models in a short period of time) and accuracy (on larger number of data dimensions) and in finding unknown patterns and relationships across the enterprise and within new types of data such as social, emails, call center interactions, video, and machine data.
Examples include identifying fraud and cyber security threats, best next offer, predictive maintenance, predictive policing, personal monitoring for alerting and optimised healthcare, early identification of adverse new drug effects, etc.
This requires new types of analysis such as sentiment, geospatial, and network analysis to find entities of interest, their relationship and influence. Organisations will also require new skill sets and may fill this gap by investing in a combination of internal skills building, outsourcing to analytics service providers or to crowd source analytic models.
Analytics will enable better decision making. Decisions are a basic unit of work for all organisations. The success of every enterprise is a function of the cumulative effect of the quality of the decisions that it makes. Despite large BI investments in the name of better decision making, poor decisions are abound. Where decision rules and logic are well known, more precise and real time analytics will be applied to automate a range of operational decisions.
For example, a retail food chain monitors refrigeration assets in real time to proactively predict and maintain an asset before it fails. At the same time, the quality of collaborative decisions and professional experiential and judgment-based decisions (clinical diagnosis, employee hiring, online education, personal health and wellness) will be enhanced by advanced analytics, man-machine partnerships or digital assistant models (think IBM Watson); and many more are emerging.
Moving toward something that looks simple and invisible from the user’s perspective will require new types of computing capacity and power, extended capabilities and skills, and extended capabilities in information management systems, including but not limited to:
Visual based data discovery
Natural-language query so non-traditional analytics users can find insights in data.
Contextual engines to understand the user context (for example, who users are, where users are, what users are doing, with whom are users interacting).
Semantic technologies, text, speech and video analytics to derive new insights from previously in accessible data along with algorithms that simulate the way the brain understands, aggregates and relates diverse pieces of data, reasons, and learns — much like the human brain.
Advanced analytics, such as predictive modeling, machine learning, graph analytics, sentiment analysis, statistics, and simulation and optimisation techniques — including linear and nonlinear programming.
In-memory computing, Hadoop, NoSQL, search technologies and event processing to handle large volumes of diverse and real time data.
Information Management Evolves
Analytics investments over the next three years will require an evolution of the information management architecture. Key enabling investment areas include data management hybrids for semantics and data integration, metadata management for transparency of source data, technology that relates business process model changes to the associated information assets, and graph analytics.
Graph analysis is the best option for presenting alternative scenarios, scoring them and comparing them when combining highly disparate information asset types. It is not only possible, but highly likely that multiple connection points will exist between information that was not designed to be used together. Innovation requires comparing how to combine the same set of disparate information under multiple models.
Deriving value from big data likely involves several sources of data with varying levels of structure and relationships. Different analytical outcomes can be realized at different points in the information lifecycle.
For example, continuously generating computations as data flows in, can yield insights in real-time; while analysing the data in batches results in different outcomes. No single technology supports both types of scenarios. Therefore, big data technology choices will be driven in part by the physical and logical attributes of data in combination with the desired analytical or business outcome.
Several different technologies must be combined when multiple outcomes are desired, such as real-time data processing and interactive data exploration. In the end, understanding what’s possible with big data through the technologies on the horizon will help organisations plot their course to innovation.
Rita Sallam covers business intelligence and analytics at Gartner, Mark Beyer is the co-leader of big data research at Gartner, and covers traditional data warehousing, data integration and information management practices, and Nick Heudecker is responsible for coverage of big data and NoSQL technologies at Gartner.
Previous articles in our Big Data series:
Part 1: What Big Data means for business
Part 2: Building a ‘Big Data’ strategy
Part 3: The art of Big Data innovation
We plan to publish the series a free downloadable e-book shortyly.
Copyright The Financial Times Limited 2015. You may share using our article tools.
Please don't cut articles from FT.com and redistribute by email or post to the web.