The appeal of Big Data is the opportunity to drive business improvement through better insights gleaned from a larger variety and volume of information than ever before. Big Data brings the potential for better customer insights, more detailed analytics and tighter risk and regulation management.
The ability to aggregate and analyze diverse datasets can results in revenue increases, more operational efficiency and a strong competitive edge over other industry players. According to Gartner, organisations integrating high value, diverse new information sources into a coherent information management infrastructure will outperform their industry peers financially by more then 20 percent.
However, whilst the potential of Big Data is alluring, myriad data sources raise concerns around quality and validity. A well-held belief amongst some business professionals is that, by virtue of sheer volume, quality errors in Big Data are less impactful. However, Gartner VP Ted Friedman recently fought back against this myopic view of Big Data: "In reality, although each individual flaw has a much smaller impact on the whole dataset than it did when there was less data, there are more flaws than before because there is more data."
Many CIOs are struggling to verify the large amounts of structured, unstructured and semi-structured data yielded from an increasingly diverse range of sources. Moreover, because much of today’s data is not validated at the point of creation, businesses are finding it difficult to implement a systematic data management approach that delivers data analysis, validation and assurance of Big Data before it’s utilised across crucial downstream applications such as CRM systems and business intelligence and analytics platforms.
Increasingly organisations are relying on third party data and information sources outside the business, such as social media streams, that are often unreliable, fragmented, and not automatically linked to existing internal data records. In the formative years of traditional analytics data quality was a costly afterthought, and it’s crucial that anyone establishing Big Data operations avoids repeating those mistakes.
To overcome these challenges organisations need to adopt a more thorough data management approach that enables seamless integration of data regardless of structure, source or application. Data science and predictive analytics applications can then query across unstructured data sources (where it was not possible before) and improve processing of previously inaccessible data. Business professionals can then more easily identify the right data to solve their specific business problems and align the right capabilities and skills to exploit the data in a way that positively affects their bottom line. For instance, from a CRM standpoint, matching data to individual customers can drive personalisation and provide an improved customer experience.
In order for Big Data initiatives to yield tangible business results, businesses need to manage the quality and validity of business information compiled in new storage systems such as Hadoop. To do otherwise risks producing misleading results. The old axiom "better data means better decisions" is just as important when dealing with Big Data, and if your data is to become a trusted information source and useful tool for employees, partners and customers, a robust Big Data governance framework is a must.
Organisations are beginning to wake up to this reality. A recent survey indicated that whilst only 11 per cent of IT leaders are currently looking into data quality, this percentage is expected to grow to 55 per cent in the next three years.
Big Data is a huge challenge for organisations but also an incredible opportunity to improve process efficiency, create more personalised customer experiences and drive innovation. Organisations that handle Big Data appropriately will undoubtebly be at the forefront of business innovation, leading the way in customer engagement and service quality.
By Ed Wrazen, VP Product Management: Big Data at Trillium Software
