This hunger for tools to support accurate predictions has driven demand for business intelligence software since IBM established the market in the 1970s. I’ve worked in it long enough to have witnessed the pendulum of buyer influence and ownership swing from IT to the Business.
IT people insisted for decades that they needed to own these systems to enforce strict governance, mitigating security breaches and supporting good decisions. This was despite business people’s growing objection over there being no point in applying all that rigor if insights arrive too late to have any impact.
Between the early 2000s and 2010, after years of costly, failed implementations, the pendulum swung. A new breed of highly visual, industry-specific, and some cloud-based tools – called “BI 2.0” – tempted customers to sidestep IT altogether. But the hunger still went unsated: though much easier to adopt and use, none of these new tools drew on enough data to inform consistently reliable, accurate predictions. At best, they provided a visually appealing look in the rear view mirror. And with IT largely out of the picture, much of what reflected back was built on unreliable, flawed data.
Then in 2010 Apache Hadoop brought about the market’s biggest disruption ever: big data. This, combined with declining computation and storage costs, meant that for the first time ever, reliable predictive analytics seemed within reach.
Now it was becoming possible to tap into a huge universe of diverse, unstructured, qualitative data held in documents, clickstreams and machine sensors. And while business people were loath to relinquish their newfound access and control over BI, they needed IT’s (including data scientists, engineers and analysts) support to design, build and gain value from these new, complex systems. They accepted that without good data hygiene and governance, these powerful new business analytics and IoT systems could prove at best, unreliable and at worst, risky.
From industry BI solutions to embedded analytics platforms
Finally the pendulum has settled in the middle. IT and Business partners are coming together to shape a new market that demands business-friendly ease of use, speed of analysis and predictive insights built on a robust foundation of IT-led data hygiene, auditability, governance and security practices. Along with this shift from BI to big data analytics, comes a change in what the business values most. Data integrity and reliability are trumping values that were the hallmarks of BI 2.0: industry customization and visual ‘eye candy’.
The de-emphasis on industry customization and visualization go hand in hand. As IT teams get more business-aligned, I’m seeing a growing trend to reconsider standalone data visualization tools in favour of embedded platforms that make analytics available from people’s everyday line of business applications. This approach is the best way to truly ‘democratize’ analytics, but would never have occurred to a line of business manager evaluating BI tools without IT. A great example of this is our customer CERN, which has rolled analytics out to 15,000 users by embedding it into its existing ERP, Finance, HR and other systems.
Meaningful insights from blended data
Another big concern that IT and business partners are working together to solve is gaining insights from blended data. With IoT use cases, for example, it’s not enough to just grab a bunch of sensor data and analyse it on its own. The real insights come when you can blend that machine-generated data, with other sources like corporate and sentiment data in social feeds. Again, this is a problem that wouldn’t have been identified without close IT and business alignment.
The ability to gain predictive insights from blended data are evident in the most business-critical use case out there today – cybersecurity. Our customer BT Assure Cyber brings together event data and telemetry from a rich variety of data sources including business systems, traditional security controls and advanced detection tools. Assure Cyber customers can harvest insights from all this data, relational and unstructured. This means vulnerabilities and incidents that would previously have taken days, even weeks, to investigate and respond to, can now be identified and acted on immediately.
Speed is non-negotiable!
One thing business users in companies like BT Assure Cyber will never sacrifice is speed. There’s still no point in having perfect data, if insights arrives too late to act upon. Fortunately because the new frontier is open source, a whole Hadoop ecosystem and global developer community is working to solve tough problems that can slow down data preparation like onboarding, transforming, blending and streaming diverse data sets.
The new analytics brief
Whether you work in a growing SME with a clean IT slate or a large enterprise with lots of legacy systems, no doubt you have felt some of the aftershocks of the big data disruption on the market as a whole. Many BI 2.0 vendors are adapting through technology partnerships and R&D investments, but the pace of change can be disruptive to existing customers. Legacy vendors that discovered too late that someone moved their cheese are getting aggressive – from deep discounting, to suing customers for extra license fees to filing patent lawsuits (always a red flag).
Selecting the right business analytics system in this turbulent era can be a minefield. Your old functional brief may no longer ask the right questions. Collected from some of our most successful enterprise customers within Pentaho and Hitachi, these are the top ten new functional requirements we see when IT and Business collaborate on briefs for big data / IoT platforms for predictive analytics:
- Is it based on open source / open standards (making it interoperable, flexible, supported by a community, future-proof)
- Does it play nice with others (including the various BYO data visualization software bought in the last decade?)
- Does it have native support for the most popular technologies that accelerate data prep (Spark, Kafka, etc.?)
- Can it be embedded into our existing systems and can we customize different interfaces for different user roles and devices?
- Does it support all data types we need to analyse today and in the future?
- Does it support onboarding and blending of diverse data types – relational and unstructured
- Does it support both cloud, on-premise and hybrid architectures
- Does it support our existing DevOps processes and technologies
- Does it provide a way for IT, data scientists and business users to collaborate?
- Can you roll out to new users, customers and deploy on multiple devices, without license costs spiralling out of control (or even having your supplier sue you for license fees)
In today’s uncertain world, predictions in business are more important than ever. However thanks to a much closer alignment between IT and business and the big data revolution, at least these predictions are becoming less hard.
