Head over to your search engine of choice and do a search for the latest news from the big data industry and chances are the results will be a stream of use cases or problems facing the industry.
In 2015 Gartner removed big data from its Hype Cycle due to the pervasiveness of data and basically because everyone is using data across many areas. Basically it’s no longer an emerging technology area, despite it being a relatively young term.
Almost like a self fulfilling prophecy big data has pretty much removed itself from the news.
The problem is that big data has been around for decades in one form or another in enterprises. It has often cynically been referred to as a re-branding of batch processing.
The hype has gone, but that isn’t necessarily a problem because it should mean that businesses are actually getting down to work and making the most of their data treasure troves.
It’s not really a case of there being no innovation in big data, but the eye catching developments have certainly slowed.
Plenty of work can be seen by the likes of HPE, which is building a strong analytics portfolio; Amazon Web Services, which wants more analytics in the cloud; and IBM which is taking both a hardware and software approach to analytics.
Content from our partners
The reality is, is that plenty of work is being done still to further tools and capabilities but the conversation has shifted. The conversation is now focused on case studies.
Case studies dominate the search results of big data and that’s because businesses still seem to be figuring out where exactly it can be used, how far it can take them.
Some of the typical big data use cases seen are in healthcare, the insurance industry, science, banking, and more. As Gartner pointed out, big data is pervasive.
Reacting to this a number of vendors have taken different approaches to dealing with customers. While I’m sure there is still an eagerness to get customers to use more of the platforms, software, and hardware on offer, the conversation now is more of a consultancy one.
Professional services firm PwC reported an 11% rise in revenue to £3.4bn in the year to 30th June, partly because it is offering expertise in technologies that businesses need help in.
I mention this because it’s where the big data market finds itself. The majority of vendors who offer big data products will also offer some form of services associated with it because, frankly, big data remains confusing.
Cloudera is one example of a big data vendor that is offering guidance around big data, helping businesses to form a coherent data strategy to become a data driven business.
This kind of service is important, not only does it mean that the customer will hopefully get the most out of the product, but it also means that they will understand the risks associated with regulatory changes.
With changes in data privacy regulations coming in with the likes of the EU’s General Data Protection Regulation, it is more important than ever that businesses have a data strategy which works.
However, data strategy is a little boring compared to the excitement that technologies like Apache Spark brought to the industry.
That’s not to say that innovation is not happening at all, because plenty of projects are going on that could bring back the excitement.
Take Apache Nifi for example, originally called Niagara Falls. It was developed by the US National Security Agency and is designed to tackle the problem of how to automate the flow of data between systems. Apache Flink is another one to look out for, it is a distributed data analysis engine for batch and streaming data, is offering programming APIs in Java and Scala.
Apache Arrow from Dremio will provide columnar in-memory analytics and it’s another one that’s recently become a Top Line Apache Software Foundation project.
In total there are currently over 300 open source initiatives at the Apache Software Foundation, which is not to suggest that all 300 will bear fruit and become a game-changer in the industry, but it does suggest that there’s still some innovation going on.
