Big data is shifting to the mainstream of the business technology conversation. Where once it was a specialism for certain departments it is becoming more common in all areas of business and across different industries.
In part this trend is possible because of a fusing of three separate types of technology – cloud, Internet of Things and big data.
IoT creates big data even for industries which previously survived without processing much information. Manufacturing, retail and agriculture are all being transformed by IoT.
But the cloud makes it possible to deal with this data for less cost and with fewer specialised staff.
The cloud also provides a variety of off-the-shelf solutions for dealing with big data challenges.
And finally the cloud provides the instant scale to deal with storing and processing data even if it arrives all at once and must be dealt with fast, like year-end financial information.
But big data is also increasingly ‘working under the hood’.
Analytics which is embedded at the edge of networks, where the data is collected, can react quickly and automatically without human intervention. This might not seem like true big data but it is working on algorithms created in a lab somewhere and not reinvented for every application.
The point of big data projects is to provide insight and answers for the business not just to create data lakes for the sake of it.
Along with IoT the other new provider of data for many enterprises is mobile. Whether directly collecting data from users devices via an app or analysing web traffic from mobile devices this is an area which will keep on growing.
First steps to spreading big data across the organisation.
Businesses can take several practical steps to take best advantage of these changes.
Firstly to stay safe you need to create policies to protect privacy while giving access to more data to more members of the team.
Making the most of big data does not mean hiring more data scientists. It means getting hold of tools which will allow more people in your company to use the data you collect.
The people at the front line – whether in sales or human resources – know the questions which their department needs to answer.
Big data tools must be open and usable to as many people as possible – self-service systems seem likely to become the norm.
There will still be a role for data specialists to help and hone queries and improve data strategies but the next generation of systems will have a far wider reach.
Infrastructure must be able to adapt to changing demands from the business and from regulators. The world is changing every more quickly and chaotically and systems need to keep up.
UK businesses are already building systems which will comply with the demands of GDPR – European data protection and privacy rules which come into force next year. But the UK government is promising to change these rules. There’s a good chance this won’t actually happen, or the changes will be minor, but to safely future-proof your systems you need to make sure the rules are not baked in and can adapt to a changing regulatory environment.
The final part of the big data puzzle will be wider adoption of machine learning and artificial intelligence to accelerate lessons learnt from big data and even speed up how the business reacts to the insights these systems can provide.
