The clichéd view of big data projects is a bunch of guys in white coats running around a massive data centre or twiddling with hugely complex algorithms on a supercomputer. But the truth is that today you can achieve a successful big data project without hiring dozens maths PhDs and spending several million on new hardware.
A successful big data project should start small. For a properly run business a big data project is likely to create lots of small wins and improvements in processes rather than one or two huge victories or savings.
A company like Amazon uses all the data provided by users like you or me on their home page. Tiny improvements in what they offer us and how quickly we make our purchases lead to big improvements to the company’s bottom line at the end of the year.
Big data for small wins
This sort of optimisation comes from lots of tiny gains. And each one comes from data not just gut feelings. Any change will be tested on a small cross-section of users. Results will then be analysed and any necessary changes made. This can then be tested again before a wider roll-out.
Everyone loves the business myths like the man who suggested that the match company Swan Vesta only put the sandpaper striker on one side of the box. Or the marketing genius who suggested instructions on shampoo always say ‘lather, rinse and repeat’.
But the boring truth is that proper business analysis is far more likely to find you a five per cent cut in overheads rather than a fifty per cent increase in profits.
Successful business analysis is all about these details rather than finding the huge untapped opportunity which turns a business on its head. Cutting call centre queues by ten per cent will have an impact on customer satisfaction, staff morale, call centre costs and likely improve sales too.
Content from our partners
And once you start making a few small savings like this the cumulative effect can be huge.
The difference between the good and the bad company is how to keep these savings coming month after month and year after year.
So any big data project needs to change the way the company functions. It won’t just run for six months, deliver savings, get you a huge bonus and be closed down.
It needs to have a permanent and ongoing impact on how the business is run and how it develops in the future – what HP Enterprise calls the ‘Data Driven Organisation’.
Getting a clearer overview of how a business functions can provide surprises. But the reality is likely to be multiple trends which will need reacting to rather than one big switch. Additionally the picture is likely to get more interesting over time – few projects deliver just one stunning insight – they provide a clearer view of how your business performs as strategy, investment and market conditions change.
Look at what you already have
The first step is to make sure you’re collecting as much data as possible. But before you go changing how this data is collected have a proper look at what is already collected in different systems. You might be surprised at the scale of the information you already have on hand and are not using.
The first technical challenge might be getting this data in one place so it can be properly used.
Again starting small makes sense.
Run a pilot project to see what useful information you can get from this data before re-building infrastructure so it always available. You might find that you don’t actually need real-time, 24/7 access to it but can get away with monthly or even quarterly updates and checks.
But for core business information you need the best and easiest access you can afford.
Such pilot projects will also help you decide what information to exclude – any database will include information which is just ‘noise’ and of little value to business decisions. Think carefully before making such exclusions – and revisit those decisions regularly. Don’t be afraid of sidelining some data sources but equally don’t ignore them for ever.
Big data doesn’t mean a big database
The good news is that new architectures like Hadoop, HPE IDOL and other database tools make it easier to grab and analyse data from various sources in today’s typical hybrid infrastructure.
Once you defined the data to include and found ways to access it then it is time to start analysis.
Business has been creating, and storing, masses of data for years. The thing that made it a useful and increasingly vital business tool is the analysis.
This covers both structured data like transaction records but also less tidy, unstructured information like that from social media responses or customer surveys.
You need tools that can deal with as much of this information as possible.
Look to change the rules, but not break them
Of course you need to remember that this brave new world is still covered by old fashioned data protection laws. Whatever you are doing with customer data needs to follow the relevant regulations for your industry. Data protection and security must remain a priority. Giving lots of people within your company access to this data doesn’t mean they need to also know customer details for instance.
It’s about questions as much as answers
Working with a partner to get the right tools and technology in place isn’t the hardest bit of the project. The tough bit is asking the right questions.
That’s where you need to get people from different areas of the business involved and supporting the project.
Once again starting small can help you win over supporters and hopefully show the usefulness of the technology. By solving small but irritating issues with business processes you can illustrate the potential of becoming a data driven organisation. Getting the right people involved and asking the right business-focussed questions will help make the project work.
When you start to get the business behind the project then you can start to show the importance of using data to make bigger decisions both better and faster.
Then you’ll be ready for the next step – not just using data to see what went right and what wrong in the past but starting to use it to predict what will or won’t work in the future. Depending on the scale of the project this might include moving towards modelling possible future products or different market scenarios.
This won’t suit every organisation – some might find data visualisation and limited predictive analysis more useful.
But almost any organisation can find value in its own data. And building the tools to access that value needn’t be a hugely expensive IT headache.
Most companies spend good money on information from specialist analysts, market watchers and survey companies while ignoring this massive untapped resource – the valuable information they already have. To compete today, and tomorrow, you can’t afford to keep ignoring this valuable resource.
