Businesses and brands have more access to data than ever before, with exponential quantities available at their fingertips.
Andreas Lang, analytics engineer, Aquila Insight now Merkle|Aquila
If harnessed correctly big data has the potential to transform businesses, but so many are struggling to keep up with what to do with it and even when they do, that’s just a small part of the process. So how can businesses extract this data? And what are the main reasons why so many are failing at it?
Well first we need to get to grips with what big data actually is. Academics describe it as data which is beyond the ability of commonly used software tools to capture, manage and process it all within a reasonable time. This description alone gives some indication as to how tricky it can be to get big data right. Here are the most common issues faced by businesses looking to tackle big data and how you can avoid them.
Capturing data
The first and perhaps also the most obvious is the capturing of data. As the definition of big data outlines; there’s no use in having the infrastructure to process and store the data, if you’re unable to get the capturing of it right in the first place. It really is a case of quality over quantity – businesses would benefit more from having the right data but less of it than having huge quantities and missing the most important things. Failing to capture the data that businesses really require or in the worst case capturing invalid data is more common than you’d think.
It’s a huge reason for failure, in fact most problems around utilising big data actually starts before businesses even begin to process it.
Content from our partners
Do you even need it?
Big data is all the craze right now, a quick google search will pull up a 1000 articles telling you that your business will implode if you don’t have an effective big data strategy already in place. But in reality, the hype of big data doesn’t mean your business has to do it right away. Ensure that you have problem to solve or use case before jumping to anything technical.
This should be something that your traditional tools are failing to solve, perhaps your business is facing an issue where it’s relying on extreme subsampling or aggregation to achieve things within the time constraints which have been imposed? The important thing is that you have a problem that’s relevant. If you don’t, then you may well not need big data at this particular time.
Processing problems
Once you’ve made sure that big data is definitely useful to your business, you’ll need to ensure that the data that you’re capturing is ready for processing. This is the tricky part, as even proving the worth of captured attributes may require a platform which is more powerful than the tools at your disposal. Despite this, be careful. Don’t rush to commit to a binding contract or expensive analytic kit right away.
Read more: Data scientists & data engineers – the answer to your big data nightmare?
Instead, hire a commercial provider to provide you with a trial platform, or use on the various cloud providers using temporary infrastructure. Feed your data into this and ensure that the data input is quality controlled and that the processes can be repeated if required. Once you start to merge these differed data streams, you may realise that attributes are missing in order for them to merge properly. Make sure to fix your existing processes first, this is the area where you really need to take your time and get it right.
Don’t forget the people
The investment in hardware that businesses should avoid or attempt to reduce in the ‘experimentation’ phase is ultimately something that will have to be faced at some point. But such is the focus on technology by businesses, that there’s another cost that tends to be forgotten and that’s people.
An equal or even bigger share of your investment needs to be into people themselves – this includes training for your current employee’s in order for them to use the tools that are being introduced and bringing in new people who can help drive the processing of large amounts of data. I’ve seen instances where there was expensive, powerful hardware around but no in house capability of actually using it. If you take your people with you on your big data journey the results will be more rewarding and the long term gains much higher. This could be an opportunity to bring in a contractor and transfer knowledge from them to drive your own needs.
Stop looking at big data like a short term project
Finally, businesses need to stop treating big data like a project. Projects have a start and then they have an end. You can’t look at big data initiatives the same way. Your data strategy should allow you to adapt to change quickly then incorporate new data streams accordingly. It’s an iterative process, and your competitors will be doing the same. So to get the head start you need to be quicker in producing higher quality insights. You can’t foresee every problem, so view your big data strategy as a continuously improving process and forget the term project.
Failure in big data strategies are often either caused by a lack of purpose, not having the right people, not having a prior experimentation stage or refusing to adapt to change. So take a step back and re-think the above, before you throw in the towel.
