Many claims for big data projects suggest that just collecting the data somehow gives business the insight they need. The hype is that companies are ignoring this huge resource which is just waiting to be tapped. Early promoters even described big data as “the new oil”.
There is no doubt that IT systems can provide CIOs with vast amounts of data but not all it is useful. There are several steps between collecting big data and turning it into intelligence or useful insight.
The traditional model is a pyramid which moves from Data – Information– Knowledge – and finally Wisdom.
Business should not look at big data as a resource just waiting to be dug up.
The reality is that the second step – turning data into information – is the most difficult, 
and often expensive, part of the process.
This means cleaning the data and removing rogue or useless results. It means adding context or even making an inference from the data collected.
Knowing that a temperature sensor is reading 800 degrees Celsius is data. Knowing that the sensor is on the office ceiling is information which provides the knowledge that the building is probably on fire.
Content from our partners
Working this process on a single sensor is simple.
But dealing with a large data set is hugely complex and runs the risk of adding subjective spin if not handled carefully. In many cases it is actually ‘gut feelings’ about data which are just as important as the data itself.
There are very few data sets which are clean and ready for processing on collection. It is not like oil which can be fed into a refinery to produce petrol and other products.
The irony is that these supposedly objective and automated systems actually require very human intervention to make them work.
The final stage is turning knowledge into wisdom – or into decisions or actions which the business can take.
This requires a system which understands more than the data – it must understand the limits within which the business, and the world, works.
Big data can provide genuine insights for business.
Successful projects are changing many aspects of how businesses function.
Big data can provide a better understanding of marketing activities, website design, data centre infrastructure, recruitment and retention of staff, logistics, manufacturing and retail.
But a successful project needs to understand the limits of big data as well as its strengths.
Not everything you can count can provide actionable insight into your business.
The danger of an obsession with big data is a reliance on numbers which are easy to get through the first two steps of the pyramid. Data which seems to be clean and useable will be seen as more important.
But in fact there are things which are easy to count but do not make you smarter.
Equally there are some things you cannot count which can still deepen understanding of your business.
Outside of the numbers are fuzzy things like brand loyalty and brand feelings. For many companies these are vitally important to their success but they do not lend themselves to being easily measured.
However good the data an intelligent outcome still requires intelligent questions to be asked.
Put simply – even with perfect data if you ask the wrong questions you will get the wrong answers.
Think about mobile phone manufacturers measuring sales success in the early 2000s.
It was easy to count sales and to accurately estimate rivals’ sales. Total market size, penetration, margin per device and revenue per user could all be measured. These numbers could be turned into predictions of future performance with a reasonable degree of confidence in their accuracy.
Then the iPhone arrived and changed the market for ever, previous predictions were rendered useless almost overnight.
Or consider recent political events in the UK and the US. Pollsters had plenty of data but either asked the wrong questions or at least got the wrong answers.
There is no doubt that data analysis will play an ever larger role in business decisions. But raw data does not just magically turn into brilliant business decisions.
Don’t be fooled into thinking that just pouring enough data into a system will magically create totally objective and value-free decision making.
Used wrongly big data systems can reinforce existing assumptions and be used to justify decisions which have already been taken.
This can stop a business making the innovative decisions required to stay competitive.
Big data will have an ever bigger role in business decision making, but the smartest businesses will still need human brains.
