Dark and Redundant, Obsolete and Trivial (ROT) business data could cost organisations around the world $3.3 trillion to manage by 2020.
The dark data, which has an unknown value and redundant, obsolete or trivial data is proving to be a huge burden to organisations that have hoarded data without processes in place to categorise and understand what they are keeping hold of.
According to the Global Databerg Report produced by Veritas 52% of all information currently stored and processed by organisations is considered dark data, with an additional 33% considered ROT.
A data hoarding culture combined with an indifferent attitude to retention policy is to blame and not only is it a huge financial burden, it also poses a significant regulatory and security risk.
The report found that only 15% of all stored data can be classified as business critical information, according to IT leaders.
For an average midsized organisation that holds 1000TB of data, the cost to store non-critical information is estimated at more than $650,000 annually.
Part of the problem has been caused by the rapid rise of the kinds of data that is being collected. Businesses are now able to collect M2M data, log files, analytics, surveillance and geo information as they look to use it to improve parts of their business.
However, the issue is that a lack of policies in place for the management and retention of that data has resulted in vast amounts being stored which offer no value or unknown value because it has not been analysed.
Data lakes may provide a way for businesses to store this data but they aren’t free, so really it doesn’t matter if the cost of storage is cheap because when the volume of data continues to rise and no business value is being derived from it then it is essentially a dead albatross around the neck.
Considering that IT is being asked to do more with less due to shrinking budgets, a simple win could be had through proper data management formed by a well thought through big data plan.
Ben Gibson, Chief Marketing Officer of Veritas said: "The problem most face is they do not know what data to start with, what risk it may contain, and where the value is discovered. Once they have visibility into that environment, they can make decisions faster, with more confidence, and bring in other business stakeholders to move forward with a well-conceived plan."
Not understanding what is being stored can pose a significant risk, particularly when considering the increasing levels of data regulations such as the EU’s General Data Protection Regulation that will come into force in the next two years.
The regulation will force businesses to know what data they have and to implement mandatory breach notifications.
Full breaches will face a financial penalty of either €100m, or 4% of the company’s annual turnover but importantly the company will have to identify what data has been breached and to notify customers if any of their data has been compromised.
The question is whether businesses can effectively do that when 52% of the data is considered dark.
While some data needs to be kept for regulatory or compliance reasons much is unnecessary and an effective data management policy aligned across business and IT.
Joe Garber, VP, Marketing, HPE Software, Big Data Solutions, told CBR: "It’s an age old question, you have IT saying keep nothing and legal and compliance saying keep everything so you have to find that balance."
Addressing the problem can be done in two ways, one is to look historically over time in order to get policies right, the other is to get close to those policies and to properly analyse them. This means running analytics on the policies to see what is necessary and what isn’t.
The study was conducted for Veritas by Vanson Bourne which surveyed 2,550 senior IT decision makers across 22 countries.
