Over the past few years we have seen businesses having to deal with ever-increasing volumes of data – our Digital Universe study predicts upwards of 40% growth, in business and beyond. Whilst this can be overwhelming, great value and insights can be gained from analysis of this data.
Big data analytics allows organisations to make better informed business decisions to ensure success. Yet despite a strong awareness of this, 92 percent of businesses recently admitted that they are still facing obstacles when it comes to big data projects.
Below are my top three recommendations, as well as three critical success factors, for businesses embarking on big data projects.
First, properly set employee expectations for big data. If you can consolidate enough data from multiple sources together in one place, commonly known as a Data Lake, it creates an opportunity to ask questions you didn’t think to ask, but always wanted to know the answer to. As you invest, emphasise that the goal is to answer questions that had previously been too complex or expensive to answer.
Second, avoid Data Scientists at the start of your big data projects. Whilst Data Scientists bring great value to the business and will be useful for offering insight later down the line, they will not be as beneficial at the outset. This is because there are two types of Data Scientists. The first understand how to set up a huge Hadoop or Elasticsearch cluster or a big NoSQL database.
The latter know statistics, the ‘R’ programming language, and graph theory. In both cases, they’re a solution in search of a problem and not necessarily required when first starting up. However if you have succeeded in the first step of building a data lake then the Data Scientists will have all the flexibility and opportunities they need for any future requirement.
Third, listen to the parts of the company that lack a voice and find areas where the business has been unwilling or unable to invest. Big data reduces the cost or complexity of solving problems. One success story here began when a support engineer observed that we could use big data to predict, within 90 days, when our Data Domain service would run out of capacity for backups.
Content from our partners
He was tired of manually taking support calls about "failed backups" because the backup teams were not trained in regularly monitoring storage capacities – but big data now allows us to do this.
Once expectations have been properly set, the pitfalls of gratuitous investment avoided, and critical but underappreciated problems have been solved, there are three success factors when working with big data:
1. The importance of bringing people and data together –
Too often employees are unable to access data stored in silos, holding back the opportunities for the wider team to glean insights from this information. Innovation comes when creative minds and data are brought together. Whilst data governance is important, IT shouldn’t lock everyone else out, as they will hinder further business innovation.
2. The revenue vs. optimisation battle –
Many people want to optimise a process (e.g. fewer support calls or faster bug triage) to make it more efficient, but optimisation is difficult to quantify and even harder to justify investment in. Instead, focus on ways that big data can augment business revenue.
At first, here at EMC we tried, with little success, to get funding by demonstrating an anticipated "reduced support case load." However, interest and funding expanded when we tracked the revenue generated by selling additional Data Domain storage and systems to customers who were about to run out of capacity.
3. Why businesses should hire both generalists and specialists –
At the beginning, you don’t need a hyper-optimised big data infrastructure. You need somebody who understands the business problem, what data they need, how to access the data, and how to deploy basic big data tools. In short, you need a problem-solving generalist who can learn quickly. As the solution expands, hire specialists to optimise each part of the process.
As with most business/technology transformations, the challenge with big data is not one of technology. To succeed with big data, it will be important to manage business expectations, avoid technology hype, and embrace revenue-generating ideas from underfunded areas. If you keep your Data Lake open and accessible, you’ll unlock passion from parts of the organisation that have been desperate to do more.
Often, starting small – especially when it comes to big data – can have the greatest payoff!
