With Big Data analytics impacting almost every aspect of our lives, from our health to our bank balance, data scientists are taking on Harry Potter-like status. Held in awe for their wizardry and algorithms, they command high fees and big respect. So how can mere mortals learn how to benefit from Big Data themselves?
It’s a serious problem. In a recent Talend survey 39% of those organisations polled cited lack of skills or time as the main barrier to successfully managing and exploiting the potential opportunities of Big Data. And, as the whole issue of Big Data has evolved, the technology required to navigate it has become increasingly complex. Consequently, the pool of engineers capable of using it or even understanding it is shrinking.
Part of the problem is the smoke and mirrors surrounding Big Data. It’s seen as a complex and intricate technology that only the most talented IT department can fathom. The survey found that in 28% of cases, the initial push towards data-driven projects originated from this quarter, whereas only 13% said that the board or senior management team were the drivers.
This technical focus just makes the skills gap worse. Most IT departments don’t have the overview, resources or the power and influence to look at the strategic picture. Over half of those polled (55%) thought that to realise Big Data’s full potential, there was a need to create multi-disciplinary teams including IT, but also business heads, analysts and industry experts to build an overarching vision.
But in some ways there’s an onus on software vendors too to focus on simplifying the technology rather than making it even more opaque. Businesses need technologies that enable non experts to use Big Data – that would allow a regular data integration developer – someone who has been working with ETL for a few years, for example, to create and deploy Big Data concepts.
Hopefully it’s only a matter of time. Already Big Data ‘sandboxes’ or remote desktops are delivering preconfigured virtual environments designed to get Big Data projects off the ground quickly. They provide an ideal way for businesses to experiment and in turn remove the perception of risk.
Other products, such as the newly-released Talend 5.6 platform, simplify the most advanced new technologies to enable otherwise leading-edge tasks. These include the ingestion of the rapidly changing data involved in financial transactions and enable ever faster analysis for real-time decision-making.
When multi-discipline teams become involved with the strategic side of Big Data and are able to achieve their Big Data goals via self-service, the IT department is free to concentrate on the skills side – and these new solutions give them the confidence to do so and master the basics without the need for the hocus pocus of the technical experts.
In other words they give businesses the confidence to develop their own commercial deployments without being deterred by risk, cost or complexity. While there will always be the need for the deep-dive data scientists at some stage, the combination of a sound strategy, teamwork and these new technologies can often work a magic of their own.
