Back in December 2014, CBR Online spoke to industry experts for their predictions regarding data scientists in the coming year. So how much has changed in a year? In our part of the world, which is focused on data intelligence to help calculate insurance risk, the change has been pretty dramatic with demand for skills seriously outstripping supply. But our experience isn’t unique, across a whole spectrum of business sectors, more and more data is being pooled – whether it’s on how we shop, drive, run our businesses or spend our leisure time. This vast array of data needs expert analysis to ensure organisations and their customers benefit from the intelligence the data can provide.
Tackling the dearth of talent
So how do we solve the problem of this dearth of talent? Well to me, attracting the best data scientist involves far more than offering the highest salary and a nice benefits package. It requires a challenging work environment where there is a constant changing problem set.
The best minds will not be satisfied for long unless they see new problems that continue to challenge them, rather than a set of repetitive tasks – regardless of the initial level of difficulty. The companies which are natural innovators and can challenge the data scientist, provide adequate tools and deliver other big data capabilities will attract the best talent. Simply having a data scientist on the payroll doesn’t create the competitive edge.
Busting the big data myths
We also need to dispel some of the most common misconceptions around data preparation and machine learning algorithm selection when considering the role of the data scientist and the value that can be derived from Big Data.
There appears to be a widely-held belief that a data scientist can point machine learning software at a problem and the machine knows what to do and how to handle the data. However, the data is never what you or the software expects it to be. There are always outliers, edge cases, and errors. Cleaning the data is critical to successfully applying any data science approach.
You also need to consider creating the right set of attributes or acquiring powerful data sources to feed into machine learning algorithms. Then comes perhaps the most challenging part – linking multiple data sources – this can have a huge bearing on the analysis.
Finally, any expectation that the output of the software knows how to perfectly fit your specific data to its selected algorithm is unrealistic and doomed to failure. Typically, historical data or past experience is used to predict the future but there is always the risk of focusing too heavily on the history and ‘overfit’ a model to the point that it won’t generalise to the ‘future’ data. As such, model ‘overfitting’ can have a significant impact on the future application of the model and rigorous processes are required to control for overfit.
Same industry, varying levels of expertise
The same principles of extracting value from big data applies across all industries but the condition of the data and the organisation’s ability to cleanse it varies widely from one business to another – even within industry sectors, as does the experience set of the data scientist applying the software and adjusting for real world challenges. It makes sense then that the businesses who have the best chance of extracting value from big data are those that understand the level of human expertise they need to put in place to gain control of their data.
Looking to the future of big data
There’s no doubt that with technology, data availability and data analysis techniques continually evolving, the future of big data is exciting. The capacity challenges previously affecting so many businesses are being resolved through big data systems, meaning that carrying out analysis of large data sets, even for rare events, will become easier and easier over time.
Applying the more time-consuming algorithms will become easier as well.
The educational environment across the globe is also changing, now embracing the data scientist as a career, with specific core competencies required to effectively transition from the academic world into the corporate world. We’ve certainly experienced this at LexisNexis Risk Solutions though graduate recruitment from various local universities.
The emergence of the knowledge manager
The last area I believe needs to change significantly is the emerging role of what I would call ‘the knowledge manager’. This is the ideas person in essence. The bridge between the technical and business resources continues to strengthen, but managing, communicating and leveraging the abilities of the data scientist is not as straight forward as one might think. Companies in every industry need to realise that often two languages are spoken in the same discussion when very analytical people converse with non-analytical people, and it is essential that the role of the knowledge manager is there to link this dialogue.
The problem is that the people with the requisite skills to be knowledge managers are even rarer than data scientists.
Big data is growing rapidly, spreading across industries and being employed by companies of all sizes. It is exciting to look forward to where it will take us in future. With more and more elements of our lives becoming ‘connected’, there are really no limits to the data which will be available in the coming years and the increasingly valuable role data scientists and knowledge managers will have in our businesses.
