View all newsletters
Receive our newsletter - data, insights and analysis delivered to you
  1. Technology
  2. Data
June 30, 2016updated 26 Sep 2016 7:35am

Big data, big opportunities, bigger challenges?

John Beal, Senior Vice President Modelling Services at LexisNexis Risk Solutions, reflects on the recent changes in the big data industry.

By Cbr Rolling Blog

Back in December 2014, CBR Online spoke to industry experts for their predictions regarding data scientists in the coming year. So how much has changed in a year? In our part of the world, which is focused on data intelligence to help calculate insurance risk, the change has been pretty dramatic with demand for skills seriously outstripping supply. But our experience isn’t unique, across a whole spectrum of business sectors, more and more data is being pooled – whether it’s on how we shop, drive, run our businesses or spend our leisure time. This vast array of data needs expert analysis to ensure organisations and their customers benefit from the intelligence the data can provide.

Tackling the dearth of talent
So how do we solve the problem of this dearth of talent? Well to me, attracting the best data scientist involves far more than offering the highest salary and a nice benefits package. It requires a challenging work environment where there is a constant changing problem set.

The best minds will not be satisfied for long unless they see new problems that continue to challenge them, rather than a set of repetitive tasks – regardless of the initial level of difficulty. The companies which are natural innovators and can challenge the data scientist, provide adequate tools and deliver other big data capabilities will attract the best talent. Simply having a data scientist on the payroll doesn’t create the competitive edge.

Busting the big data myths
We also need to dispel some of the most common misconceptions around data preparation and machine learning algorithm selection when considering the role of the data scientist and the value that can be derived from Big Data.

There appears to be a widely-held belief that a data scientist can point machine learning software at a problem and the machine knows what to do and how to handle the data. However, the data is never what you or the software expects it to be. There are always outliers, edge cases, and errors. Cleaning the data is critical to successfully applying any data science approach.

You also need to consider creating the right set of attributes or acquiring powerful data sources to feed into machine learning algorithms. Then comes perhaps the most challenging part – linking multiple data sources – this can have a huge bearing on the analysis.

Finally, any expectation that the output of the software knows how to perfectly fit your specific data to its selected algorithm is unrealistic and doomed to failure. Typically, historical data or past experience is used to predict the future but there is always the risk of focusing too heavily on the history and ‘overfit’ a model to the point that it won’t generalise to the ‘future’ data. As such, model ‘overfitting’ can have a significant impact on the future application of the model and rigorous processes are required to control for overfit.

Content from our partners
Green for go: Transforming trade in the UK
Manufacturers are switching to personalised customer experience amid fierce competition
How many ends in end-to-end service orchestration?

Same industry, varying levels of expertise
The same principles of extracting value from big data applies across all industries but the condition of the data and the organisation’s ability to cleanse it varies widely from one business to another – even within industry sectors, as does the experience set of the data scientist applying the software and adjusting for real world challenges. It makes sense then that the businesses who have the best chance of extracting value from big data are those that understand the level of human expertise they need to put in place to gain control of their data.

Looking to the future of big data
There’s no doubt that with technology, data availability and data analysis techniques continually evolving, the future of big data is exciting. The capacity challenges previously affecting so many businesses are being resolved through big data systems, meaning that carrying out analysis of large data sets, even for rare events, will become easier and easier over time.

Applying the more time-consuming algorithms will become easier as well.

The educational environment across the globe is also changing, now embracing the data scientist as a career, with specific core competencies required to effectively transition from the academic world into the corporate world. We’ve certainly experienced this at LexisNexis Risk Solutions though graduate recruitment from various local universities.

The emergence of the knowledge manager
The last area I believe needs to change significantly is the emerging role of what I would call ‘the knowledge manager’. This is the ideas person in essence. The bridge between the technical and business resources continues to strengthen, but managing, communicating and leveraging the abilities of the data scientist is not as straight forward as one might think. Companies in every industry need to realise that often two languages are spoken in the same discussion when very analytical people converse with non-analytical people, and it is essential that the role of the knowledge manager is there to link this dialogue.

The problem is that the people with the requisite skills to be knowledge managers are even rarer than data scientists.

Big data is growing rapidly, spreading across industries and being employed by companies of all sizes. It is exciting to look forward to where it will take us in future. With more and more elements of our lives becoming ‘connected’, there are really no limits to the data which will be available in the coming years and the increasingly valuable role data scientists and knowledge managers will have in our businesses.

Topics in this article :
Websites in our network
Select and enter your corporate email address Tech Monitor's research, insight and analysis examines the frontiers of digital transformation to help tech leaders navigate the future. Our Changelog newsletter delivers our best work to your inbox every week.
  • CIO
  • CTO
  • CISO
  • CSO
  • CFO
  • CDO
  • CEO
  • Architect Founder
  • MD
  • Director
  • Manager
  • Other
Visit our privacy policy for more information about our services, how New Statesman Media Group may use, process and share your personal data, including information on your rights in respect of your personal data and how you can unsubscribe from future marketing communications. Our services are intended for corporate subscribers and you warrant that the email address submitted is your corporate email address.