A data scientist is a professional who works to extract insights and make sense of large sets of data, or ‘big data’.
As the term is broad and continually evolving, CBR turned to Cloudera’s Sean Owen to help explain this ever important role.
EB: What is a data scientist?
SO: I still see it as a term used to mean a variety of things. To my ears it sounds like “enterprise architect,” a title claimed by everyone from junior DBAs to software engineers writing anything that doesn’t have a UI. Perhaps they’re all correct, but the title is now irritatingly meaningless.
To be sure it’s something to do with software engineering, domain knowledge, and statistics. The data scientists are those making plots on a laptop in R, but also building big ML pipelines on Apache Hadoop, even if those are quite different things. The best data scientists can do both, and, it has become expected that people good at either of these now become versed in both.
As the field matures we may fall back to more specific terms like data analyst, data engineer and statistician rather than bundle all of them in “data scientist”.
EB: What is the difference between a data analyst/statistician and a data scientist?
SO: Supposedly, the data scientist also possesses software engineering skills. However, a lot of data science profiles I see, I’d more readily call statisticians. They typically have a formal background in applied math, statistics or related fields like operations research. They tend to be skilled in exploratory analytics tools like R, SAS or SPSS and they tend to not have as much exposure to software engineering practices or tools.
These are the original “data scientists” yet find themselves at some disadvantage where there isn’t enough distinction between the role of modelers working in a lab and the role of engineers maintaining the production systems these models affect. Suddenly there’s a demand to also write and maintain production software or build models at scale on clusters.
EB: Do all organisations need a data scientist?
SO: Only hire if your data house is in order. Hiring a data scientist as a ‘fixer’ for lack of data strategy will be a waste of time. Until data is abundant, clean and accessible, weaving it into magic business results will be an uphill struggle. Build a data engineering discipline first, and you’ll then find that data science is the easy part. It’s also important to be clear about which type of data scientist you need. Some are ‘exploratory’, tend to work in environments like R, Python, and SAS, work on a laptop and create visualisations. Some are ‘operational’ and productionise large scale software implementations of learning systems.
EB: What are the specific skills a data scientist must have?
SO: It’s determined by what kind of “data scientist” is required. The “data engineer” type needs understanding of information architecture, tools for storing and structuring data. He or she also needs software engineering skills to create, debug and maintain pipelines that feed models, and score models. These skills usually centre on the Apache Hadoop ecosystem, including Apache Spark.
The “statistician” type needs some background in linear algebra and statistics, and knowledge of some kind of exploratory analytics environment like Python’s scikit, or R. This type of data scientist is often assumed to have skill in visualisation, though that is more of a bonus.
All the better if a data scientist can do some of both. These skills are usually paired with something like a management consultant’s skill set, an ability to quickly understand the basics of a business domain, extract a solvable data problem from conflicting requirements, and explain the results to the business.
EB: Why are data scientists in such high demand at the moment?
SO: The implication is that they’re also in short supply, and I challenge that assumption. Hiring managers would tell you that they see lots of interest in, and applications for, data science requisitions. Can they be both in high supply and demand at once?
Anyone highly-skilled in two valuable fields will always be in demand. This is self-evident. Hiring for a jack-of-all-trades profile as a hedge against uncertainty, or some attempt to make a role more appealing, will cause problems. If it’s really a business analyst that’s needed — with pay on offer to match — then hiring for a bunch of other attributes at the same time would make it seem like there’s a shortage, when there are in fact plenty of business analysts out there wondering why employer keep wanting them to know Scala.