Until a few years ago the work of data scientists was isolated and mattered primarily for research and/or R&D purposes. The industry has been extremely thankful for the contributions of these clever individuals but we need them now in the mainstream! This need is creating a huge demand in the industry and it's radically transforming the hunt for big data and machine learning talent.
Fast forward 2018 and today every CEO, CIO, CDO, CMO is seeking answers to questions that don't exist yet.
Typical questions in the minds of the leaders could be...
How do I drive innovation within my company, industry or community in a dramatic fashion?
How do we disrupt the industry with our products and services?
How do we change and challenge our workforce's mindset by providing insightful "out-of-the-box" data analytics?
How do we launch a new LOB (Line of Business), product and/or service using these thin/thick/lazy/idle datafied analytics with machine - and deep learning?
Think about it from your own industry perspective.
An Example: The Oil & Gas sector. How do we enhance and optimise exploration and production (E&P)? With conventional technologies becoming commoditised and depleting traditional resources one has to competitively use unconventional methods such as deepwater, tight oil or shale gas. I won't get deeper into aspects such as seismic plotting, WAZ and NATS, etc. as that is only generating more data waiting to be exploited!
Think what it can mean for your industry if you had the "right eyes" to spot the gold!
No industry today can say that it is not data driven. With vast volume, speed and variety of data coming from external and internal sources, the need to scientifically approach data is paramount for competitive intelligence of organisations. That is what is keeping CxOs awake at odd hours.
Data science is the study of the generalizable extraction of knowledge from data, yet the key word is science. It incorporates varying elements and builds on techniques and theories from many fields, including signal processing, mathematics, probability models, machine learning, statistical learning, computer programming, data engineering, pattern recognition and learning, visualization, uncertainty modeling, data warehousing and high performance computing with the goal of extracting meaning from data and creating data products. Data Science is not restricted to only big data, although the fact that data is scaling up makes big data an important aspect of data science.
And who is a Data Scientist?
Data scientists solve complex data problems through employing deep expertise in some scientific discipline.
It is generally expected that data scientists are able to work with various elements of mathematics, statistics and computer science, although expertise in these subjects are not required. However, a data scientist is most likely to be an expert in only one or two of these disciplines and proficient in another two or three. This means that data science must be practiced as a team, where across the membership of the team there is expertise and proficiency across all the disciplines.
Why and Why Now Data Science?
Ok, so having addressed the boring what part, lets get on to the why of it. What is really happening in the industry?
I was having an interesting conversation with one of the co-founders of my previous management consulting startup and we were discussing why the consulting industry was getting disrupted so dramatically. If you didn't notice, many consulting firms such as Booz, Monitor Group, etc. have succumbed already to their traditional "ways of working," and many will follow suit if they don't change the way they work!
The key challenge was that folks did (and still do!) template-based consulting. If you are employed by those firms or have been a client working with those firms then you know exactly what I am talking about.
Customers are increasingly asking for the 'what-if' approach; they don't care anymore about your diagnostic/prescriptive approach. They can do that themselves!
Customers & consumers today are more advanced and data savvy than you can imagine. They can search, collate, splice/dice and come with standard and even relatively advanced options before you can. What they need is expertise, insight and powerful "what-if" thinking! If you don't provide for that, you're out!*
Similar change is happening in the enterprise data management landscape. We are increasingly moving away from a "descriptive or prescriptive, causal, "tell-me-what-happened" model" to "predictive, correlational, all-structured, what-if model."
We have officially entered the era of data-intensive computing. Two factors are encouraging this trend. First, vast amounts of data is becoming available in more and more application areas, and second it is affordable infrastructures (cloud, etc.) allowing for persistently storage/retrieval/sharing of data which was previously unthinkable and/or inaccessible.
This is only getting more intense as application of Artificial Intelligence into IoT (Internet of Things) - which is touted as the next platform, is using emerging technologies such as near-field communications, real-time localisation, and embedded sensors to convert dumb things into smarter things.
This tectonic shift of application of theory, experiments and simulations into mainstream business allows unifying knowledge from scientific research with vast amounts of multidisciplinary data and thereby creating real business wisdom which was previously unthinkable!
The following diagram should make it clear what I mean. Where do you see your organisation in this picture?
Harnessing this Opportunity
If you can spot, sort and commercialise your unique patterns, you can escape competition!
The massive amount of personalised information being pushed out by mankind is a huge opportunity to spot, sort and commercialise on the patterns applicable to your business. Whether it is analyzing billions of transactions to detect fraud (risk management) or forecast/predict consumer behaviour to plan for growth via customer acquisition/retention or development, you need more than just a cool data platform. You need a solid team and a big data management plan.
To build and run successful big data teams and projects you need data scientists within your teams. Big data is not just about Hadoop or some cool new technology, it is about doing the "what-if" before your competitor does. And then move on to the next "what-if."
Instead of only focusing on using a consistent set of data to measure past performance and report for business planning, businesses must also focus on a combination of analytics and machine learning techniques so they can draw inferences and insights out of the massive sea of data. This will help you solve the higher order questions and derive far greater business value that you could have ever imagined!
* We have built a successful business with that mindset and there have been customers who kicked out the "template-based, prescriptive" big consulting guys to choose to work with us. Repeatedly!